Re: [PATCH v1 1/3] io: Enable write flags for QIOChannel

2021-09-01 Thread Eric Blake
On Tue, Aug 31, 2021 at 08:02:37AM -0300, Leonardo Bras wrote: > Some syscalls used for writting, such as sendmsg(), accept flags that > can modify their behavior, even allowing the usage of features such as > MSG_ZEROCOPY. > > Change qio_channel_write*() interface to allow passing down flags, >

Re: [PATCH v1 0/3] QIOChannel flags + multifd zerocopy

2021-09-01 Thread Leonardo Bras Soares Passos
Hello Peter, On Tue, Aug 31, 2021 at 6:24 PM Peter Xu wrote: > > On Tue, Aug 31, 2021 at 08:02:36AM -0300, Leonardo Bras wrote: > > Results: > > So far, the resource usage of __sys_sendmsg() reduced 15 times, and the > > overall migration took 13-18% less time, based in synthetic workload. > >

Re: [PATCH v1 2/3] io: Add zerocopy and errqueue

2021-09-01 Thread Peter Xu
On Wed, Sep 01, 2021 at 09:50:56AM +0100, Daniel P. Berrangé wrote: > On Tue, Aug 31, 2021 at 04:27:04PM -0400, Peter Xu wrote: > > On Tue, Aug 31, 2021 at 01:57:33PM +0100, Daniel P. Berrangé wrote: > > > On Tue, Aug 31, 2021 at 08:02:38AM -0300, Leonardo Bras wrote: > > > > MSG_ZEROCOPY is a

Re: [PATCH v1 2/3] io: Add zerocopy and errqueue

2021-09-01 Thread Daniel P . Berrangé
On Wed, Sep 01, 2021 at 11:52:13AM -0400, Peter Xu wrote: > On Wed, Sep 01, 2021 at 09:50:56AM +0100, Daniel P. Berrangé wrote: > > On Tue, Aug 31, 2021 at 04:27:04PM -0400, Peter Xu wrote: > > > On Tue, Aug 31, 2021 at 01:57:33PM +0100, Daniel P. Berrangé wrote: > > > > On Tue, Aug 31, 2021 at

Re: [PATCH v1 3/3] migration: multifd: Enable zerocopy

2021-09-01 Thread Peter Xu
On Wed, Sep 01, 2021 at 04:44:30PM +0100, Daniel P. Berrangé wrote: > QEMU has mptcp support already: > > commit 8bd1078aebcec5eac196a83ef1a7e74be0ba67b7 > Author: Dr. David Alan Gilbert > Date: Wed Apr 21 12:28:34 2021 +0100 > > sockets: Support multipath TCP > > Multipath

Re: [PATCH v1 3/3] migration: multifd: Enable zerocopy

2021-09-01 Thread Daniel P . Berrangé
On Wed, Sep 01, 2021 at 11:35:33AM -0400, Peter Xu wrote: > On Wed, Sep 01, 2021 at 09:53:07AM +0100, Daniel P. Berrangé wrote: > > On Tue, Aug 31, 2021 at 04:29:09PM -0400, Peter Xu wrote: > > > On Tue, Aug 31, 2021 at 02:16:42PM +0100, Daniel P. Berrangé wrote: > > > > On Tue, Aug 31, 2021 at

[PULL 53/56] iotests/image-fleecing: add test-case for copy-before-write filter

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy New fleecing method becomes available: copy-before-write filter. Actually we don't need backup job to setup image fleecing. Add test for new recommended way of image fleecing. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id:

[PULL 52/56] iotests/image-fleecing: prepare for adding new test-case

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We are going to add a test-case with some behavior modifications. So, let's prepare a function to be reused. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210824083856.17408-33-vsement...@virtuozzo.com> Signed-off-by:

Re: [PATCH v1 3/3] migration: multifd: Enable zerocopy

2021-09-01 Thread Peter Xu
On Wed, Sep 01, 2021 at 09:53:07AM +0100, Daniel P. Berrangé wrote: > On Tue, Aug 31, 2021 at 04:29:09PM -0400, Peter Xu wrote: > > On Tue, Aug 31, 2021 at 02:16:42PM +0100, Daniel P. Berrangé wrote: > > > On Tue, Aug 31, 2021 at 08:02:39AM -0300, Leonardo Bras wrote: > > > > Call

[PULL 51/56] iotests/image-fleecing: rename tgt_node

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Actually target of backup(sync=None) is not a final backup target: image fleecing is intended to be used with external tool, which will copy data from fleecing node to some real backup target. Also, we are going to add a test case for "push backup with

[PULL 48/56] iotests: move 222 to tests/image-fleecing

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Give a good name to test file. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210824083856.17408-29-vsement...@virtuozzo.com> [hreitz: Adjust .gitlab-ci.d/buildtest.yml] Signed-off-by: Hanna Reitz ---

[PULL 55/56] block/export/fuse.c: fix fuse-lseek on uclibc or musl

2021-09-01 Thread Hanna Reitz
From: Fabrice Fontaine Include linux/fs.h to avoid the following build failure on uclibc or musl raised since version 6.0.0: ../block/export/fuse.c: In function 'fuse_lseek': ../block/export/fuse.c:641:19: error: 'SEEK_HOLE' undeclared (first use in this function) 641 | if (whence !=

[PULL 54/56] block/block-copy: block_copy_state_new(): drop extra arguments

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy The only caller pass copy_range and compress both false. Let's just drop these arguments. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210824083856.17408-35-vsement...@virtuozzo.com> Reviewed-by: Hanna Reitz Signed-off-by: Hanna Reitz ---

[PULL 49/56] iotests.py: hmp_qemu_io: support qdev

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Reviewed-by: John Snow Message-Id: <20210824083856.17408-30-vsement...@virtuozzo.com> Signed-off-by: Hanna Reitz --- tests/qemu-iotests/iotests.py | 5 +++-- 1 file changed, 3

[PULL 56/56] block/file-win32: add reopen handlers

2021-09-01 Thread Hanna Reitz
From: Viktor Prutyanov Make 'qemu-img commit' work on Windows. Command 'commit' requires reopening backing file in RW mode. So, add reopen prepare/commit/abort handlers and change dwShareMode for CreateFile call in order to allow further read/write reopening. Resolves:

[PULL 43/56] python/qemu/machine.py: refactor _qemu_args()

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy - use shorter construction - don't create new dict if not needed - drop extra unpacking key-val arguments - drop extra default values Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Reviewed-by: John Snow Message-Id:

[PULL 38/56] block/copy-before-write: cbw_init(): use options

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy One more step closer to .bdrv_open(): use options instead of plain arguments. Move to bdrv_open_child() calls, native for drive open handlers. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz Message-Id:

[PULL 45/56] python:QEMUMachine: template typing for self returning methods

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy mypy thinks that return value of these methods in subclusses is QEMUMachine, which is wrong. So, make typing smarter. Suggested-by: John Snow Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210824083856.17408-26-vsement...@virtuozzo.com>

[PULL 44/56] python/qemu/machine: QEMUMachine: improve qmp() method

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We often call qmp() with unpacking dict, like qmp('foo', **{...}). mypy don't really like it, it thinks that passed unpacked dict is a positional argument and complains that it type should be bool (because second argument of qmp() is conv_keys: bool). Allow

[PULL 30/56] block/copy-before-write: relax permission requirements when no parents

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We are going to publish copy-before-write filter. So, user should be able to create it with blockdev-add first, specifying both filtered and target children. And then do blockdev-reopen, to actually insert the filter where needed. Currently, filter unshares

[PULL 29/56] block/backup: move cluster size calculation to block-copy

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy The main consumer of cluster-size is block-copy. Let's calculate it here instead of passing through backup-top. We are going to publish copy-before-write filter soon, so it will be created through options. But we don't want for now to make explicit option for

[PULL 41/56] block/copy-before-write: make public block driver

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Finally, copy-before-write gets own .bdrv_open and .bdrv_close handlers, block_init() call and becomes available through bdrv_open(). To achieve this: - cbw_init gets unused flags argument and becomes cbw_open - block_copy_state_free() call moved to new

[PULL 39/56] block/copy-before-write: initialize block-copy bitmap

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We are going to publish copy-before-write filter to be used in separate of backup. Future step would support bitmap for the filter. But let's start from full set bitmap. We have to modify backup, as bitmap is first initialized by copy-before-write filter, and

[PULL 27/56] block/block-copy: introduce block_copy_set_copy_opts()

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We'll need a possibility to set compress and use_copy_range options after initialization of the state. So make corresponding part of block_copy_state_new() separate and public. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id:

[PULL 26/56] block-copy: move detecting fleecing scheme to block-copy

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We want to simplify initialization interface of copy-before-write filter as we are going to make it public. So, let's detect fleecing scheme exactly in block-copy code, to not pass this information through extra levels. Why not just set BDRV_REQ_SERIALISING

[PULL 37/56] block/copy-before-write: bdrv_cbw_append(): drop unused compress arg

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz Message-Id: <20210824083856.17408-18-vsement...@virtuozzo.com> Signed-off-by: Hanna Reitz --- block/copy-before-write.h | 1 - block/backup.c| 2 +- block/copy-before-write.c |

[PULL 36/56] block/copy-before-write: cbw_init(): use file child after attaching

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy In the next commit we'll get rid of source argument of cbw_init(). Prepare to it now, to make next commit simpler: move the code block that uses source below attaching the child and use bs->file->bs instead of source variable. Signed-off-by: Vladimir

[PULL 25/56] block: rename backup-top to copy-before-write

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We are going to convert backup_top to full featured public filter, which can be used in separate of backup job. Start from renaming from "how it used" to "what it does". While updating comments in 283 iotest, drop and rephrase also things about ".active", as

[PULL 24/56] qdev: allow setting drive property for realized device

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We need an ability to insert filters above top block node, attached to block device. It can't be achieved with blockdev-reopen command. So, we want do it with help of qom-set. Intended usage: Assume there is a node A that is attached to some guest device. 1.

[PULL 22/56] block: introduce blk_replace_bs

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Add function to change bs inside blk. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210824083856.17408-3-vsement...@virtuozzo.com> Signed-off-by: Hanna Reitz --- include/sysemu/block-backend.h | 1 +

[PULL 35/56] block/copy-before-write: cbw_init(): rename variables

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy One more step closer to real .bdrv_open() handler: use more usual names for bs being initialized and its state. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210824083856.17408-16-vsement...@virtuozzo.com> Signed-off-by:

[PULL 32/56] block/copy-before-write: use file child instead of backing

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We are going to publish copy-before-write filter, and there no public backing-child-based filter in Qemu. No reason to create a precedent, so let's refactor copy-before-write filter instead. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz

[PULL 20/56] iotests: use subprocess.DEVNULL instead of open("/dev/null")

2021-09-01 Thread Hanna Reitz
From: John Snow Avoids a warning from pylint not to use open() outside of a with-statement, and is ... probably more portable anyway. Not that I think we care too much about running tests *on* Windows, but... eh. Signed-off-by: John Snow Message-Id: <20210720173336.1876937-3-js...@redhat.com>

[PULL 19/56] iotests: use with-statement for open() calls

2021-09-01 Thread Hanna Reitz
From: John Snow Silences a new pylint warning. The dangers of *not* doing this are somewhat unclear; I believe the file object gets garbage collected eventually, but possibly the way in which it happens is non-deterministic. Maybe this is a valid warning, but if there are consequences of not

[PULL 18/56] raw-format: drop WRITE and RESIZE child perms when possible

2021-09-01 Thread Hanna Reitz
From: Stefan Hajnoczi The following command-line fails due to a permissions conflict: $ qemu-storage-daemon \ --blockdev driver=nvme,node-name=nvme0,device=:08:00.0,namespace=1 \ --blockdev driver=raw,node-name=l1-1,file=nvme0,offset=0,size=1073741824 \ --blockdev

[PULL 16/56] docs/devel/testing: add -p option to the debug section of QEMU iotests

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210809090114.64834-17-eespo...@redhat.com> Signed-off-by: Hanna Reitz --- docs/devel/testing.rst | 4 1 file changed, 4

[PULL 47/56] iotests/222: constantly use single quotes for strings

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy The file use both single and double quotes for strings. Let's be consistent. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210824083856.17408-28-vsement...@virtuozzo.com> Signed-off-by: Hanna Reitz ---

[PULL 50/56] iotests/image-fleecing: proper source device

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Define scsi device to operate with it by qom-set in further patch. Give a new node-name to source block node, to not look like device name. Job now don't want to work without giving explicit id, so, let's call it "fleecing". Signed-off-by: Vladimir

[PULL 33/56] block/copy-before-write: bdrv_cbw_append(): replace child at last

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Refactor the function to replace child at last. Thus we don't need to revert it and code is simplified. block-copy state initialization being done before replacing the child doesn't need any drained section. Signed-off-by: Vladimir Sementsov-Ogievskiy

[PULL 46/56] iotests/222: fix pylint and mypy complains

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Here: - long line - move to new interface of vm.qmp() (direct passing dict), to avoid mypy false-positive, as it thinks that unpacked dict is a positional argument. - extra parenthesis - handle event_wait possible None value Signed-off-by: Vladimir

[PULL 42/56] qapi: publish copy-before-write filter

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Acked-by: Markus Armbruster Message-Id: <20210824083856.17408-23-vsement...@virtuozzo.com> Signed-off-by: Hanna Reitz --- qapi/block-core.json | 25 +++-- 1 file

[PULL 40/56] block/block-copy: make setting progress optional

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Now block-copy will crash if user don't set progress meter by block_copy_set_progress_meter(). copy-before-write filter will be used in separate of backup job, and it doesn't want any progress meter (for now). So, allow not setting it. Signed-off-by: Vladimir

[PULL 31/56] block/copy-before-write: drop extra bdrv_unref on failure path

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy bdrv_attach_child() do bdrv_unref() on failure, so we shouldn't do it by hand here. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210824083856.17408-12-vsement...@virtuozzo.com> Signed-off-by: Hanna Reitz ---

[PULL 28/56] block/backup: set copy_range and compress after filter insertion

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy We are going to publish copy-before-write filter, so it would be initialized through options. Still we don't want to publish compress and copy-range options, as 1. Modern way to enable compression is to use compress filter. 2. For copy-range it's unclean how

[PULL 34/56] block/copy-before-write: introduce cbw_init()

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Move part of bdrv_cbw_append() to new function cbw_open(). It's an intermediate step for adding normal .bdrv_open() handler to the filter. With this commit no logic is changed, but we have a function which will be turned into .bdrv_open() handler in future

[PULL 14/56] docs/devel/testing: add -valgrind option to the debug section of QEMU iotests

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210809090114.64834-15-eespo...@redhat.com> Signed-off-by: Hanna Reitz --- docs/devel/testing.rst | 6 ++ 1 file changed, 6

[PULL 12/56] qemu-iotests: allow valgrind to read/delete the generated log file

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito When using -valgrind on the script tests, it generates a log file in $TEST_DIR that is either read (if valgrind finds problems) or otherwise deleted. Provide the same exact behavior when using -valgrind on the python tests. Signed-off-by: Emanuele Giuseppe

[PULL 23/56] qdev-properties: PropertyInfo: add realized_set_allowed field

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Add field, so property can declare support for setting the property when device is realized. To be used in the following commit. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210824083856.17408-4-vsement...@virtuozzo.com>

[PULL 21/56] block: introduce bdrv_replace_child_bs()

2021-09-01 Thread Hanna Reitz
From: Vladimir Sementsov-Ogievskiy Add function to transactionally replace bs inside BdrvChild. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210824083856.17408-2-vsement...@virtuozzo.com> Signed-off-by: Hanna Reitz --- include/block/block.h | 2 ++

[PULL 15/56] qemu-iotests: add option to show qemu binary logs on stdout

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Using the flag -p, allow the qemu binary to print to stdout. Also create the common function _close_qemu_log_file() to avoid accessing machine.py private fields directly and have duplicate code. Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Vladimir

[PULL 13/56] qemu-iotests: insert valgrind command line as wrapper for qemu binary

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito If -gdb and -valgrind are both defined, return an error. Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210809090114.64834-14-eespo...@redhat.com> Signed-off-by: Hanna Reitz ---

[PULL 17/56] block/monitor: Consolidate hmp_handle_error calls to reduce redundant code

2021-09-01 Thread Hanna Reitz
From: Mao Zhongyi Signed-off-by: Mao Zhongyi Message-Id: <20210802062507.347555-1-maozhon...@cmss.chinamobile.com> Signed-off-by: Hanna Reitz --- block/monitor/block-hmp-cmds.c | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/block/monitor/block-hmp-cmds.c

[PULL 11/56] qemu-iotests: extend QMP socket timeout when using valgrind

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito As with gdbserver, valgrind delays the test execution, so the default QMP socket timeout and the generic class Timeout in iotests.py timeouts too soon. Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz

[PULL 07/56] qemu_iotests: insert gdbserver command line as wrapper for qemu binary

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210809090114.64834-8-eespo...@redhat.com> Signed-off-by: Hanna Reitz --- tests/qemu-iotests/iotests.py | 3 ++- 1 file changed, 2

[PULL 06/56] qemu-iotests: delay QMP socket timers

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Attaching gdbserver implies that the qmp socket should wait indefinitely for an answer from QEMU. Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Max Reitz Message-Id: <20210809090114.64834-7-eespo...@redhat.com> Signed-off-by: Hanna Reitz ---

[PULL 10/56] qemu-iotests: extend the check script to prepare supporting valgrind for python tests

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Currently, the check script only parses the option and sets the VALGRIND_QEMU environmental variable to "y". Add another local python variable that prepares the command line, identical to the one provided in the test scripts. Because the python script does not

[PULL 08/56] qemu-iotests: add gdbserver option to script tests too

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Remove read timer in test script when GDB_OPTIONS are set, so that the bash tests won't timeout while running gdb. The only limitation here is that running a script with gdbserver will make the test output mismatch with the expected results, making the test

[PULL 05/56] qemu-iotests: add option to attach gdbserver

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Define -gdb flag and GDB_OPTIONS environment variable to python tests to attach a gdbserver to each qemu instance. This patch only adds and parses this flag, it does not yet add the implementation for it. if -gdb is not provided but $GDB_OPTIONS is set, ignore

[PULL 09/56] docs/devel/testing: add -gdb option to the debugging section of QEMU iotests

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210809090114.64834-10-eespo...@redhat.com> Signed-off-by: Hanna Reitz --- docs/devel/testing.rst | 11 +++ 1 file changed, 11

[PULL 02/56] python: Reduce strictness of pylint's duplicate-code check

2021-09-01 Thread Hanna Reitz
From: John Snow Pylint prior to 2.8.3 (We pin at >= 2.8.0) includes function and method signatures as part of its duplicate checking algorithm. This check does not listen to pragmas, so the only way to disable it is to turn it off completely or increase the minimum duplicate lines so that it

[PULL 03/56] python: qemu: pass the wrapper field from QEMUQtestmachine to QEMUMachine

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: John Snow Reviewed-by: Max Reitz Acked-by: John Snow Message-Id: <20210809090114.64834-4-eespo...@redhat.com> Signed-off-by: Hanna Reitz ---

[PULL 04/56] docs/devel/testing: add debug section to the QEMU iotests chapter

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Introduce the "Debugging a test case" section, in preparation to the additional flags that will be added in the next patches. Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id:

[PULL 00/56] Block patches

2021-09-01 Thread Hanna Reitz
The following changes since commit ec397e90d21269037280633b6058d1f280e27667: Merge remote-tracking branch 'remotes/alistair/tags/pull-riscv-to-apply-20210901-2' into staging (2021-09-01 08:33:02 +0100) are available in the Git repository at: https://github.com/XanClic/qemu.git tags/pull

[PULL 01/56] python: qemu: add timer parameter for qmp.accept socket

2021-09-01 Thread Hanna Reitz
From: Emanuele Giuseppe Esposito Also add a new _qmp_timer field to the QEMUMachine class. Let's change the default socket timeout to None, so that if a subclass needs to add a timer, it can be done by modifying this private field. At the same time, restore the timer to be 15 seconds in

Re: [PATCH v3 0/4] iotests/297: Cover tests/

2021-09-01 Thread Hanna Reitz
On 01.09.21 15:34, Vladimir Sementsov-Ogievskiy wrote: A kind of ping:) Seems that never landed into master? Yes, that’s true… I was waiting for John (CC-ed) to send v3 of https://lists.nongnu.org/archive/html/qemu-block/2021-07/msg00611.html, because in

Re: [PATCH v3 00/10] qcow2 check: check some reserved bits and subcluster bitmaps

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
Ping again:) Nothing changed: patches applies to master, 08 doesn't have r-b. 03.07.2021 14:17, Vladimir Sementsov-Ogievskiy wrote: Ping :) This still applies to master with no conflicts. All patches reviewed except for 08. 24.05.2021 17:20, Vladimir Sementsov-Ogievskiy wrote: Hi all!

Re: [PATCH v3 0/4] iotests/297: Cover tests/

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
A kind of ping:) Seems that never landed into master? 14.05.2021 18:43, Max Reitz wrote: v1: https://lists.nongnu.org/archive/html/qemu-block/2021-03/msg01471.html v2: https://lists.nongnu.org/archive/html/qemu-block/2021-05/msg00492.html Hi, When reviewing Vladimir’s new addition to

Re: [PATCH v2 3/3] qapi: deprecate drive-backup

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
24.05.2021 21:37, John Snow wrote: On 5/24/21 10:06 AM, Vladimir Sementsov-Ogievskiy wrote: 15.05.2021 01:38, John Snow wrote: On 5/6/21 5:57 AM, Kashyap Chamarthy wrote: TODO: We also need to deprecate drive-backup transaction action.. But union members in QAPI doesn't support 'deprecated'

Re: [PATCH v2 00/17] iotests: support zstd

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
Ping. This applies to current master with a small obvious conflict in tests/qemu-iotests/common.filter 20.07.2021 14:38, Vladimir Sementsov-Ogievskiy wrote: These series makes tests pass with IMGOPTS='compression_type=zstd' Also, python iotests start to support IMGOPTS (they didn't

Re: [PATCH v8 28/34] iotests: move 222 to tests/image-fleecing

2021-09-01 Thread Daniel P . Berrangé
On Wed, Sep 01, 2021 at 02:37:52PM +0200, Hanna Reitz wrote: > On 24.08.21 10:38, Vladimir Sementsov-Ogievskiy wrote: > > Give a good name to test file. > > > > Signed-off-by: Vladimir Sementsov-Ogievskiy > > Reviewed-by: Max Reitz > > --- > > tests/qemu-iotests/{222 => tests/image-fleecing}

Re: [PATCH v8 28/34] iotests: move 222 to tests/image-fleecing

2021-09-01 Thread Hanna Reitz
On 01.09.21 14:47, Vladimir Sementsov-Ogievskiy wrote: 01.09.2021 15:37, Hanna Reitz wrote: On 24.08.21 10:38, Vladimir Sementsov-Ogievskiy wrote: Give a good name to test file. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz ---   tests/qemu-iotests/{222 =>

Re: [PATCH for-6.2 v3 05/12] job: @force parameter for job_cancel_sync{,_all}()

2021-09-01 Thread Hanna Reitz
On 01.09.21 13:04, Vladimir Sementsov-Ogievskiy wrote: 06.08.2021 12:38, Max Reitz wrote: @@ -726,7 +726,7 @@ static void replication_stop(ReplicationState *rs, bool failover, Error **errp)    * disk, secondary disk in backup_job_completed().    */   if (s->backup_job)

Re: [PATCH for-6.2 v3 01/12] job: Context changes in job_completed_txn_abort()

2021-09-01 Thread Hanna Reitz
On 01.09.21 12:05, Vladimir Sementsov-Ogievskiy wrote: 06.08.2021 12:38, Max Reitz wrote: Finalizing the job may cause its AioContext to change.  This is noted by job_exit(), which points at job_txn_apply() to take this fact into account. However, job_completed() does not necessarily invoke

Re: [PATCH v8 28/34] iotests: move 222 to tests/image-fleecing

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
01.09.2021 15:37, Hanna Reitz wrote: On 24.08.21 10:38, Vladimir Sementsov-Ogievskiy wrote: Give a good name to test file. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz ---   tests/qemu-iotests/{222 => tests/image-fleecing} | 0   tests/qemu-iotests/{222.out =>

Re: [PATCH for-6.2 v3 05/12] job: @force parameter for job_cancel_sync{,_all}()

2021-09-01 Thread Hanna Reitz
On 01.09.21 12:20, Vladimir Sementsov-Ogievskiy wrote: 06.08.2021 12:38, Max Reitz wrote: Callers should be able to specify whether they want job_cancel_sync() to force-cancel the job or not. In fact, almost all invocations do not care about consistency of the result and just want the job to

Re: [PATCH v8 28/34] iotests: move 222 to tests/image-fleecing

2021-09-01 Thread Hanna Reitz
On 24.08.21 10:38, Vladimir Sementsov-Ogievskiy wrote: Give a good name to test file. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- tests/qemu-iotests/{222 => tests/image-fleecing} | 0 tests/qemu-iotests/{222.out => tests/image-fleecing.out} | 0 2 files

Re: [PATCH for-6.2 v3 11/12] mirror: Do not clear .cancelled

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
06.08.2021 12:38, Max Reitz wrote: Clearing .cancelled before leaving the main loop when the job has been soft-cancelled is no longer necessary since job_is_cancelled() only returns true for jobs that have been force-cancelled. Therefore, this only makes a differences in places that call

Re: [PATCH for-6.2 v3 10/12] mirror: Stop active mirroring after force-cancel

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
06.08.2021 12:38, Max Reitz wrote: Once the mirror job is force-cancelled (job_is_cancelled() is true), we should not generate new I/O requests. This applies to active mirroring, too, so stop it once the job is cancelled. (We must still forward all I/O requests to the source, though, of

Re: [PATCH for-6.2 v3 09/12] mirror: Check job_is_cancelled() earlier

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
06.08.2021 12:38, Max Reitz wrote: We must check whether the job is force-cancelled early in our main loop, most importantly before any `continue` statement. For example, we used to have `continue`s before our current checking location that are triggered by `mirror_flush()` failing. So, if

Re: [PATCH v8 09/34] block/backup: move cluster size calculation to block-copy

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
01.09.2021 14:57, Hanna Reitz wrote: On 24.08.21 10:38, Vladimir Sementsov-Ogievskiy wrote: The main consumer of cluster-size is block-copy. Let's calculate it here instead of passing through backup-top. We are going to publish copy-before-write filter soon, so it will be created through

Re: [PATCH v8 09/34] block/backup: move cluster size calculation to block-copy

2021-09-01 Thread Hanna Reitz
On 24.08.21 10:38, Vladimir Sementsov-Ogievskiy wrote: The main consumer of cluster-size is block-copy. Let's calculate it here instead of passing through backup-top. We are going to publish copy-before-write filter soon, so it will be created through options. But we don't want for now to make

Re: [PATCH for-6.2 v3 08/12] mirror: Use job_is_cancelled()

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
06.08.2021 12:38, Max Reitz wrote: mirror_drained_poll() returns true whenever the job is cancelled, because "we [can] be sure that it won't issue more requests". However, this is only true for force-cancelled jobs, so use job_is_cancelled(). Signed-off-by: Max Reitz Reviewed-by: Vladimir

Re: [PATCH for-6.2 v3 07/12] job: Add job_cancel_requested()

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
06.08.2021 12:38, Max Reitz wrote: Most callers of job_is_cancelled() actually want to know whether the job is on its way to immediate termination. For example, we refuse to pause jobs that are cancelled; but this only makes sense for jobs that are really actually cancelled. A mirror job that

Re: [PATCH v5] block/file-win32: add reopen handlers

2021-09-01 Thread Hanna Reitz
On 25.08.21 19:36, Viktor Prutyanov wrote: Make 'qemu-img commit' work on Windows. Command 'commit' requires reopening backing file in RW mode. So, add reopen prepare/commit/abort handlers and change dwShareMode for CreateFile call in order to allow further read/write reopening. Resolves:

Re: [PATCH for-6.2 v3 05/12] job: @force parameter for job_cancel_sync{,_all}()

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
06.08.2021 12:38, Max Reitz wrote: @@ -726,7 +726,7 @@ static void replication_stop(ReplicationState *rs, bool failover, Error **errp) * disk, secondary disk in backup_job_completed(). */ if (s->backup_job) { -job_cancel_sync(>backup_job->job); +

Re: [PATCH] block/export/fuse.c: fix fuse-lseek on uclibc or musl

2021-09-01 Thread Hanna Reitz
On 28.08.21 00:03, Fabrice Fontaine wrote: Include linux/fs.h to avoid the following build failure on uclibc or musl raised since version 6.0.0: ../block/export/fuse.c: In function 'fuse_lseek': ../block/export/fuse.c:641:19: error: 'SEEK_HOLE' undeclared (first use in this function) 641 |

Re: [PATCH for-6.2 v3 05/12] job: @force parameter for job_cancel_sync{,_all}()

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
06.08.2021 12:38, Max Reitz wrote: Callers should be able to specify whether they want job_cancel_sync() to force-cancel the job or not. In fact, almost all invocations do not care about consistency of the result and just want the job to terminate as soon as possible, so they should pass

Re: [PATCH for-6.2 v3 01/12] job: Context changes in job_completed_txn_abort()

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
06.08.2021 12:38, Max Reitz wrote: Finalizing the job may cause its AioContext to change. This is noted by job_exit(), which points at job_txn_apply() to take this fact into account. However, job_completed() does not necessarily invoke job_txn_apply() (through job_completed_txn_success()), but

Re: [PATCH for-6.2 v3 04/12] job: Force-cancel jobs in a failed transaction

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
06.08.2021 12:38, Max Reitz wrote: When a transaction is aborted, no result matters, and so all jobs within should be force-cancelled. Signed-off-by: Max Reitz --- job.c | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/job.c b/job.c index 3fe23bb77e..24e7c4fcb7

Re: [PATCH] nbd/server: Advertise MULTI_CONN for shared writable exports

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
27.08.2021 21:45, Eric Blake wrote: On Fri, Aug 27, 2021 at 07:58:10PM +0300, Vladimir Sementsov-Ogievskiy wrote: 27.08.2021 18:09, Eric Blake wrote: According to the NBD spec, a server advertising NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache

Re: [PATCH v1 3/3] migration: multifd: Enable zerocopy

2021-09-01 Thread Daniel P . Berrangé
On Tue, Aug 31, 2021 at 04:29:09PM -0400, Peter Xu wrote: > On Tue, Aug 31, 2021 at 02:16:42PM +0100, Daniel P. Berrangé wrote: > > On Tue, Aug 31, 2021 at 08:02:39AM -0300, Leonardo Bras wrote: > > > Call qio_channel_set_zerocopy(true) in the start of every multifd thread. > > > > > > Change the

Re: [PATCH v1 2/3] io: Add zerocopy and errqueue

2021-09-01 Thread Daniel P . Berrangé
On Tue, Aug 31, 2021 at 04:27:04PM -0400, Peter Xu wrote: > On Tue, Aug 31, 2021 at 01:57:33PM +0100, Daniel P. Berrangé wrote: > > On Tue, Aug 31, 2021 at 08:02:38AM -0300, Leonardo Bras wrote: > > > MSG_ZEROCOPY is a feature that enables copy avoidance in TCP/UDP socket > > > send calls. It does

Re: [PATCH] nbd/server: Advertise MULTI_CONN for shared writable exports

2021-09-01 Thread Vladimir Sementsov-Ogievskiy
27.08.2021 18:09, Eric Blake wrote: According to the NBD spec, a server advertising NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another