Re: [PATCH v2 5/6] tests/qtest: massively speed up migration-tet

2023-04-21 Thread Juan Quintela
Daniel P. Berrangé wrote: > The migration test cases that actually exercise live migration want to > ensure there is a minimum of two iterations of pre-copy, in order to > exercise the dirty tracking code. > > Historically we've queried the migration status, looking for the > 'dirty-sync-count'

Re: [PATCH v2 4/6] tests/qtest: make more migration pre-copy scenarios run non-live

2023-04-21 Thread Juan Quintela
Daniel P. Berrangé wrote: > There are 27 pre-copy live migration scenarios being tested. In all of > these we force non-convergance and run for one iteration, then let it > converge and wait for completion during the second (or following) > iterations. At 3 mbps bandwidth limit the first

Re: [PATCH v2 3/6] tests/qtest: capture RESUME events during migration

2023-04-21 Thread Juan Quintela
Daniel P. Berrangé wrote: > When running migration tests we monitor for a STOP event so we can skip > redundant waits. This will be needed for the RESUME event too shortly. > > Signed-off-by: Daniel P. Berrangé Reviewed-by: Juan Quintela i.e. it is better that what we have now. But > diff

Re: [PATCH v2 2/6] tests/qtests: remove migration test iterations config

2023-04-21 Thread Juan Quintela
Daniel P. Berrangé wrote: > The 'unsigned int interations' config for migration is somewhat > overkill. Most tests don't set it, and a value of '0' is treated > as equivalent to '1'. The only test that does set it, xbzrle, > used a value of '2'. > > This setting, however, only relates to the

Re: [PATCH v2 1/6] tests/qtest: replace qmp_discard_response with qtest_qmp_assert_success

2023-04-21 Thread Juan Quintela
Daniel P. Berrangé wrote: > The qmp_discard_response method simply ignores the result of the QMP > command, merely unref'ing the object. This is a bad idea for tests > as it leaves no trace if the QMP command unexpectedly failed. The > qtest_qmp_assert_success method will validate that the QMP

[PATCH v2 3/6] tests/qtest: capture RESUME events during migration

2023-04-21 Thread Daniel P . Berrangé
When running migration tests we monitor for a STOP event so we can skip redundant waits. This will be needed for the RESUME event too shortly. Signed-off-by: Daniel P. Berrangé --- tests/qtest/migration-helpers.c | 12 +--- tests/qtest/migration-helpers.h | 1 +

[PATCH v2 6/6] tests/migration: Only run auto_converge in slow mode

2023-04-21 Thread Daniel P . Berrangé
From: Juan Quintela Signed-off-by: Juan Quintela --- tests/qtest/migration-test.c | 23 +-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 63bd8a1893..9ed178aa03 100644 ---

[PATCH v2 5/6] tests/qtest: massively speed up migration-tet

2023-04-21 Thread Daniel P . Berrangé
The migration test cases that actually exercise live migration want to ensure there is a minimum of two iterations of pre-copy, in order to exercise the dirty tracking code. Historically we've queried the migration status, looking for the 'dirty-sync-count' value to increment to track iterations.

[PATCH v2 0/6] tests/qtest: make migration-test massively faster

2023-04-21 Thread Daniel P . Berrangé
This makes migration-test faster by observing that most of the pre-copy tests don't need to be doing a live migration. They get sufficient code coverage with the guest CPUs paused. On my machine this cuts the overall execution time of migration-test by 50% from 15 minutes, down to 8 minutes,

[PATCH v2 4/6] tests/qtest: make more migration pre-copy scenarios run non-live

2023-04-21 Thread Daniel P . Berrangé
There are 27 pre-copy live migration scenarios being tested. In all of these we force non-convergance and run for one iteration, then let it converge and wait for completion during the second (or following) iterations. At 3 mbps bandwidth limit the first iteration takes a very long time (~30

[PATCH v2 1/6] tests/qtest: replace qmp_discard_response with qtest_qmp_assert_success

2023-04-21 Thread Daniel P . Berrangé
The qmp_discard_response method simply ignores the result of the QMP command, merely unref'ing the object. This is a bad idea for tests as it leaves no trace if the QMP command unexpectedly failed. The qtest_qmp_assert_success method will validate that the QMP command returned without error, and

[PATCH v2 2/6] tests/qtests: remove migration test iterations config

2023-04-21 Thread Daniel P . Berrangé
The 'unsigned int interations' config for migration is somewhat overkill. Most tests don't set it, and a value of '0' is treated as equivalent to '1'. The only test that does set it, xbzrle, used a value of '2'. This setting, however, only relates to the migration iterations that take place prior

Re: [PATCH 4/4] block: convert more bdrv_is_allocated* and bdrv_block_status* calls to coroutine versions

2023-04-21 Thread Eric Blake
On Wed, Apr 05, 2023 at 12:32:16PM +0200, Paolo Bonzini wrote: > Signed-off-by: Paolo Bonzini > --- > block/copy-before-write.c | 2 +- > block/copy-on-read.c | 8 > block/io.c| 6 +++--- > block/mirror.c| 10 +- > block/qcow2.c |

Re: [PATCH 3/4] block: switch to co_wrapper for bdrv_is_allocated_*

2023-04-21 Thread Eric Blake
On Wed, Apr 05, 2023 at 12:32:15PM +0200, Paolo Bonzini wrote: > Signed-off-by: Paolo Bonzini > --- > block/io.c | 53 ++-- > include/block/block-io.h | 12 + > 2 files changed, 14 insertions(+), 51 deletions(-) > Reviewed-by: Eric Blake

Re: [PATCH 2/4] block: complete public block status API

2023-04-21 Thread Eric Blake
On Wed, Apr 05, 2023 at 12:32:14PM +0200, Paolo Bonzini wrote: > Include both coroutine and non-coroutine versions, the latter being > co_wrapper_mixed_bdrv_rdlock of the former. > > Signed-off-by: Paolo Bonzini > --- > block/io.c | 18 +- >

Re: [PATCH 1/4] block: rename the bdrv_co_block_status static function

2023-04-21 Thread Eric Blake
On Wed, Apr 05, 2023 at 12:32:13PM +0200, Paolo Bonzini wrote: > bdrv_block_status exists as a wrapper for bdrv_block_status_above, > but the name of the (hypothetical) coroutine version, bdrv_co_block_status, > is squatted by a random static function. Rename it to bdrv_do_block_status. > >

[PATCH v8 8/8] memory: abort on re-entrancy in debug builds

2023-04-21 Thread Alexander Bulekov
This is useful for using unit-tests/fuzzing to detect bugs introduced by the re-entrancy guard mechanism into devices that are intentionally re-entrant. Signed-off-by: Alexander Bulekov --- softmmu/memory.c | 3 +++ util/async.c | 3 +++ 2 files changed, 6 insertions(+) diff --git

[PATCH v8 4/8] hw: replace most qemu_bh_new calls with qemu_bh_new_guarded

2023-04-21 Thread Alexander Bulekov
This protects devices from bh->mmio reentrancy issues. Thanks: Thomas Huth for diagnosing OS X test failure. Reviewed-by: Darren Kenny Reviewed-by: Stefan Hajnoczi Reviewed-by: Michael S. Tsirkin Reviewed-by: Paul Durrant Signed-off-by: Alexander Bulekov Reviewed-by: Thomas Huth ---

[PATCH v8 2/8] async: Add an optional reentrancy guard to the BH API

2023-04-21 Thread Alexander Bulekov
Devices can pass their MemoryReentrancyGuard (from their DeviceState), when creating new BHes. Then, the async API will toggle the guard before/after calling the BH call-back. This prevents bh->mmio reentrancy issues. Reviewed-by: Darren Kenny Signed-off-by: Alexander Bulekov ---

Re: [PATCH v5 00/16] iotests: use vm.cmd()

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
On 15.02.23 16:25, Vladimir Sementsov-Ogievskiy wrote: Hi all! Let's get rid of pattern result = self.vm.qmp(...) self.assert_qmp(result, 'return', {}) And switch to just self.vm.cmd(...) v5: - include fixups by John - fix 'make check-dev' - split first patch into several

[PATCH v8 5/5] blockdev: qmp_transaction: drop extra generic layer

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Let's simplify things: First, actions generally don't need access to common BlkActionState structure. The only exclusion are backup actions that need block_job_txn. Next, for transaction actions of Transaction API is more native to allocated state structure in the action itself. So, do the

[PATCH v8 4/5] blockdev: transaction: refactor handling transaction properties

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Only backup supports GROUPED mode. Make this logic more clear. And avoid passing extra thing to each action. Signed-off-by: Vladimir Sementsov-Ogievskiy --- blockdev.c | 92 +++--- 1 file changed, 19 insertions(+), 73 deletions(-) diff --git

[PATCH v8 3/5] blockdev: qmp_transaction: refactor loop to classic for

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- blockdev.c | 9 +++-- 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/blockdev.c b/blockdev.c index 2174ab2694..89c573a094 100644 --- a/blockdev.c +++ b/blockdev.c @@ -2375,7 +2375,7 @@ void qmp_transaction(TransactionActionList

[PATCH v8 2/5] blockdev: transactions: rename some things

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Look at qmp_transaction(): dev_list is not obvious name for list of actions. Let's look at qapi spec, this argument is "actions". Let's follow the common practice of using same argument names in qapi scheme and code. To be honest, rename props to properties for same reason. Next, we have to

[PATCH v8 0/5] block: refactor blockdev transactions

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Hi all! v8: resend, fix my emails That's a portion of old good "[PATCH v5 00/45] Transactional block-graph modifying API". Let's refactor QMP transactions implementation into new (relatively) transaction API. Vladimir Sementsov-Ogievskiy (5): blockdev: refactor transaction to use Transaction

[PATCH v8 1/5] blockdev: refactor transaction to use Transaction API

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
We are going to add more block-graph modifying transaction actions, and block-graph modifying functions are already based on Transaction API. Next, we'll need to separately update permissions after several graph-modifying actions, and this would be simple with help of Transaction API. So, now

[PATCH v7 7/7] iotests: add filter-insertion

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Demonstrate new blockdev-replace API for filter insertion and removal. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/filter-insertion | 257 ++ tests/qemu-iotests/tests/filter-insertion.out | 5 + 2 files changed, 262 insertions(+) create mode

[PATCH v7 5/7] block: bdrv_get_xdbg_block_graph(): report export ids

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Currently for block exports we report empty blk names. That's not good. Let's try to find corresponding block export and report its id. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block.c | 4 block/export/export.c | 13 + include/block/export.h

[PATCH v7 4/7] qapi: add x-blockdev-replace command

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Add a command that can replace bs in following BdrvChild structures: - qdev blk root child - block-export blk root child - any child of BlockDriverState selected by child-name Signed-off-by: Vladimir Sementsov-Ogievskiy --- blockdev.c | 56 ++

[PATCH v7 2/7] block/export: add blk_by_export_id()

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
We need it for further blockdev-replace functionality. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/export/export.c | 18 ++ include/sysemu/block-backend-global-state.h | 1 + 2 files changed, 19 insertions(+) diff --git a/block/export/export.c

[PATCH v7 1/7] block-backend: blk_root(): drop const specifier on return type

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
We'll need get non-const child pointer for graph modifications in further commits. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/block-backend.c | 2 +- include/sysemu/block-backend-global-state.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git

[PATCH v7 6/7] iotests.py: introduce VM.assert_edges_list() method

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Add an alternative method to check block graph, to be used in further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 17 + 1 file changed, 17 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index

[PATCH v7 3/7] block: make bdrv_find_child() function public

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
To be reused soon for blockdev-replace functionality. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block.c | 13 + blockdev.c | 14 -- include/block/block_int-io.h | 1 + 3 files changed, 14 insertions(+), 14 deletions(-) diff

[PATCH v7 0/7] blockdev-replace

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Hi all! That's a non-transactional part of "[PATCH v5 00/45] Transactional block-graph modifying API". The new command helps to insert filters everywhere you want: - replace qdev block-node by qdev-id - replace export block-node by export-id - replace any child of parent block-node by

Re: test-blockjob: intermittent CI failures in msys2-64bit job

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
On 17.03.23 15:35, Thomas Huth wrote: On 17/03/2023 11.17, Peter Maydell wrote: On Mon, 6 Mar 2023 at 11:16, Peter Maydell wrote: On Fri, 3 Mar 2023 at 18:36, Peter Maydell wrote: I've noticed that test-blockjob seems to fail intermittently on the msys2-64bit job:

Re: [PATCH 2/3] block: do not check bdrv_file_open

2023-04-21 Thread Paolo Bonzini
On Thu, Jan 19, 2023 at 2:17 PM Kevin Wolf wrote: > > assert(!drv->bdrv_needs_filename || bs->filename[0]); > > -if (drv->bdrv_file_open) { > > +if (drv->bdrv_open) { > > ret = drv->bdrv_file_open(bs, options, open_flags, _err); > > } else if (drv->bdrv_open) { > >

Re: [PATCH 0/4] block: clean up coroutine versions of bdrv_{is_allocated, block_status}*

2023-04-21 Thread Paolo Bonzini
Ping. Paolo On Wed, Apr 5, 2023 at 12:32 PM Paolo Bonzini wrote: > > Provide coroutine versions of bdrv_is_allocated* and bdrv_block_status*, > since the underlying BlockDriver API is coroutine-based, and use > automatically-generated wrappers for the "mixed" versions. > > Paolo > > Paolo

[PATCH v2] block: add configure options for excluding vmdk, vhdx and vpc

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
Let's add --enable / --disable configure options for these formats, so that those who don't need them may not build them. Signed-off-by: Vladimir Sementsov-Ogievskiy --- v2: just a resend instead of ping, bonus: fix mistake in commit message. block/meson.build | 18

Re: [PATCH v2 4/4] configure: add --disable-colo-filters option

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
On 21.04.23 05:22, Zhang, Chen wrote: -Original Message- From: Vladimir Sementsov-Ogievskiy Sent: Thursday, April 20, 2023 7:26 PM To: Zhang, Chen ; qemu-de...@nongnu.org Cc: qemu-block@nongnu.org; michael.r...@amd.com; arm...@redhat.com; ebl...@redhat.com; jasow...@redhat.com;

Re: [PATCH v2 3/4] build: move COLO under CONFIG_REPLICATION

2023-04-21 Thread Vladimir Sementsov-Ogievskiy
On 21.04.23 06:02, Zhang, Chen wrote: -Original Message- From: Vladimir Sementsov-Ogievskiy Sent: Thursday, April 20, 2023 6:53 AM To: qemu-de...@nongnu.org Cc: qemu-block@nongnu.org; michael.r...@amd.com; arm...@redhat.com; ebl...@redhat.com; jasow...@redhat.com;

Re: [PATCH] block/vhost-user-blk: Fix hang on boot for some odd guests

2023-04-21 Thread Michael S. Tsirkin
On Tue, Apr 18, 2023 at 07:20:29PM +0200, Andrey Ryabinin wrote: > > > On 4/18/23 08:17, Michael S. Tsirkin wrote: > > On Tue, Apr 18, 2023 at 05:13:11AM +, Raphael Norwitz wrote: > >> Hey Andrey - apologies for the late reply here. > >> > >> It sounds like you are dealing with a buggy