[Qemu-block] [PATCH 3/3] iotests: Allow 147 to be run concurrently

2018-12-21 Thread Max Reitz
To do this, we need to allow creating the NBD server on various ports instead of a single one (which may not even work if you run just one instance, because something entirely else might be using that port). So we just pick a random port in [32768, 32768 + 1024) and try to create a server there.

[Qemu-block] [PATCH 0/3] iotests: Allow 147 to be run concurrently

2018-12-21 Thread Max Reitz
147 currently cannot be run concurrently for two reasons: (1) It creates TCP/IP NBD servers on a fixed port. (2) It uses a mix of "0.0.0.0", "localhost", and "::1" as host addresses to bind to. As explained in the commit messages of patches 2 and 3, this results in it potentially

[Qemu-block] [PATCH 1/3] iotests.py: Add qemu_nbd_pipe()

2018-12-21 Thread Max Reitz
In some cases, we may want to deal with qemu-nbd errors (e.g. by launching it in a different configuration until it no longer throws any). In that case, we do not want its output ending up in the test output. It may still be useful for handling the error, though, so add a new function that works

[Qemu-block] [PATCH 2/3] iotests: Bind qemu-nbd to localhost in 147

2018-12-21 Thread Max Reitz
By default, qemu-nbd binds to 0.0.0.0. However, we then proceed to connect to "localhost". Usually, this works out fine; but if this test is run concurrently, some other test function may have bound a different server to ::1 (on the same port -- you can bind different serves to the same port, as

Re: [Qemu-block] [PATCH v6 09/11] iotests: change qmp_log filters to expect QMP objects only

2018-12-21 Thread John Snow
On 12/21/18 7:41 AM, Vladimir Sementsov-Ogievskiy wrote: > Hmm. This made me check, is enumerate applicable to dicts, > and, yes it is. > enumerate on dicts gives you a numerical index paired with the key, so... k is the numerical index and v is the key. > so, it may be as easy as > >

[Qemu-block] [PATCH] iotests: Make 234 stable

2018-12-21 Thread Max Reitz
This test waits for a MIGRATION event with status=completed on the source VM before querying the migration status on both source and destination. However, just because the source says migration has completed does not mean the destination thinks the same. Therefore, in some cases, the destination

[Qemu-block] [PATCH v6 3/3] not-for-commit: example of new command usage for debugging

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/222 | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/qemu-iotests/222 b/tests/qemu-iotests/222 index 0ead56d574..91d88aa5c0 100644 --- a/tests/qemu-iotests/222 +++ b/tests/qemu-iotests/222 @@ -137,6 +137,8 @@ with

[Qemu-block] [PATCH v6 0/3] block nodes graph visualization

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
Hi all! On the way of backup schemes development (and in general any complicated developments in Qemu block layer) it would be good to have an ability to print out graph of block nodes with their permissions. v6: s/3.1/4.0 v5: 01: fix !name -> !*name for checking the result of blk_name

[Qemu-block] [PATCH v6 1/3] qapi: add x-debug-query-block-graph

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
Add a new command, returning block nodes (and their users) graph. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 108 include/block/block.h | 1 + include/sysemu/block-backend.h | 2 + block.c| 148

[Qemu-block] [PATCH v6 2/3] scripts: add render_block_graph function for QEMUMachine

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
Render block nodes graph with help of graphviz. This new function is for debugging, so there is no sense to put it into qemu.py as a method of QEMUMachine. Let's instead put it separately. Signed-off-by: Vladimir Sementsov-Ogievskiy Acked-by: Eduardo Habkost Reviewed-by: Max Reitz ---

[Qemu-block] [PATCH v3 0/2] improve block-latency-histogram-set

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
Support QOM path for block-latency-histogram-set and drop x- prefix. Libvirt discussion (not yet updated for using id instead of device): https://www.redhat.com/archives/libvir-list/2018-September/msg00011.html v3: rebase on master, s/3.1/4.0. v2: id,device -> id (with fall back to device name)

[Qemu-block] [PATCH v3 1/2] qapi: move to QOM path for x-block-latency-histogram-set

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
Move to way of device selecting, however fall back to device name if path is not found. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 4 ++-- blockdev.c | 22 +++--- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git

[Qemu-block] [PATCH v3 2/2] qapi: drop x- from x-block-latency-histogram-set

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
Drop x- and x_ prefixes for latency histograms and update version to 3.1 Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 20 ++-- block/qapi.c | 12 ++-- blockdev.c | 2 +- 3 files changed, 17 insertions(+), 17 deletions(-) diff

Re: [Qemu-block] [PATCH v2 0/2] improve block-latency-histogram-set

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
21.12.2018 19:31, Vladimir Sementsov-Ogievskiy wrote: > ping oops, should s/3.1/4.0 > > 12.10.2018 16:15, Vladimir Sementsov-Ogievskiy wrote: >> Support QOM path for block-latency-histogram-set and drop x- prefix. >> >> Libvirt discussion (not yet updated for using id instead of device): >>

Re: [Qemu-block] [PATCH v2 0/2] improve block-latency-histogram-set

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
ping 12.10.2018 16:15, Vladimir Sementsov-Ogievskiy wrote: > Support QOM path for block-latency-histogram-set and drop x- prefix. > > Libvirt discussion (not yet updated for using id instead of device): > https://www.redhat.com/archives/libvir-list/2018-September/msg00011.html > > v2: id,device

Re: [Qemu-block] [PATCH v11 09/10] qcow2: skip writing zero buffers to empty COW areas

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
18.12.2018 10:57, Anton Nefedov wrote: > If COW areas of the newly allocated clusters are zeroes on the backing image, > efficient bdrv_write_zeroes(flags=BDRV_REQ_ALLOCATE) can be used on the whole > cluster instead of writing explicit zero buffers later in perform_cow(). > > iotest 060: > write

Re: [Qemu-block] [PATCH v11 07/10] file-posix: support BDRV_REQ_ALLOCATE

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
18.12.2018 10:57, Anton Nefedov wrote: > Current write_zeroes implementation is good enough to satisfy this flag too > > Signed-off-by: Anton Nefedov Reviewed-by: Vladimir Sementsov-Ogievskiy > --- > block/file-posix.c | 8 +++- > 1 file changed, 7 insertions(+), 1 deletion(-) > >

[Qemu-block] [PATCH] dmg: fix binary search

2018-12-21 Thread yuchenlin
There is a possible hang in original binary searsh implemtation. That is if chunk1 = 4, chunk2 = 5, chunk3 = 4, and we go else case. The chunk1 will be still 4, and so on. Signed-off-by: yuchenlin --- block/dmg.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git

Re: [Qemu-block] [PATCH v11 06/10] file-posix: reset fallocate-related flags without CONFIG_FALLOCATE*

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
18.12.2018 10:57, Anton Nefedov wrote: > these flags currently affect nothing without CONFIG_FALLOCATE*, so it's > not a bug. Fixing it makes possible to adjust supported zero flag > BDRV_REQ_ALLOCATE regardless of configuration (in the following patch). > > Signed-off-by: Anton Nefedov I'd

Re: [Qemu-block] [PATCH v6 09/11] iotests: change qmp_log filters to expect QMP objects only

2018-12-21 Thread Vladimir Sementsov-Ogievskiy
21.12.2018 12:35, John Snow wrote: > As laid out in the previous commit's message: > > ``` > Several places in iotests deal with serializing objects into JSON > strings, but to add pretty-printing it seems desirable to localize > all of those cases. > > log() seems like a good candidate for that

[Qemu-block] [PATCH] tcmu: Introduce qemu-tcmu utility

2018-12-21 Thread Yaowei Bai
This patch introduces a new utility, qemu-tcmu. Apart from the underlaying protocol it interacts with the world much like qemu-nbd. This patch bases on Fam's version. Qemu-tcmu handles SCSI commands which are passed through userspace from kernel by LIO subsystem using TCMU protocol. Libtcmu is

[Qemu-block] [PATCH v6 09/11] iotests: change qmp_log filters to expect QMP objects only

2018-12-21 Thread John Snow
As laid out in the previous commit's message: ``` Several places in iotests deal with serializing objects into JSON strings, but to add pretty-printing it seems desirable to localize all of those cases. log() seems like a good candidate for that centralized behavior. log() can already serialize

[Qemu-block] [PATCH v6 11/11] iotests: add iotest 236 for testing bitmap merge

2018-12-21 Thread John Snow
New interface, new smoke test. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- tests/qemu-iotests/236 | 161 + tests/qemu-iotests/236.out | 351 + tests/qemu-iotests/group | 1 + 3 files

[Qemu-block] [PATCH v6 10/11] iotests: implement pretty-print for log and qmp_log

2018-12-21 Thread John Snow
If iotests have lines exceeding >998 characters long, git doesn't want to send it plaintext to the list. We can solve this by allowing the iotests to use pretty printed QMP output that we can match against instead. As a bonus, it's much nicer for human eyes too. Signed-off-by: John Snow

[Qemu-block] [PATCH v6 07/11] iotests: add qmp recursive sorting function

2018-12-21 Thread John Snow
Python before 3.6 does not sort dictionaries (including kwargs). Therefore, printing QMP objects involves sorting the keys to have a predictable ordering in the iotests output. This means that iotests output will sometimes show arguments in an order not specified by the test author. Presently, we

[Qemu-block] [PATCH v6 06/11] iotests: add filter_generated_node_ids

2018-12-21 Thread John Snow
To mimic the common filter of the same name, but for the python tests. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemu-iotests/iotests.py

[Qemu-block] [PATCH v6 05/11] iotests.py: don't abort if IMGKEYSECRET is undefined

2018-12-21 Thread John Snow
Instead of using os.environ[], use .get with a default of empty string to match the setup in check to allow us to import the iotests module (for debugging, say) without needing a crafted environment just to import the module. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by:

[Qemu-block] [PATCH v6 08/11] iotests: remove default filters from qmp_log

2018-12-21 Thread John Snow
Several places in iotests deal with serializing objects into JSON strings, but to add pretty-printing it seems desirable to localize all of those cases. log() seems like a good candidate for that centralized behavior. log() can already serialize json objects, but when it does so, it assumes

[Qemu-block] [PATCH v6 04/11] block: remove 'x' prefix from experimental bitmap APIs

2018-12-21 Thread John Snow
The 'x' prefix was added because I was uncertain of the direction we'd take for the libvirt API. With the general approach solidified, I feel comfortable committing to this API for 4.0. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- blockdev.c

[Qemu-block] [PATCH v6 02/11] block/dirty-bitmap: remove assertion from restore

2018-12-21 Thread John Snow
When making a backup of a dirty bitmap (for transactions), we want to restore that backup whether or not the bitmap is enabled. It is perfectly valid to write into bitmaps that are disabled. It is only illegitimate for the guest to have done so. Remove this assertion. Reviewed-by: Eric Blake

[Qemu-block] [PATCH v6 03/11] blockdev: n-ary bitmap merge

2018-12-21 Thread John Snow
Especially outside of transactions, it is helpful to provide all-or-nothing semantics for bitmap merges. This facilitates the coalescing of multiple bitmaps into a single target for the "checkpoint" interpretation when assembling bitmaps that represent arbitrary points in time from component

[Qemu-block] [PATCH v6 00/11] bitmaps: remove x- prefix from QMP api

2018-12-21 Thread John Snow
Fix some outstanding bugs, change the design of an API element, remove the x- prefix to signify stability, and add iotests. V6: - Cover letter touchups - 09: Rewrite qmp_filter to cope with lists or dicts. - 11: small touchups. V5: 002: New bugfix. 003: I forgot to actually capture

[Qemu-block] [PATCH v6 01/11] blockdev: abort transactions in reverse order

2018-12-21 Thread John Snow
Presently, we abort transactions in the same order they were processed in. Bitmap commands, though, attempt to restore backup data structures on abort. That's not valid, they need to be aborted in reverse chronological order. Replace the QSIMPLEQ data structure with a QTAILQ one, so we can