Re: [Qemu-block] [Qemu-devel] [PATCH v2 00/10] cleanup qemu-iotests

2017-09-12 Thread Thomas Huth
On 12.09.2017 16:44, Paolo Bonzini wrote: > The purpose of this series is to separate the "check" sources from > the tests. After these patches, common.config is reduced to simple > shell initialization, and common.rc is only included by the tests. > > Along the way, a lot of dead code is removed

Re: [Qemu-block] [Qemu-devel] [PATCH 04/10] qemu-iotests: cleanup and fix search for programs

2017-09-12 Thread Eric Blake
On 09/12/2017 04:26 PM, Paolo Bonzini wrote: > > >>> +then >>> +if [ -x "$build_iotests/qemu" ]; then >>> +export QEMU_PROG="$build_iotests/qemu" >>> +elif [ -x "$build_root/$arch-softmmu/qemu-system-$arch" ]; then >>> +export QEMU_PROG="$build_root/$arch-softmmu/qemu-syst

Re: [Qemu-block] [Qemu-devel] [PATCH 10/10] qemu-iotests: merge "check" and "common"

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > "check" is full of qemu-iotests--specific details. Separating it > from "common" does not make much sense anymore. > > Signed-off-by: Paolo Bonzini > --- > tests/qemu-iotests/check | 533 +++- > tests/qemu-i

Re: [Qemu-block] [Qemu-devel] [PATCH 09/10] qemu-iotests: get rid of $iam

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > The variable is almost unused, and one of the two uses is actually > uninitialized. > > Signed-off-by: Paolo Bonzini > --- > tests/qemu-iotests/check | 5 + > tests/qemu-iotests/common.rc | 2 +- > 2 files changed, 2 insertions(+), 5 deletio

Re: [Qemu-block] [Qemu-devel] [PATCH 08/10] qemu-iotests: fix uninitialized variable

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > The variable is used in "common" but defined only after the file > is sourced. > > Signed-off-by: Paolo Bonzini > --- > tests/qemu-iotests/check | 2 -- > tests/qemu-iotests/common | 2 ++ > 2 files changed, 2 insertions(+), 2 deletions(-) > Revi

Re: [Qemu-block] [Qemu-devel] [PATCH 04/10] qemu-iotests: cleanup and fix search for programs

2017-09-12 Thread Paolo Bonzini
> > +then > > +if [ -x "$build_iotests/qemu" ]; then > > +export QEMU_PROG="$build_iotests/qemu" > > +elif [ -x "$build_root/$arch-softmmu/qemu-system-$arch" ]; then > > +export QEMU_PROG="$build_root/$arch-softmmu/qemu-system-$arch" > > +else > > +pushd "$buil

Re: [Qemu-block] [Qemu-devel] [PATCH 07/10] qemu-iotests: disintegrate more parts of common.config

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > Split "check" parts from tests part. > > For the directory setup, the actual computation of directories goes > in "check", while the sanity checks go in the tests. > > Signed-off-by: Paolo Bonzini > --- > tests/qemu-iotests/common| 24 +

Re: [Qemu-block] [Qemu-devel] [PATCH 06/10] qemu-iotests: do not include common.rc in "check"

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > It only provides functions used by the test programs. > > Signed-off-by: Paolo Bonzini > --- > tests/qemu-iotests/check | 6 -- > tests/qemu-iotests/common.rc | 13 + > 2 files changed, 5 insertions(+), 14 deletions(-) > Revie

Re: [Qemu-block] [Qemu-devel] [PATCH 05/10] qemu-iotests: limit non-_PROG-suffixed variables to common.rc

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > These are never used by "check", with one exception that does not need > $QEMU_OPTIONS. Keep them in common.rc, which will be soon included only > by the tests. > > Signed-off-by: Paolo Bonzini > --- > tests/qemu-iotests/039.out | 10 +++--- >

Re: [Qemu-block] [Qemu-devel] [PATCH 04/10] qemu-iotests: cleanup and fix search for programs

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > Instead of ./check failing when a binary is missing, we try each test > case now and each one fails with tons of test case diffs. Also, all the > variables were initialized by "check" prior to "common" being sourced, > and then (uselessly) checked for

Re: [Qemu-block] [Qemu-devel] [PATCH 03/10] qemu-iotests: move "check" code out of common.rc

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > Some functions in common.rc are never used by the tests. Move > them out of that file and into common, which is already included > only by "check". > > Code that actually *is* common to "check" and tests can be placed in > common.config. > > Signed-

Re: [Qemu-block] [Qemu-devel] [PATCH 02/10] qemu-iotests: get rid of AWK_PROG

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > Signed-off-by: Paolo Bonzini > --- > tests/qemu-iotests/check | 4 ++-- > tests/qemu-iotests/common| 2 +- > tests/qemu-iotests/common.config | 3 --- > 3 files changed, 3 insertions(+), 6 deletions(-) Reviewed-by: Eric Blake -- E

[Qemu-block] [PATCH v7 19/20] dirty-bitmap: Switch bdrv_set_dirty() to bytes

2017-09-12 Thread Eric Blake
Both callers already had bytes available, but were scaling to sectors. Move the scaling to internal code. In the case of bdrv_aligned_pwritev(), we are now passing the exact offset rather than a rounded sector-aligned value, but that's okay as long as dirty bitmap widens start/bytes to granularit

[Qemu-block] [PATCH v7 18/20] qcow2: Switch store_bitmap_data() to byte-based iteration

2017-09-12 Thread Eric Blake
Now that we have adjusted the majority of the calls this function makes to be byte-based, it is easier to read the code if it makes passes over the image using bytes rather than sectors. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy --- v7: rebase t

Re: [Qemu-block] [Qemu-devel] [PATCH 01/10] qemu-iotests: remove dead code

2017-09-12 Thread Eric Blake
On 09/12/2017 09:44 AM, Paolo Bonzini wrote: > This includes shell function, shell variables and command line options > (randomize.awk does not exist). > > Signed-off-by: Paolo Bonzini > --- > tests/qemu-iotests/check | 28 - > tests/qemu-iotests/common| 23 --

[Qemu-block] [PATCH v7 17/20] qcow2: Switch load_bitmap_data() to byte-based iteration

2017-09-12 Thread Eric Blake
Now that we have adjusted the majority of the calls this function makes to be byte-based, it is easier to read the code if it makes passes over the image using bytes rather than sectors. Signed-off-by: Eric Blake Reviewed-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy --- v5: no chang

[Qemu-block] [PATCH v7 15/20] mirror: Switch mirror_dirty_init() to byte-based iteration

2017-09-12 Thread Eric Blake
Now that we have adjusted the majority of the calls this function makes to be byte-based, it is easier to read the code if it makes passes over the image using bytes rather than sectors. Signed-off-by: Eric Blake Reviewed-by: John Snow --- v6: no change v5: rebase to earlier changes v2-v4: no c

[Qemu-block] [PATCH v7 16/20] qcow2: Switch qcow2_measure() to byte-based iteration

2017-09-12 Thread Eric Blake
This is new code, but it is easier to read if it makes passes over the image using bytes rather than sectors (and will get easier in the future when bdrv_get_block_status is converted to byte-based). Signed-off-by: Eric Blake Reviewed-by: John Snow --- v7: tweak constant given to MIN (no semant

[Qemu-block] [PATCH v7 14/20] dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes

2017-09-12 Thread Eric Blake
Some of the callers were already scaling bytes to sectors; others can be easily converted to pass byte offsets, all in our shift towards a consistent byte interface everywhere. Making the change will also make it easier to write the hold-out callers to use byte rather than sectors for their iterat

[Qemu-block] [PATCH v7 20/20] dirty-bitmap: Convert internal hbitmap size/granularity

2017-09-12 Thread Eric Blake
Now that all callers are using byte-based interfaces, there's no reason for our internal hbitmap to remain with sector-based granularity. It also simplifies our internal scaling, since we already know that hbitmap widens requests out to granularity boundaries. Signed-off-by: Eric Blake Reviewed-

[Qemu-block] [PATCH v7 13/20] dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes

2017-09-12 Thread Eric Blake
Half the callers were already scaling bytes to sectors; the other half can eventually be simplified to use byte iteration. Both callers were already using the result as a bool, so make that explicit. Making the change also makes it easier for a future dirty-bitmap patch to offload scaling over to

[Qemu-block] [PATCH v7 10/20] dirty-bitmap: Set iterator start by offset, not sector

2017-09-12 Thread Eric Blake
All callers to bdrv_dirty_iter_new() passed 0 for their initial starting point, drop that parameter. Most callers to bdrv_set_dirty_iter() were scaling a byte offset to a sector number; the exception qcow2-bitmap will be converted later to use byte rather than sector iteration. Move the scaling t

[Qemu-block] [PATCH v7 11/20] dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset

2017-09-12 Thread Eric Blake
Thanks to recent cleanups, most callers were scaling a return value of sectors into bytes (the exception, in qcow2-bitmap, will be converted to byte-based iteration later). Update the interface to do the scaling internally instead. In qcow2-bitmap, the code was specifically checking for an error

[Qemu-block] [PATCH v7 07/20] dirty-bitmap: Track bitmap size by bytes

2017-09-12 Thread Eric Blake
We are still using an internal hbitmap that tracks a size in sectors, with the granularity scaled down accordingly, because it lets us use a shortcut for our iterators which are currently sector-based. But there's no reason we can't track the dirty bitmap size in bytes, since it is (mostly) an inte

[Qemu-block] [PATCH v7 09/20] qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based

2017-09-12 Thread Eric Blake
We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Change the qcow2 bitmap helper function sectors_covered_by_bitmap_cluster(), renaming it to bytes_covered_by_bitmap_cluster() in the process. Signed-off-by: Eric Blake Reviewed-by: John Sn

[Qemu-block] [PATCH v7 12/20] dirty-bitmap: Change bdrv_get_dirty_count() to report bytes

2017-09-12 Thread Eric Blake
Thanks to recent cleanups, all callers were scaling a return value of sectors into bytes; do the scaling internally instead. Signed-off-by: Eric Blake --- v7: fix one more trace caller [Kevin] v4-v6: no change v3: no change, add R-b v2: no change --- block/dirty-bitmap.c | 4 ++-- block/mirror

[Qemu-block] [PATCH v7 05/20] dirty-bitmap: Check for size query failure during truncate

2017-09-12 Thread Eric Blake
We've previously fixed several places where we failed to account for possible errors from bdrv_nb_sectors(). Fix another one by making bdrv_dirty_bitmap_truncate() report the error rather then silently resizing bitmaps to -1. Then adjust the sole caller bdrv_truncate() to both reduce the likeliho

[Qemu-block] [PATCH v7 04/20] dirty-bitmap: Drop unused functions

2017-09-12 Thread Eric Blake
We had several functions that no one is currently using, and which use sector-based interfaces. I'm trying to convert towards byte-based interfaces, so it's easier to just drop the unused functions: bdrv_dirty_bitmap_get_meta bdrv_dirty_bitmap_get_meta_locked bdrv_dirty_bitmap_reset_meta bdrv_dir

[Qemu-block] [PATCH v7 08/20] dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes

2017-09-12 Thread Eric Blake
Right now, the dirty-bitmap code exposes the fact that we use a scale of sector granularity in the underlying hbitmap to anything that wants to serialize a dirty bitmap. It's nicer to uniformly expose bytes as our dirty-bitmap interface, matching the previous change to bitmap size. The only calle

[Qemu-block] [PATCH v7 03/20] qcow2: Ensure bitmap serialization is aligned

2017-09-12 Thread Eric Blake
When subdividing a bitmap serialization, the code in hbitmap.c enforces that start/count parameters are aligned (except that count can end early at end-of-bitmap). We exposed this required alignment through bdrv_dirty_bitmap_serialization_align(), but forgot to actually check that we comply with i

[Qemu-block] [PATCH v7 06/20] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes

2017-09-12 Thread Eric Blake
We're already reporting bytes for bdrv_dirty_bitmap_granularity(); mixing bytes and sectors in our return values is a recipe for confusion. A later cleanup will convert dirty bitmap internals to be entirely byte-based, but in the meantime, we should report the bitmap size in bytes. The only exter

[Qemu-block] [PATCH v7 01/20] block: Make bdrv_img_create() size selection easier to read

2017-09-12 Thread Eric Blake
All callers of bdrv_img_create() pass in a size, or -1 to read the size from the backing file. We then set that size as the QemuOpt default, which means we will reuse that default rather than the final parameter to qemu_opt_get_size() several lines later. But it is rather confusing to read subseq

[Qemu-block] [PATCH v7 02/20] hbitmap: Rename serialization_granularity to serialization_align

2017-09-12 Thread Eric Blake
The only client of hbitmap_serialization_granularity() is dirty-bitmap's bdrv_dirty_bitmap_serialization_align(). Keeping the two names consistent is worthwhile, and the shorter name is more representative of what the function returns (the required alignment to be used for start/count of other ser

[Qemu-block] [PATCH v7 00/20] make dirty-bitmap byte-based

2017-09-12 Thread Eric Blake
There are patches floating around to add NBD_CMD_BLOCK_STATUS, but NBD wants to report status on byte granularity (even if the reporting will probably be naturally aligned to sectors or even much higher levels). I've therefore started the task of converting our block status code to report at a byt

Re: [Qemu-block] [Qemu-devel] [PATCH v6 05/18] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes

2017-09-12 Thread Eric Blake
On 09/08/2017 09:04 AM, Eric Blake wrote: >>> void bdrv_dirty_bitmap_truncate(BlockDriverState *bs) >>> { >>> BdrvDirtyBitmap *bitmap; >>> -uint64_t size = bdrv_nb_sectors(bs); >>> +int64_t size = bdrv_getlength(bs); >>> >>> +assert(size >= 0); >> >> How can you assert that ther

Re: [Qemu-block] [Qemu-devel] [PATCH 02/10] qemu-iotests: get rid of AWK_PROG

2017-09-12 Thread Paolo Bonzini
On 12/09/2017 18:30, Philippe Mathieu-Daudé wrote: > Hi Paolo, > > Hmm did you just resend your v1? Except the cover the patches don't have > "v2" and I remember reviewing this one (also Eric Blake replied with his > R-b). I didn't include the R-bs, sorry (most of the later patches had non-trivia

Re: [Qemu-block] [Qemu-devel] [PATCH 02/10] qemu-iotests: get rid of AWK_PROG

2017-09-12 Thread Philippe Mathieu-Daudé
Hi Paolo, Hmm did you just resend your v1? Except the cover the patches don't have "v2" and I remember reviewing this one (also Eric Blake replied with his R-b). On 09/12/2017 11:44 AM, Paolo Bonzini wrote: Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/check | 4 ++-- tests

Re: [Qemu-block] [Qemu-devel] [PATCH 2/3] iotests: use -ccw on s390x for 051

2017-09-12 Thread Kevin Wolf
Am 12.09.2017 um 18:05 hat Cornelia Huck geschrieben: > On Fri, 8 Sep 2017 17:55:29 +0200 > Thomas Huth wrote: > > > On 08.09.2017 13:54, Kevin Wolf wrote: > > > Am 08.09.2017 um 13:24 hat Cornelia Huck geschrieben: > > >> On Fri, 8 Sep 2017 13:04:25 +0200 > > >> Kevin Wolf wrote: > > >> > >

Re: [Qemu-block] [Qemu-devel] [PATCH 2/3] iotests: use -ccw on s390x for 051

2017-09-12 Thread Cornelia Huck
On Fri, 8 Sep 2017 17:55:29 +0200 Thomas Huth wrote: > On 08.09.2017 13:54, Kevin Wolf wrote: > > Am 08.09.2017 um 13:24 hat Cornelia Huck geschrieben: > >> On Fri, 8 Sep 2017 13:04:25 +0200 > >> Kevin Wolf wrote: > >> > >>> Am 05.09.2017 um 17:16 hat Cornelia Huck geschrieben: > The

[Qemu-block] [PATCH 05/10] qemu-iotests: limit non-_PROG-suffixed variables to common.rc

2017-09-12 Thread Paolo Bonzini
These are never used by "check", with one exception that does not need $QEMU_OPTIONS. Keep them in common.rc, which will be soon included only by the tests. Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/039.out | 10 +++--- tests/qemu-iotests/061.out | 4 +-- tests/qemu-iotes

[Qemu-block] [PATCH 09/10] qemu-iotests: get rid of $iam

2017-09-12 Thread Paolo Bonzini
The variable is almost unused, and one of the two uses is actually uninitialized. Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/check | 5 + tests/qemu-iotests/common.rc | 2 +- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/tests/qemu-iotests/check b/tests/qemu-iot

[Qemu-block] [PATCH 10/10] qemu-iotests: merge "check" and "common"

2017-09-12 Thread Paolo Bonzini
"check" is full of qemu-iotests--specific details. Separating it from "common" does not make much sense anymore. Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/check | 533 +++- tests/qemu-iotests/common | 552 ---

[Qemu-block] [PATCH 06/10] qemu-iotests: do not include common.rc in "check"

2017-09-12 Thread Paolo Bonzini
It only provides functions used by the test programs. Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/check | 6 -- tests/qemu-iotests/common.rc | 13 + 2 files changed, 5 insertions(+), 14 deletions(-) diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check in

[Qemu-block] [PATCH 08/10] qemu-iotests: fix uninitialized variable

2017-09-12 Thread Paolo Bonzini
The variable is used in "common" but defined only after the file is sourced. Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/check | 2 -- tests/qemu-iotests/common | 2 ++ 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check ind

[Qemu-block] [PATCH 07/10] qemu-iotests: disintegrate more parts of common.config

2017-09-12 Thread Paolo Bonzini
Split "check" parts from tests part. For the directory setup, the actual computation of directories goes in "check", while the sanity checks go in the tests. Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/common| 24 tests/qemu-iotests/common.config | 49 --

[Qemu-block] [PATCH 01/10] qemu-iotests: remove dead code

2017-09-12 Thread Paolo Bonzini
This includes shell function, shell variables and command line options (randomize.awk does not exist). Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/check | 28 - tests/qemu-iotests/common| 23 -- tests/qemu-iotests/common.config | 26 ---

[Qemu-block] [PATCH 03/10] qemu-iotests: move "check" code out of common.rc

2017-09-12 Thread Paolo Bonzini
Some functions in common.rc are never used by the tests. Move them out of that file and into common, which is already included only by "check". Code that actually *is* common to "check" and tests can be placed in common.config. Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/common

[Qemu-block] [PATCH 04/10] qemu-iotests: cleanup and fix search for programs

2017-09-12 Thread Paolo Bonzini
Instead of ./check failing when a binary is missing, we try each test case now and each one fails with tons of test case diffs. Also, all the variables were initialized by "check" prior to "common" being sourced, and then (uselessly) checked for emptiness again in "check". Centralize the search f

[Qemu-block] [PATCH v2 00/10] cleanup qemu-iotests

2017-09-12 Thread Paolo Bonzini
The purpose of this series is to separate the "check" sources from the tests. After these patches, common.config is reduced to simple shell initialization, and common.rc is only included by the tests. Along the way, a lot of dead code is removed too. In v2, the following patches: qemu-iotests

[Qemu-block] [PATCH 02/10] qemu-iotests: get rid of AWK_PROG

2017-09-12 Thread Paolo Bonzini
Signed-off-by: Paolo Bonzini --- tests/qemu-iotests/check | 4 ++-- tests/qemu-iotests/common| 2 +- tests/qemu-iotests/common.config | 3 --- 3 files changed, 3 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check index 4a6ed67b42..5c0d0

Re: [Qemu-block] [PATCH v7 31/38] libqtest: Merge qtest_clock_*() with clock_*()

2017-09-12 Thread Eric Blake
On 09/12/2017 05:45 AM, Thomas Huth wrote: > On 11.09.2017 19:20, Eric Blake wrote: >> Maintaining two layers of libqtest APIs, one that takes an explicit >> QTestState object, and the other that uses the implicit global_qtest, >> is annoying. In the interest of getting rid of global implicit >> s

Re: [Qemu-block] [PATCH v7 28/38] libqtest: Add qtest_[v]startf()

2017-09-12 Thread Eric Blake
On 09/12/2017 05:14 AM, Thomas Huth wrote: > On 11.09.2017 19:20, Eric Blake wrote: >> We have several callers that were formatting the argument strings >> themselves; consolidate this effort by adding new convenience >> functions directly in libqtest, and update all call-sites that >> can benefit

Re: [Qemu-block] [PATCH v7 10/38] libqos: Move/rename qpci_unplug_acpi_device_test() to pci.c

2017-09-12 Thread Eric Blake
On 09/12/2017 02:29 AM, Thomas Huth wrote: > On 11.09.2017 19:19, Eric Blake wrote: >> Commit 2f8b2767 originally added qpci_plug_device_test() and >> qpci_unplug_acpi_device_test() as a pair, both in pci-pc.c. >> Later, commit cf716b31 moved one half of the pair to pci.c >> when adding PPC64 suppo

Re: [Qemu-block] [PATCH v7 09/38] libqos: Track QTestState with QVirtioBus

2017-09-12 Thread Eric Blake
On 09/12/2017 02:21 AM, Thomas Huth wrote: > On 11.09.2017 19:19, Eric Blake wrote: >> When initializing a QVirtioDevice (which always has an associated >> QVirtioBus), we want to track which QTestState to use for all >> I/O processed through that bus and device. Copy the paradigm >> used for QPCI

Re: [Qemu-block] [PATCH 07/12] qemu-iotests: disintegrate more parts of common.config

2017-09-12 Thread Paolo Bonzini
On 12/09/2017 14:57, Kevin Wolf wrote: > Am 12.09.2017 um 14:31 hat Paolo Bonzini geschrieben: >> On 12/09/2017 12:40, Kevin Wolf wrote: >>> Am 09.08.2017 um 23:55 hat Paolo Bonzini geschrieben: Split "check" parts from tests part. For the directory setup, the actual computation of d

Re: [Qemu-block] [PATCH 07/12] qemu-iotests: disintegrate more parts of common.config

2017-09-12 Thread Kevin Wolf
Am 12.09.2017 um 14:31 hat Paolo Bonzini geschrieben: > On 12/09/2017 12:40, Kevin Wolf wrote: > > Am 09.08.2017 um 23:55 hat Paolo Bonzini geschrieben: > >> Split "check" parts from tests part. > >> > >> For the directory setup, the actual computation of directories goes > >> in "check", while the

Re: [Qemu-block] [PATCH 04/12] qemu-iotests: limit non-_PROG-suffixed variables to common.rc

2017-09-12 Thread Paolo Bonzini
On 12/09/2017 14:51, Kevin Wolf wrote: > Am 12.09.2017 um 14:28 hat Paolo Bonzini geschrieben: >> On 12/09/2017 12:31, Kevin Wolf wrote: >>> Hm, does this mean that instead of ./check failing when a binary is >>> missing, we try each test case now and each one fails with the same >>> error message?

Re: [Qemu-block] [PATCH 04/12] qemu-iotests: limit non-_PROG-suffixed variables to common.rc

2017-09-12 Thread Kevin Wolf
Am 12.09.2017 um 14:28 hat Paolo Bonzini geschrieben: > On 12/09/2017 12:31, Kevin Wolf wrote: > > Hm, does this mean that instead of ./check failing when a binary is > > missing, we try each test case now and each one fails with the same > > error message? > > > > *tries it out* > > > > Okay, it

Re: [Qemu-block] [PATCH 07/12] qemu-iotests: disintegrate more parts of common.config

2017-09-12 Thread Paolo Bonzini
On 12/09/2017 12:40, Kevin Wolf wrote: > Am 09.08.2017 um 23:55 hat Paolo Bonzini geschrieben: >> Split "check" parts from tests part. >> >> For the directory setup, the actual computation of directories goes >> in "check", while the sanity checks go in the tests. >> >> Signed-off-by: Paolo Bonzini

Re: [Qemu-block] [PATCH 04/12] qemu-iotests: limit non-_PROG-suffixed variables to common.rc

2017-09-12 Thread Paolo Bonzini
On 12/09/2017 12:31, Kevin Wolf wrote: > Hm, does this mean that instead of ./check failing when a binary is > missing, we try each test case now and each one fails with the same > error message? > > *tries it out* > > Okay, it's already broken today because the strings are never empty but > cont

Re: [Qemu-block] [PATCH v3 5/7] block: convert crypto driver to bdrv_co_preadv|pwritev

2017-09-12 Thread Daniel P. Berrange
On Tue, Sep 12, 2017 at 12:28:53PM +0100, Daniel P. Berrange wrote: > Make the crypto driver implement the bdrv_co_preadv|pwritev > callbacks, and also use bdrv_co_preadv|pwritev for I/O > with the protocol driver beneath. > > Signed-off-by: Daniel P. Berrange > --- > block/crypto.c | 104 > +++

[Qemu-block] [PATCH v3 7/7] block: support passthrough of BDRV_REQ_FUA in crypto driver

2017-09-12 Thread Daniel P. Berrange
The BDRV_REQ_FUA flag can trivially be allowed in the crypt driver as a passthrough to the underlying block driver. Signed-off-by: Daniel P. Berrange --- block/crypto.c | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/block/crypto.c b/block/crypto.c index 1f77095336..d1

[Qemu-block] [PATCH v3 1/7] block: use 1 MB bounce buffers for crypto instead of 16KB

2017-09-12 Thread Daniel P. Berrange
Using 16KB bounce buffers creates a significant performance penalty for I/O to encrypted volumes on storage which high I/O latency (rotating rust & network drives), because it triggers lots of fairly small I/O operations. On tests with rotating rust, and cache=none|directsync, write speed increase

[Qemu-block] [PATCH v3 6/7] block: convert qcrypto_block_encrypt|decrypt to take bytes offset

2017-09-12 Thread Daniel P. Berrange
Instead of sector offset, take the bytes offset when encrypting or decrypting data. Signed-off-by: Daniel P. Berrange --- block/crypto.c | 12 block/qcow.c | 7 +-- block/qcow2-cluster.c | 8 +++- block/qcow2.c | 4 ++-- crypto/block-luks.c

[Qemu-block] [PATCH v3 5/7] block: convert crypto driver to bdrv_co_preadv|pwritev

2017-09-12 Thread Daniel P. Berrange
Make the crypto driver implement the bdrv_co_preadv|pwritev callbacks, and also use bdrv_co_preadv|pwritev for I/O with the protocol driver beneath. Signed-off-by: Daniel P. Berrange --- block/crypto.c | 104 +++-- 1 file changed, 56 insertions

[Qemu-block] [PATCH v3 4/7] block: don't use constant 512 as sector size in crypto driver

2017-09-12 Thread Daniel P. Berrange
Use the qcrypto_block_get_sector_size() value in the block crypto driver instead of hardcoding 512 as the sector size. Signed-off-by: Daniel P. Berrange --- block/crypto.c | 34 ++ 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/block/crypto.c b/bl

[Qemu-block] [PATCH v3 2/7] crypto: expose encryption sector size in APIs

2017-09-12 Thread Daniel P. Berrange
While current encryption schemes all have a fixed sector size of 512 bytes, this is not guaranteed to be the case in future. Expose the sector size in the APIs so the block layer can remove assumptions about fixed 512 byte sectors. Signed-off-by: Daniel P. Berrange --- crypto/block-luks.c|

[Qemu-block] [PATCH v3 3/7] block: fix data type casting for crypto payload offset

2017-09-12 Thread Daniel P. Berrange
The crypto APIs report the offset of the data payload as an uint64_t type, but the block driver is casting to size_t or ssize_t which will potentially truncate. Most of the block APIs use int64_t for offsets meanwhile, so even if using uint64_t in the crypto block driver we are still at risk of tr

[Qemu-block] [PATCH v3 0/7] Misc improvements to crypto block driver

2017-09-12 Thread Daniel P. Berrange
This is a followup to v1: https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg00781.html v2: https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg06464.html This collection of patches first improves the performance of the crypto block driver and then does various cleanups to improve

Re: [Qemu-block] [PATCH v7 33/38] libqtest: Merge qtest_{in, out}[bwl]() with {in, out}[bwl]()

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:20, Eric Blake wrote: > Maintaining two layers of libqtest APIs, one that takes an explicit > QTestState object, and the other that uses the implicit global_qtest, > is annoying. In the interest of getting rid of global implicit > state and having less code to maintain, merge: >

Re: [Qemu-block] [PATCH v7 32/38] libqtest: Merge qtest_irq*() with irq*()

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:20, Eric Blake wrote: > Maintaining two layers of libqtest APIs, one that takes an explicit > QTestState object, and the other that uses the implicit global_qtest, > is annoying. In the interest of getting rid of global implicit > state and having less code to maintain, merge: >

Re: [Qemu-block] [PATCH v7 31/38] libqtest: Merge qtest_clock_*() with clock_*()

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:20, Eric Blake wrote: > Maintaining two layers of libqtest APIs, one that takes an explicit > QTestState object, and the other that uses the implicit global_qtest, > is annoying. In the interest of getting rid of global implicit > state and having less code to maintain, merge: >

Re: [Qemu-block] [PATCH 07/12] qemu-iotests: disintegrate more parts of common.config

2017-09-12 Thread Kevin Wolf
Am 09.08.2017 um 23:55 hat Paolo Bonzini geschrieben: > Split "check" parts from tests part. > > For the directory setup, the actual computation of directories goes > in "check", while the sanity checks go in the tests. > > Signed-off-by: Paolo Bonzini Same comment as for patch 1, we may want t

Re: [Qemu-block] [PATCH v7 29/38] libqtest: Merge qtest_init() into qtest_start()

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:20, Eric Blake wrote: > Remove the trivial wrapper qtest_init(), and change qtest_start() > to no longer implicitly set global_qtest, to make it obvious in the > rest of the testsuite where we are still relying on global_qtest. > Everything now uses qtest_start() (and friends) and

Re: [Qemu-block] [PATCH 04/12] qemu-iotests: limit non-_PROG-suffixed variables to common.rc

2017-09-12 Thread Kevin Wolf
Am 09.08.2017 um 23:55 hat Paolo Bonzini geschrieben: > These are never used by "check", with one exception that does not need > $QEMU_OPTIONS. Keep them in common.rc, which will be soon included only > by the tests. > > Signed-off-by: Paolo Bonzini > diff --git a/tests/qemu-iotests/common b/te

Re: [Qemu-block] [PATCH v7 28/38] libqtest: Add qtest_[v]startf()

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:20, Eric Blake wrote: > We have several callers that were formatting the argument strings > themselves; consolidate this effort by adding new convenience > functions directly in libqtest, and update all call-sites that > can benefit from it. [...] > diff --git a/tests/libqtest.c b/

Re: [Qemu-block] [PATCH v2 3/4] block: convert crypto driver to bdrv_co_preadv|pwritev

2017-09-12 Thread Daniel P. Berrange
On Thu, Aug 31, 2017 at 10:08:00AM -0500, Eric Blake wrote: > On 08/31/2017 06:05 AM, Daniel P. Berrange wrote: > > Make the crypto driver implement the bdrv_co_preadv|pwritev > > callbacks, and also use bdrv_co_preadv|pwritev for I/O > > with the protocol driver beneath. > > > > Signed-off-by: D

Re: [Qemu-block] [Qemu-devel] [PATCH 01/12] qemu-iotests: remove dead code

2017-09-12 Thread Kevin Wolf
Am 10.08.2017 um 00:43 hat Paolo Bonzini geschrieben: > > > - Original Message - > > From: "Eric Blake" > > To: "Paolo Bonzini" , qemu-de...@nongnu.org > > Cc: kw...@redhat.com, qemu-block@nongnu.org > > Sent: Thursday, August 10, 2017 12:18:54 AM > > Subject: Re: [Qemu-devel] [PATCH 01/

Re: [Qemu-block] loading bitmaps in invalidate_cache fails

2017-09-12 Thread Kevin Wolf
Am 11.09.2017 um 18:51 hat Vladimir Sementsov-Ogievskiy geschrieben: > Hi Kevin! > > I'm confused with relations of permissions and invalidation, can you please > help? > > Now dirty bitmaps are loaded in invalidate_cache. Here is a problem with > migration: > > 1. destination starts (inactive)

Re: [Qemu-block] [PATCH v7 26/38] libqtest: Merge qtest_end() into qtest_quit()

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:20, Eric Blake wrote: > Rather than have two similar shutdown functions, where one requires > the use of global_qtest in the header, it is better to have a single > shutdown function that still takes care of cleaning up global_qtest > if it is set. All callers are updated. > > Si

Re: [Qemu-block] [PATCH v7 16/38] libqos: Use explicit QTestState for ahci operations

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:20, Eric Blake wrote: > Drop one more client of global_qtest by teaching all ahci test > functionality to pass in an explicit QTestState. The state was > already available, so no callers had to be adjusted. > > Signed-off-by: Eric Blake Reviewed-by: Thomas Huth

Re: [Qemu-block] [PATCH v7 13/38] libqos: Use explicit QTestState for fw_cfg operations

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:19, Eric Blake wrote: > Drop one more client of global_qtest by teaching all fw_cfg test > functionality (invoked through alloc-pc) to pass in an explicit > QTestState, adjusting all callers. In particular, fw_cfg-test > had to reorder things to create the test state prior to crea

Re: [Qemu-block] [PATCH v7 12/38] libqos: Use explicit QTestState for virtio operations

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:19, Eric Blake wrote: > Now that QVirtioDevice and QVirtQueue point back to QVirtioBus, > we can reuse the explicit QTestState stored there rather than > relying on implicit global_qtest. We also have to pass QTestState > through a few functions that can't trace back through > QVi

Re: [Qemu-block] [PATCH v7 10/38] libqos: Move/rename qpci_unplug_acpi_device_test() to pci.c

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:19, Eric Blake wrote: > Commit 2f8b2767 originally added qpci_plug_device_test() and > qpci_unplug_acpi_device_test() as a pair, both in pci-pc.c. > Later, commit cf716b31 moved one half of the pair to pci.c > when adding PPC64 support. Keep the implementations of the > two functi

Re: [Qemu-block] [PATCH v7 09/38] libqos: Track QTestState with QVirtioBus

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:19, Eric Blake wrote: > When initializing a QVirtioDevice (which always has an associated > QVirtioBus), we want to track which QTestState to use for all > I/O processed through that bus and device. Copy the paradigm > used for QPCIBus, and track the test state at the bus level; t

Re: [Qemu-block] [PATCH v7 08/38] libqos: Track QTestState with QPCIBus

2017-09-12 Thread Thomas Huth
On 11.09.2017 19:19, Eric Blake wrote: > When initializing a QPCIBus, track which QTestState the bus is > associated with (so that a later patch can then explicitly use > that test state for all communication on the bus, rather than > blindly relying on global_qtest). Update the initialization > f