Re: [PATCH v2 4/9] tests/acceptance/boot_linux: Expand SD card image to power of 2

2020-07-13 Thread Cleber Rosa
On Mon, Jul 13, 2020 at 08:32:04PM +0200, Philippe Mathieu-Daudé wrote: > In few commits we won't allow SD card images with invalid size > (not aligned to a power of 2). Prepare the tests: add the > pow2ceil() and image_pow2ceil_expand() methods and resize the > images (expanding) of the tests

Re: [PATCH v2 3/9] tests/acceptance/boot_linux: Tag tests using a SD card with 'device:sd'

2020-07-13 Thread Cleber Rosa
On Mon, Jul 13, 2020 at 08:32:03PM +0200, Philippe Mathieu-Daudé wrote: > Avocado tags are handy to automatically select tests matching > the tags. Since these tests use a SD card, tag them. > > We can run all the tests using a SD card at once with: > > $ avocado --show=app run -t u-boot

[PATCH v10 01/10] qcow2: Fix capitalization of header extension constant.

2020-07-13 Thread Andrey Shinkevich
Make the capitalization of the hexadecimal numbers consistent for the QCOW2 header extension constants in docs/interop/qcow2.txt. Suggested-by: Eric Blake Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.c | 2 +- docs/interop/qcow2.txt | 2

[PATCH v10 05/10] qcow2_format.py: Dump bitmap directory information

2020-07-13 Thread Andrey Shinkevich
Read and dump entries from the bitmap directory of QCOW2 image. Header extension: magic 0x23852875 (Bitmaps) ... Bitmap name bitmap-1 bitmap_table_offset 0xf bitmap_table_size 1 flags 0x2 (['auto']) type

[PATCH v10 03/10] qcow2_format.py: change Qcow2BitmapExt initialization method

2020-07-13 Thread Andrey Shinkevich
There are two ways to initialize a class derived from Qcow2Struct: 1. Pass a block of binary data to the constructor. 2. Pass the file descriptor to allow reading the file from constructor. Let's change the Qcow2BitmapExt initialization method from 1 to 2 to support a scattered reading in the

[PATCH v10 06/10] qcow2_format.py: pass cluster size to substructures

2020-07-13 Thread Andrey Shinkevich
The cluster size of an image is the QcowHeader class member and may be obtained by dependent extension structures such as Qcow2BitmapExt for further bitmap table details print. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 18 +++--- 1 file changed, 11

[PATCH v10 00/10] iotests: Dump QCOW2 dirty bitmaps metadata

2020-07-13 Thread Andrey Shinkevich
Add dirty bitmap information to QCOW2 metadata dump in the qcow2_format.py. v10: 01: The modification of the 291.out was removed. 02: Qcow2BitmapExt modified to take control over file cursor position while reading the bitmap extension data from the file (suggested by Vladimir). 03:

[PATCH v10 04/10] qcow2_format.py: dump bitmap flags in human readable way.

2020-07-13 Thread Andrey Shinkevich
Introduce the class BitmapFlags that parses a bitmap flags mask. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 16 1 file changed, 16 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py

[PATCH v10 02/10] qcow2_format.py: make printable data an extension class member

2020-07-13 Thread Andrey Shinkevich
Let us differ binary data type from string one for the extension data variable and keep the string as the QcowHeaderExtension class member. Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 14

[PATCH v10 10/10] qcow2_format.py: support dumping metadata in JSON format

2020-07-13 Thread Andrey Shinkevich
Implementation of dumping QCOW2 image metadata. The sample output: { "Header_extensions": [ { "name": "Feature table", "magic": 1745090647, "length": 192, "data_str": "" }, { "name": "Bitmaps",

[PATCH v10 08/10] qcow2.py: Introduce '-j' key to dump in JSON format

2020-07-13 Thread Andrey Shinkevich
Add the command key to the qcow2.py arguments list to dump QCOW2 metadata in JSON format. Here is the suggested way to do that. The implementation of the dump in JSON format is in the patch that follows. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2.py| 19

[PATCH v10 07/10] qcow2_format.py: Dump bitmap table serialized entries

2020-07-13 Thread Andrey Shinkevich
Add bitmap table information to the QCOW2 metadata dump. Bitmap name bitmap-1 ... Bitmap table typeoffset size 0 serialized 4718592 65536 1 serialized 4294967296 65536 2

[PATCH v10 09/10] qcow2_format.py: collect fields to dump in JSON format

2020-07-13 Thread Andrey Shinkevich
As __dict__ is being extended with class members we do not want to print, make a light copy of the initial __dict__ and extend the copy by adding lists we have to print in the JSON output. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 4 1 file changed, 4

Re: [PATCH v2 7/9] hw/sd/sdcard: Do not allow invalid SD card sizes

2020-07-13 Thread Peter Maydell
On Mon, 13 Jul 2020 at 19:32, Philippe Mathieu-Daudé wrote: > > QEMU allows to create SD card with unrealistic sizes. This could > work, but some guests (at least Linux) consider sizes that are not > a power of 2 as a firmware bug and fix the card size to the next > power of 2. > > +

Re: [PATCH v2 7/9] hw/sd/sdcard: Do not allow invalid SD card sizes

2020-07-13 Thread Alistair Francis
On Mon, Jul 13, 2020 at 11:33 AM Philippe Mathieu-Daudé wrote: > > QEMU allows to create SD card with unrealistic sizes. This could > work, but some guests (at least Linux) consider sizes that are not > a power of 2 as a firmware bug and fix the card size to the next > power of 2. > > While the

Re: [PATCH v2 4/9] tests/acceptance/boot_linux: Expand SD card image to power of 2

2020-07-13 Thread Alistair Francis
On Mon, Jul 13, 2020 at 11:34 AM Philippe Mathieu-Daudé wrote: > > In few commits we won't allow SD card images with invalid size > (not aligned to a power of 2). Prepare the tests: add the > pow2ceil() and image_pow2ceil_expand() methods and resize the > images (expanding) of the tests using SD

Re: [PATCH v2 2/9] docs/orangepi: Add instructions for resizing SD image to power of two

2020-07-13 Thread Alistair Francis
On Mon, Jul 13, 2020 at 11:32 AM Philippe Mathieu-Daudé wrote: > > From: Niek Linnenbank > > SD cards need to have a size of a power of two. > Update the Orange Pi machine documentation to include > instructions for resizing downloaded images using the > qemu-img command. > > Signed-off-by: Niek

Re: [PATCH v2 8/9] hw/sd/sdcard: Update coding style to make checkpatch.pl happy

2020-07-13 Thread Alexander Bulekov
On 200713 2032, Philippe Mathieu-Daudé wrote: > To make the next commit easier to review, clean this code first. > > Reviewed-by: Peter Maydell > Signed-off-by: Philippe Mathieu-Daudé > Reviewed-by: Alistair Francis > Message-Id: <20200630133912.9428-3-f4...@amsat.org> > --- Reviewed-by:

Re: [PATCH v2 3/9] tests/acceptance/boot_linux: Tag tests using a SD card with 'device:sd'

2020-07-13 Thread Alistair Francis
On Mon, Jul 13, 2020 at 11:34 AM Philippe Mathieu-Daudé wrote: > > Avocado tags are handy to automatically select tests matching > the tags. Since these tests use a SD card, tag them. > > We can run all the tests using a SD card at once with: > > $ avocado --show=app run -t u-boot

[PATCH v2 8/9] hw/sd/sdcard: Update coding style to make checkpatch.pl happy

2020-07-13 Thread Philippe Mathieu-Daudé
To make the next commit easier to review, clean this code first. Reviewed-by: Peter Maydell Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Alistair Francis Message-Id: <20200630133912.9428-3-f4...@amsat.org> --- hw/sd/sd.c | 24 1 file changed, 16 insertions(+), 8

[PATCH v2 9/9] hw/sd/sdcard: Do not switch to ReceivingData if address is invalid

2020-07-13 Thread Philippe Mathieu-Daudé
Only move the state machine to ReceivingData if there is no pending error. This avoids later OOB access while processing commands queued. "SD Specifications Part 1 Physical Layer Simplified Spec. v3.01" 4.3.3 Data Read Read command is rejected if BLOCK_LEN_ERROR or ADDRESS_ERROR

[PATCH v2 7/9] hw/sd/sdcard: Do not allow invalid SD card sizes

2020-07-13 Thread Philippe Mathieu-Daudé
QEMU allows to create SD card with unrealistic sizes. This could work, but some guests (at least Linux) consider sizes that are not a power of 2 as a firmware bug and fix the card size to the next power of 2. While the possibility to use small SD card images has been seen as a feature, it became

[PATCH v2 5/9] hw/sd/sdcard: Restrict Class 6 commands to SCSD cards

2020-07-13 Thread Philippe Mathieu-Daudé
Only SCSD cards support Class 6 (Block Oriented Write Protection) commands. "SD Specifications Part 1 Physical Layer Simplified Spec. v3.01" 4.3.14 Command Functional Difference in Card Capacity Types * Write Protected Group SDHC and SDXC do not support write-protected groups. Issuing

[PATCH v2 1/9] MAINTAINERS: Cc qemu-block mailing list

2020-07-13 Thread Philippe Mathieu-Daudé
From: Philippe Mathieu-Daudé We forgot to include the qemu-block mailing list while adding this section in commit 076a0fc32a7. Fix this. Suggested-by: Paolo Bonzini Reviewed-by: Peter Maydell Signed-off-by: Philippe Mathieu-Daudé Message-Id: <20200630133912.9428-2-f4...@amsat.org> ---

[PATCH v2 4/9] tests/acceptance/boot_linux: Expand SD card image to power of 2

2020-07-13 Thread Philippe Mathieu-Daudé
In few commits we won't allow SD card images with invalid size (not aligned to a power of 2). Prepare the tests: add the pow2ceil() and image_pow2ceil_expand() methods and resize the images (expanding) of the tests using SD cards. Signed-off-by: Philippe Mathieu-Daudé --- Since v1: Addressed

[PATCH v2 2/9] docs/orangepi: Add instructions for resizing SD image to power of two

2020-07-13 Thread Philippe Mathieu-Daudé
From: Niek Linnenbank SD cards need to have a size of a power of two. Update the Orange Pi machine documentation to include instructions for resizing downloaded images using the qemu-img command. Signed-off-by: Niek Linnenbank Reviewed-by: Philippe Mathieu-Daudé Message-Id:

[PATCH v2 3/9] tests/acceptance/boot_linux: Tag tests using a SD card with 'device:sd'

2020-07-13 Thread Philippe Mathieu-Daudé
Avocado tags are handy to automatically select tests matching the tags. Since these tests use a SD card, tag them. We can run all the tests using a SD card at once with: $ avocado --show=app run -t u-boot tests/acceptance/ $ AVOCADO_ALLOW_LARGE_STORAGE=ok \ avocado --show=app \ run

[PATCH v2 6/9] hw/sd/sdcard: Simplify realize() a bit

2020-07-13 Thread Philippe Mathieu-Daudé
We don't need to check if sd->blk is set twice. Reviewed-by: Peter Maydell Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Alistair Francis Message-Id: <20200630133912.9428-18-f4...@amsat.org> --- hw/sd/sd.c | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git

[PATCH v2 0/9] hw/sd/sdcard: Fix CVE-2020-13253

2020-07-13 Thread Philippe Mathieu-Daudé
This series fixes CVE-2020-13253 by only allowing SD card image sizes power of 2, and not switching to SEND_DATA state when the address is invalid (out of range). Patches missing review: 3: boot_linux: Tag tests using a SD card with 'device:sd' 4: boot_linux: Expand SD card image to power of 2

[PATCH v5] scripts/simplebench: compare write request performance

2020-07-13 Thread Andrey Shinkevich
The script 'bench_write_req.py' allows comparing performances of write request for two qemu-img binary files. An example with (qemu-img binary 1) and without (qemu-img binary 2) the applied patch "qcow2: skip writing zero buffers to empty COW areas" (git commit ID: c8bb23cbdbe32f5) has the

Re: [PATCH for-5.1 1/2] block: Require aligned image size to avoid assertion failure

2020-07-13 Thread Kevin Wolf
Am 13.07.2020 um 18:33 hat Nir Soffer geschrieben: > On Fri, Jul 10, 2020 at 5:22 PM Kevin Wolf wrote: > > > > Unaligned requests will automatically be aligned to bl.request_alignment > > and we don't want to extend requests to access space beyond the end of > > the image, so it's required that

Re: [PATCH v3 03/11] hw/sd/sdcard: Do not switch to ReceivingData if address is invalid

2020-07-13 Thread Philippe Mathieu-Daudé
On 7/13/20 6:36 PM, Philippe Mathieu-Daudé wrote: > On 6/15/20 4:06 PM, Peter Maydell wrote: >> On Fri, 5 Jun 2020 at 11:25, Philippe Mathieu-Daudé >> wrote: >>> >>> From: Philippe Mathieu-Daudé >>> >>> Only move the state machine to ReceivingData if there is no >>> pending error. This avoids

Re: [PATCH v3 03/11] hw/sd/sdcard: Do not switch to ReceivingData if address is invalid

2020-07-13 Thread Philippe Mathieu-Daudé
On 6/15/20 4:06 PM, Peter Maydell wrote: > On Fri, 5 Jun 2020 at 11:25, Philippe Mathieu-Daudé wrote: >> >> From: Philippe Mathieu-Daudé >> >> Only move the state machine to ReceivingData if there is no >> pending error. This avoids later OOB access while processing >> commands queued. >> >>

Re: [PATCH for-5.1 1/2] block: Require aligned image size to avoid assertion failure

2020-07-13 Thread Nir Soffer
On Fri, Jul 10, 2020 at 5:22 PM Kevin Wolf wrote: > > Unaligned requests will automatically be aligned to bl.request_alignment > and we don't want to extend requests to access space beyond the end of > the image, so it's required that the image size is aligned. > > With write requests, this could

Re: [PATCH for-5.1 2/2] file-posix: Allow byte-aligned O_DIRECT with NFS

2020-07-13 Thread Nir Soffer
On Fri, Jul 10, 2020 at 5:22 PM Kevin Wolf wrote: > > Since commit a6b257a08e3 ('file-posix: Handle undetectable alignment'), > we assume that if we open a file with O_DIRECT and alignment probing > returns 1, we just couldn't find out the real alignment requirement > because some filesystems

Re: [PATCH] vvfat: set status to odd fixes

2020-07-13 Thread Kevin Wolf
Am 10.07.2020 um 21:04 hat P J P geschrieben: > From: Prasad J Pandit > > Virtual VFAT driver is quite old and rarely used. Set its status > to Odd Fixes. > > Signed-off-by: Prasad J Pandit Makes sense, yes. Thanks, applied to the block branch. Kevin

Re: [PATCH v2 4/5] iotests.py: filter_testfiles(): filter SOCK_DIR too

2020-07-13 Thread Vladimir Sementsov-Ogievskiy
13.07.2020 17:00, Eric Blake wrote: On 7/13/20 8:07 AM, Eric Blake wrote: On 7/1/20 5:53 AM, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy ---   tests/qemu-iotests/iotests.py | 5 +++--   1 file changed, 3 insertions(+), 2 deletions(-) Reviewed-by: Eric

Re: [PATCH for-5.1 1/2] block: Require aligned image size to avoid assertion failure

2020-07-13 Thread Kevin Wolf
Am 13.07.2020 um 13:19 hat Max Reitz geschrieben: > On 10.07.20 16:21, Kevin Wolf wrote: > > Unaligned requests will automatically be aligned to bl.request_alignment > > and we don't want to extend requests to access space beyond the end of > > the image, so it's required that the image size is

[PULL v3] Block layer patches

2020-07-13 Thread Kevin Wolf
The following changes since commit 6c87d9f311dba0641bdc2df556056938a8bf2a12: Merge remote-tracking branch 'remotes/elmarco/tags/chardev-pull-request' into staging (2020-07-13 09:34:24 +0100) are available in the Git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for you

[PULL 1/5] nbd: Avoid off-by-one in long export name truncation

2020-07-13 Thread Eric Blake
When snprintf returns the same value as the buffer size, the final byte was truncated to ensure a NUL terminator. Fortunately, such long export names are unusual enough, with no real impact other than what is displayed to the user. Fixes: 5c86bdf12089 Reported-by: Max Reitz Signed-off-by: Eric

[PULL 4/5] iotests.py: QemuIoInteractive: print output on failure

2020-07-13 Thread Eric Blake
From: Vladimir Sementsov-Ogievskiy Make it simpler to debug when qemu-io fails due to wrong arguments or environment. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Message-Id: <20200701105331.121670-3-vsement...@virtuozzo.com> Signed-off-by: Eric Blake ---

[PULL 5/5] iotests.py: filter_testfiles(): filter SOCK_DIR too

2020-07-13 Thread Eric Blake
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200701105331.121670-5-vsement...@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/iotests.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git

[PULL 3/5] iotests: QemuIoInteractive: use qemu_io_args_no_fmt

2020-07-13 Thread Eric Blake
From: Vladimir Sementsov-Ogievskiy The only user (iotest 205) of QemuIoInteractive provides -f argument, so it's a bit inefficient to use qemu_io_args, which contains -f too. And we are going to add one more test, which wants to specify -f by hand. Let's use qemu_io_args_no_fmt. Signed-off-by:

Re: [PATCH v2 4/5] iotests.py: filter_testfiles(): filter SOCK_DIR too

2020-07-13 Thread Eric Blake
On 7/13/20 8:07 AM, Eric Blake wrote: On 7/1/20 5:53 AM, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy ---   tests/qemu-iotests/iotests.py | 5 +++--   1 file changed, 3 insertions(+), 2 deletions(-) Reviewed-by: Eric Blake Queuing 1, 2, and 4 through my

Re: [PATCH for-5.1] file-posix: Mitigate file fragmentation with extent size hints

2020-07-13 Thread Kevin Wolf
Am 13.07.2020 um 15:12 hat Kevin Wolf geschrieben: > Am 13.07.2020 um 11:08 hat Max Reitz geschrieben: > > On 10.07.20 18:12, Max Reitz wrote: > > > On 07.07.20 18:17, Kevin Wolf wrote: > > >> Am 07.07.2020 um 16:23 hat Kevin Wolf geschrieben: > > >>> Espeically when O_DIRECT is used with image

Re: [PATCH] fixup! iotests: Specify explicit backing format where sensible

2020-07-13 Thread Eric Blake
On 7/13/20 7:50 AM, Kevin 'Max' Wolf wrote: Am 13.07.2020 um 13:40 hat Eric Blake geschrieben: Max spotted that I didn't test -qed... Signed-off-by: Eric Blake I guess I have a new name now... :-) Sorry, too many replies on the thread, and I messed up on which one I was reading!

Re: [PATCH v2 5/5] iotests: test shutdown when bitmap is exported through NBD

2020-07-13 Thread Eric Blake
On 7/1/20 5:53 AM, Vladimir Sementsov-Ogievskiy wrote: Test shutdown when bitmap is exported through NBD and active client exists. The previous patch fixes a crash, provoked by this scenario. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/299 | 65

Re: [PATCH for-5.1] file-posix: Mitigate file fragmentation with extent size hints

2020-07-13 Thread Kevin Wolf
Am 13.07.2020 um 11:08 hat Max Reitz geschrieben: > On 10.07.20 18:12, Max Reitz wrote: > > On 07.07.20 18:17, Kevin Wolf wrote: > >> Am 07.07.2020 um 16:23 hat Kevin Wolf geschrieben: > >>> Espeically when O_DIRECT is used with image files so that the page cache > >>> indirection can't cause a

Re: [PATCH v2 4/5] iotests.py: filter_testfiles(): filter SOCK_DIR too

2020-07-13 Thread Eric Blake
On 7/1/20 5:53 AM, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) Reviewed-by: Eric Blake diff --git a/tests/qemu-iotests/iotests.py

Re: [PATCH v2 3/5] nbd: make nbd_export_close_all() synchronous

2020-07-13 Thread Eric Blake
On 7/1/20 5:53 AM, Vladimir Sementsov-Ogievskiy wrote: Consider nbd_export_close_all(). The call-stack looks like this: nbd_export_close_all() -> nbd_export_close -> call client_close() for each client. client_close() doesn't guarantee that client is closed: nbd_trip() keeps reference to it.

Re: [PATCH v9 00/10] iotests: Dump QCOW2 dirty bitmaps metadata

2020-07-13 Thread Andrey Shinkevich
On 13.07.2020 15:41, Eric Blake wrote: On 7/13/20 2:27 AM, Andrey Shinkevich wrote: Add dirty bitmap information to QCOW2 metadata dump in the qcow2_format.py. v9:    01: In patch 0003, removed explicit constructor in the class Qcow2BitmapExt.    02: In patch 0004, the format string was

Re: [PATCH] fixup! iotests: Specify explicit backing format where sensible

2020-07-13 Thread Kevin 'Max' Wolf
Am 13.07.2020 um 13:40 hat Eric Blake geschrieben: > Max spotted that I didn't test -qed... > > Signed-off-by: Eric Blake I guess I have a new name now... :-) Kevin^WMax

Re: [PATCH v9 00/10] iotests: Dump QCOW2 dirty bitmaps metadata

2020-07-13 Thread Eric Blake
On 7/13/20 2:27 AM, Andrey Shinkevich wrote: Add dirty bitmap information to QCOW2 metadata dump in the qcow2_format.py. v9: 01: In patch 0003, removed explicit constructor in the class Qcow2BitmapExt. 02: In patch 0004, the format string was changed. Andrey Shinkevich (10):

Re: [PATCH v2] scripts/simplebench: compare write request performance

2020-07-13 Thread Vladimir Sementsov-Ogievskiy
13.07.2020 13:26, Andrey Shinkevich wrote: On 13.07.2020 11:43, Vladimir Sementsov-Ogievskiy wrote: 12.07.2020 19:07, Andrey Shinkevich wrote: On 11.07.2020 16:05, Vladimir Sementsov-Ogievskiy wrote: 26.06.2020 17:31, Andrey Shinkevich wrote: The script 'bench_write_req.py' allows comparing

Re: [PATCH v4 1/3] scripts/simplebench: compare write request performance

2020-07-13 Thread Vladimir Sementsov-Ogievskiy
12.07.2020 20:49, Andrey Shinkevich wrote: The script 'bench_write_req.py' allows comparing performances of write request for two qemu-img binary files. An example with (qemu-img binary 1) and without (qemu-img binary 2) the applied patch "qcow2: skip writing zero buffers to empty COW areas"

Re: [PATCH for-5.1 1/2] block: Require aligned image size to avoid assertion failure

2020-07-13 Thread Max Reitz
On 13.07.20 13:19, Max Reitz wrote: > On 10.07.20 16:21, Kevin Wolf wrote: >> Unaligned requests will automatically be aligned to bl.request_alignment >> and we don't want to extend requests to access space beyond the end of >> the image, so it's required that the image size is aligned. >> >> With

[PATCH] fixup! iotests: Specify explicit backing format where sensible

2020-07-13 Thread Eric Blake
Max spotted that I didn't test -qed... Signed-off-by: Eric Blake --- tests/qemu-iotests/024 | 2 +- tests/qemu-iotests/024.out | 1 - tests/qemu-iotests/043 | 5 +++-- tests/qemu-iotests/043.out | 4 4 files changed, 4 insertions(+), 8 deletions(-) diff --git

Re: [PATCH 3/5] block: introduce preallocate filter

2020-07-13 Thread Stefan Hajnoczi
On Wed, Jul 08, 2020 at 07:17:52PM +0300, Vladimir Sementsov-Ogievskiy wrote: > 08.07.2020 15:07, Stefan Hajnoczi wrote: > > On Sat, Jun 20, 2020 at 05:36:47PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > It may be used for file-systems with slow allocation. > > > > > > Signed-off-by:

Re: [PATCH 2/5] block/io: introduce bdrv_co_range_try_lock

2020-07-13 Thread Stefan Hajnoczi
On Wed, Jul 08, 2020 at 06:56:53PM +0300, Vladimir Sementsov-Ogievskiy wrote: > 07.07.2020 19:10, Stefan Hajnoczi wrote: > > On Sat, Jun 20, 2020 at 05:36:46PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > +int coroutine_fn bdrv_co_pwrite_zeroes_locked(BdrvChild *child, int64_t > > >

Re: [PATCH 1/5] block/io: introduce bdrv_try_mark_request_serialising

2020-07-13 Thread Stefan Hajnoczi
On Wed, Jul 08, 2020 at 06:51:18PM +0300, Vladimir Sementsov-Ogievskiy wrote: > 07.07.2020 18:56, Stefan Hajnoczi wrote: > > On Sat, Jun 20, 2020 at 05:36:45PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > Introduce a function to mark the request serialising only if there are > > > no

[PATCH] fixup! qemu-img: Flush stdout before before potential stderr messages

2020-07-13 Thread Eric Blake
Max noticed that I didn't run iotests on -luks... Signed-off-by: Eric Blake --- tests/qemu-iotests/282.out | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/282.out b/tests/qemu-iotests/282.out index 5d079dabce1d..67db7ab15aff 100644 ---

Re: [PATCH for-5.1 1/2] block: Require aligned image size to avoid assertion failure

2020-07-13 Thread Max Reitz
On 10.07.20 16:21, Kevin Wolf wrote: > Unaligned requests will automatically be aligned to bl.request_alignment > and we don't want to extend requests to access space beyond the end of > the image, so it's required that the image size is aligned. > > With write requests, this could cause

Re: [PATCH v2] scripts/simplebench: compare write request performance

2020-07-13 Thread Andrey Shinkevich
On 13.07.2020 11:43, Vladimir Sementsov-Ogievskiy wrote: 12.07.2020 19:07, Andrey Shinkevich wrote: On 11.07.2020 16:05, Vladimir Sementsov-Ogievskiy wrote: 26.06.2020 17:31, Andrey Shinkevich wrote: The script 'bench_write_req.py' allows comparing performances of write request for two

Re: [PATCH v2] scripts/simplebench: compare write request performance

2020-07-13 Thread Andrey Shinkevich
On 13.07.2020 11:43, Vladimir Sementsov-Ogievskiy wrote: 12.07.2020 19:07, Andrey Shinkevich wrote: On 11.07.2020 16:05, Vladimir Sementsov-Ogievskiy wrote: 26.06.2020 17:31, Andrey Shinkevich wrote: The script 'bench_write_req.py' allows comparing performances of write request for two

Re: [PATCH v5 3/5] virtio-scsi: default num_queues to -smp N

2020-07-13 Thread Stefan Hajnoczi
On Wed, Jul 08, 2020 at 06:50:12PM +0200, Cornelia Huck wrote: > On Wed, 8 Jul 2020 14:05:26 +0100 > Stefan Hajnoczi wrote: > > > On Tue, Jul 07, 2020 at 05:44:53PM +0200, Cornelia Huck wrote: > > > On Mon, 6 Jul 2020 14:56:48 +0100 > > > Stefan Hajnoczi wrote: > > > > > diff --git

Re: [PATCH v7 02/47] block: Add chain helper functions

2020-07-13 Thread Vladimir Sementsov-Ogievskiy
25.06.2020 18:21, Max Reitz wrote: Add some helper functions for skipping filters in a chain of block nodes. Signed-off-by: Max Reitz --- include/block/block_int.h | 3 +++ block.c | 55 +++ 2 files changed, 58 insertions(+) diff

Re: [PATCH] block: Raise an error when backing file parameter is an empty string

2020-07-13 Thread Max Reitz
On 17.06.20 20:27, Sorry :/ Connor Kuehl wrote: > Providing an empty string for the backing file parameter like so: > > qemu-img create -f qcow2 -b '' /tmp/foo > > allows the flow of control to reach and subsequently fail an assert > statement because passing an empty string to > >

Re: [PATCH v7 01/47] block: Add child access functions

2020-07-13 Thread Vladimir Sementsov-Ogievskiy
25.06.2020 18:21, Max Reitz wrote: There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be a filtered child;

Re: [PATCH 08/12] block/vmdk: Remove superfluous breaks

2020-07-13 Thread Thomas Huth
On 13/07/2020 11.05, Yi Wang wrote: > From: Liao Pingfang > > Remove superfluous breaks, as there is a "return" before them. > > Signed-off-by: Liao Pingfang > Signed-off-by: Yi Wang > Reviewed-by: Philippe Mathieu-Daudé > --- > block/vmdk.c | 3 --- > 1 file changed, 3 deletions(-) > >

Re: [PATCH v5 12/16] python/machine.py: Add _qmp access shim

2020-07-13 Thread Philippe Mathieu-Daudé
On 7/10/20 7:22 AM, John Snow wrote: > Like many other Optional[] types, it's not always a given that this > object will be set. Wrap it in a type-shim that raises a meaningful > error and will always return a concrete type. > > Signed-off-by: John Snow > --- > python/qemu/machine.py | 24

Re: [PATCH v5 16/16] python/qemu: Add mypy type annotations

2020-07-13 Thread Philippe Mathieu-Daudé
On 7/10/20 7:22 AM, John Snow wrote: > These should all be purely annotations with no changes in behavior at > all. You need to be in the python folder, but you should be able to > confirm that these annotations are correct (or at least self-consistent) > by running `mypy --strict qemu`. > >

Re: [PATCH v5 13/16] python/machine.py: fix _popen access

2020-07-13 Thread Philippe Mathieu-Daudé
On 7/10/20 7:22 AM, John Snow wrote: > As always, Optional[T] causes problems with unchecked access. Add a > helper that asserts the pipe is present before we attempt to talk with > it. > > Signed-off-by: John Snow > --- > python/qemu/machine.py | 16 +++- > 1 file changed, 11

Re: [PATCH v6 01/10] qemu-img: Flush stdout before before potential stderr messages

2020-07-13 Thread Max Reitz
On 06.07.20 22:39, Eric Blake wrote: > During 'qemu-img create ... 2>&1', if --quiet is not in force, we can > end up with buffered I/O in stdout that was produced before failure, > but which appears in output after failure. This is confusing; the fix > is to flush stdout prior to attempting

Re: [PATCH for-5.1] file-posix: Mitigate file fragmentation with extent size hints

2020-07-13 Thread Max Reitz
On 10.07.20 18:12, Max Reitz wrote: > On 07.07.20 18:17, Kevin Wolf wrote: >> Am 07.07.2020 um 16:23 hat Kevin Wolf geschrieben: >>> Espeically when O_DIRECT is used with image files so that the page cache >>> indirection can't cause a merge of allocating requests, the file will >>> fragment on

Re: [PATCH v7 01/47] block: Add child access functions

2020-07-13 Thread Vladimir Sementsov-Ogievskiy
25.06.2020 18:21, Max Reitz wrote: There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be a filtered child;

Re: [PATCH v2] scripts/simplebench: compare write request performance

2020-07-13 Thread Vladimir Sementsov-Ogievskiy
12.07.2020 19:07, Andrey Shinkevich wrote: On 11.07.2020 16:05, Vladimir Sementsov-Ogievskiy wrote: 26.06.2020 17:31, Andrey Shinkevich wrote: The script 'bench_write_req.py' allows comparing performances of write request for two qemu-img binary files. An example with (qemu-img binary 1) and

Re: [PATCH v8 03/10] qcow2_format.py: change Qcow2BitmapExt initialization method

2020-07-13 Thread Vladimir Sementsov-Ogievskiy
13.07.2020 07:49, Andrey Shinkevich wrote: On 11.07.2020 19:34, Vladimir Sementsov-Ogievskiy wrote: 03.07.2020 16:13, Andrey Shinkevich wrote: There are two ways to initialize a class derived from Qcow2Struct: 1. Pass a block of binary data to the constructor. 2. Pass the file descriptor to

Re: [PATCH v8 05/10] qcow2_format.py: Dump bitmap directory information

2020-07-13 Thread Vladimir Sementsov-Ogievskiy
13.07.2020 10:07, Andrey Shinkevich wrote: On 11.07.2020 22:11, Vladimir Sementsov-Ogievskiy wrote: 03.07.2020 16:13, Andrey Shinkevich wrote: Read and dump entries from the bitmap directory of QCOW2 image. It extends the output in the test case #291. ...   diff --git

Re: [PATCH 2/2] iotests: Set LC_ALL=C for sort

2020-07-13 Thread Max Reitz
On 11.07.20 10:57, Alex Bennée wrote: > > Max Reitz writes: > >> Otherwise the result is basically unpredictable. >> >> (Note that the precise environment variable to control sorting order is >> LC_COLLATE, but LC_ALL overrides LC_COLLATE, and we do not want the >> sorting order to be messed up

Re: [PATCH v3 0/4] hw/block/nvme: Fix I/O BAR structure

2020-07-13 Thread Klaus Jensen
On Jun 30 13:04, Philippe Mathieu-Daudé wrote: > Improvements for the I/O BAR structure: > - correct pmrmsc register size (Klaus) > - pack structures > - align to 4KB > > Since v2: > - Added Klaus' tags with correct address > > $ git backport-diff -u v2 > Key: > [] : patches are identical >

[PATCH v9 10/10] qcow2_format.py: support dumping metadata in JSON format

2020-07-13 Thread Andrey Shinkevich
Implementation of dumping QCOW2 image metadata. The sample output: { "Header_extensions": [ { "name": "Feature table", "magic": 1745090647, "length": 192, "data_str": "" }, { "name": "Bitmaps",

[PATCH v9 05/10] qcow2_format.py: Dump bitmap directory information

2020-07-13 Thread Andrey Shinkevich
Read and dump entries from the bitmap directory of QCOW2 image. It extends the output in the test case #291. Header extension: magic 0x23852875 (Bitmaps) ... Bitmap name bitmap-1 bitmap_table_offset 0xf bitmap_table_size 1 flags

[PATCH v9 06/10] qcow2_format.py: pass cluster size to substructures

2020-07-13 Thread Andrey Shinkevich
The cluster size of an image is the QcowHeader class member and may be obtained by dependent extension structures such as Qcow2BitmapExt for further bitmap table details print. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 17 +++-- 1 file changed, 11

[PATCH v9 01/10] qcow2: Fix capitalization of header extension constant.

2020-07-13 Thread Andrey Shinkevich
Make the capitalization of the hexadecimal numbers consistent for the QCOW2 header extension constants in docs/interop/qcow2.txt. Suggested-by: Eric Blake Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.c | 2 +- docs/interop/qcow2.txt | 2

[PATCH v9 00/10] iotests: Dump QCOW2 dirty bitmaps metadata

2020-07-13 Thread Andrey Shinkevich
Add dirty bitmap information to QCOW2 metadata dump in the qcow2_format.py. v9: 01: In patch 0003, removed explicit constructor in the class Qcow2BitmapExt. 02: In patch 0004, the format string was changed. Andrey Shinkevich (10): qcow2: Fix capitalization of header extension constant.

[PATCH v9 09/10] qcow2_format.py: collect fields to dump in JSON format

2020-07-13 Thread Andrey Shinkevich
As __dict__ is being extended with class members we do not want to print, make a light copy of the initial __dict__ and extend the copy by adding lists we have to print in the JSON output. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 4 1 file changed, 4

[PATCH v9 07/10] qcow2_format.py: Dump bitmap table serialized entries

2020-07-13 Thread Andrey Shinkevich
Add bitmap table information to the QCOW2 metadata dump. It extends the output of the test case #291 Bitmap name bitmap-1 ... Bitmap table typeoffset size 0 serialized 4718592 65536 1 serialized

[PATCH v9 03/10] qcow2_format.py: change Qcow2BitmapExt initialization method

2020-07-13 Thread Andrey Shinkevich
There are two ways to initialize a class derived from Qcow2Struct: 1. Pass a block of binary data to the constructor. 2. Pass the file descriptor to allow reading the file from constructor. Let's change the Qcow2BitmapExt initialization method from 1 to 2 to support a scattered reading in the

[PATCH v9 04/10] qcow2_format.py: dump bitmap flags in human readable way.

2020-07-13 Thread Andrey Shinkevich
Introduce the class BitmapFlags that parses a bitmap flags mask. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 16 1 file changed, 16 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py

[PATCH v9 08/10] qcow2.py: Introduce '-j' key to dump in JSON format

2020-07-13 Thread Andrey Shinkevich
Add the command key to the qcow2.py arguments list to dump QCOW2 metadata in JSON format. Here is the suggested way to do that. The implementation of the dump in JSON format is in the patch that follows. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2.py| 19

[PATCH v9 02/10] qcow2_format.py: make printable data an extension class member

2020-07-13 Thread Andrey Shinkevich
Let us differ binary data type from string one for the extension data variable and keep the string as the QcowHeaderExtension class member. Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 14

Re: [PATCH v8 05/10] qcow2_format.py: Dump bitmap directory information

2020-07-13 Thread Andrey Shinkevich
On 11.07.2020 22:11, Vladimir Sementsov-Ogievskiy wrote: 03.07.2020 16:13, Andrey Shinkevich wrote: Read and dump entries from the bitmap directory of QCOW2 image. It extends the output in the test case #291. ...   diff --git a/tests/qemu-iotests/qcow2_format.py

Re: [PATCH 0/2] iotests: More _filter_img_create fixes

2020-07-13 Thread Max Reitz
On 10.07.20 19:18, John Snow wrote: > > > On 7/10/20 12:32 PM, Max Reitz wrote: >> Hi, >> >> I’m sorry. >> >> John, could I ask you to test whether this series fixes the problems >> you’re seeing? >> > > This is based on kwolf/block, I see. > > By the time you return to reading work email,