[PULL 3/6] hw/nvme: move format parameter parsing

2022-03-03 Thread Klaus Jensen
From: Klaus Jensen There is no need to extract the format command parameters for each namespace. Move it to the entry point. Reviewed-by: Keith Busch Signed-off-by: Klaus Jensen --- hw/nvme/ctrl.c | 31 ++- 1 file changed, 18 insertions(+), 13 deletions(-) diff

[PULL 6/6] hw/nvme: 64-bit pi support

2022-03-03 Thread Klaus Jensen
From: Naveen Nagar This adds support for one possible new protection information format introduced in TP4068 (and integrated in NVMe 2.0): the 64-bit CRC guard and 48-bit reference tag. This version does not support storage tags. Like the CRC16 support already present, this uses a software

[PULL 4/6] hw/nvme: add support for the lbafee hbs feature

2022-03-03 Thread Klaus Jensen
From: Naveen Nagar Add support for up to 64 LBA formats through the LBAFEE field of the Host Behavior Support feature. Reviewed-by: Keith Busch Signed-off-by: Naveen Nagar Signed-off-by: Klaus Jensen --- hw/nvme/ctrl.c | 34 +++--- hw/nvme/ns.c | 15

[PULL 5/6] hw/nvme: add pi tuple size helper

2022-03-03 Thread Klaus Jensen
From: Klaus Jensen A subsequent patch will introduce a new tuple size; so add a helper and use that instead of sizeof() and magic numbers. Reviewed-by: Keith Busch Signed-off-by: Klaus Jensen --- hw/nvme/ctrl.c | 14 -- hw/nvme/dif.c | 16 hw/nvme/dif.h | 5

[PULL 2/6] hw/nvme: add host behavior support feature

2022-03-03 Thread Klaus Jensen
From: Naveen Nagar Add support for getting and setting the Host Behavior Support feature. Reviewed-by: Keith Busch Signed-off-by: Naveen Nagar Signed-off-by: Klaus Jensen --- hw/nvme/ctrl.c | 8 hw/nvme/nvme.h | 4 +++- include/block/nvme.h | 9 + 3 files

[PULL 1/6] hw/nvme: move dif/pi prototypes into dif.h

2022-03-03 Thread Klaus Jensen
From: Klaus Jensen Move dif/pi data structures and inlines to dif.h. Reviewed-by: Keith Busch Signed-off-by: Klaus Jensen --- hw/nvme/ctrl.c | 1 + hw/nvme/dif.c | 1 + hw/nvme/dif.h | 53 ++ hw/nvme/nvme.h | 50

[PULL 0/6] hw/nvme updates

2022-03-03 Thread Klaus Jensen
From: Klaus Jensen Hi Peter, Last round of hw/nvme updates for v7.0. The following changes since commit 64ada298b98a51eb2512607f6e6180cb330c47b1: Merge remote-tracking branch 'remotes/legoater/tags/pull-ppc-20220302' into staging (2022-03-02 12:38:46 +) are available in the Git

Re: [PATCH 5/5] iotests: fortify compare_images() against crashes

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 03:59:02PM -0500, John Snow wrote: > Fority compare_images() to be more discerning about the status codes it Fortify > receives. If qemu_img() returns an exit code that implies it didn't > actually perform the comparison, treat that as an exceptional > circumstance and

Re: [PATCH 4/5] iotests: make qemu_img raise on non-zero rc by default

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 03:59:01PM -0500, John Snow wrote: > re-write qemu_img() as a function that will by default raise a > VerboseProcessException (extended from CalledProcessException) on > non-zero return codes. This will produce a stack trace that will show > the command line arguments and

Re: [PATCH 2/5] python/utils: add VerboseProcessError

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 03:58:59PM -0500, John Snow wrote: > This adds an Exception that extends the Python stdlib > subprocess.CalledProcessError. > > The difference is that the str() method of this exception also adds the > stdout/stderr logs. In effect, if this exception goes unhandled, Python

Re: [PATCH 4/9] util/oslib-win32: Return NULL on qemu_try_memalign() with zero size

2022-03-03 Thread Richard Henderson
On 3/3/22 06:55, Peter Maydell wrote: Alternately, force size == 1, so that we always get a non-NULL value that can be freed. That's a change on the POSIX side as well, of course. Yes, I had a look at what actual malloc() implementations tend to do, and the answer seems to be that forcing

Re: [PATCH 04/12] qemu-nbd: add --tls-hostname option for TLS certificate validation

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:22PM +, Daniel P. Berrangé wrote: > When using the --list option, qemu-nbd acts as an NBD client rather > than a server. As such when using TLS, it has a need to validate > the server certificate. This adds a --tls-hostname option which can > be used to override

Re: [PATCH 1/5] python/utils: add add_visual_margin() text decoration utility

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 03:58:58PM -0500, John Snow wrote: > >>> print(add_visual_margin(msg, width=72, name="Commit Message")) > ┏━ Commit Message ━━ > ┃ add_visual_margin() takes a chunk of text and wraps it in a visual > ┃ container that

Re: [PATCH v2 4/4] iotests/185: Add post-READY quit tests

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 05:48:14PM +0100, Hanna Reitz wrote: > 185 tests quitting qemu while a block job is active. It does not > specifically test quitting qemu while a mirror or active commit job is > in its READY phase. > > Add two test cases for this, where we respectively mirror or commit

Re: [PATCH v2 3/4] qsd: Add --daemonize

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 05:48:13PM +0100, Hanna Reitz wrote: > To implement this, we reuse the existing daemonizing functions from the > system emulator, which mainly do the following: > - Fork off a child process, and set up a pipe between parent and child > - The parent process waits until the

Re: [PATCH v2 2/4] qsd: Add pre-init argument parsing pass

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 05:48:12PM +0100, Hanna Reitz wrote: > In contrast to qemu-nbd (where it is called --fork) and the system > emulator, QSD does not have a --daemonize switch yet. Just like them, > QSD allows setting up block devices and exports on the command line. > When doing so, it is

Re: [PATCH 03/12] block/nbd: support override of hostname for TLS certificate validation

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:21PM +, Daniel P. Berrangé wrote: > When connecting to an NBD server with TLS and x509 credentials, > the client must validate the hostname it uses for the connection, > against that published in the server's certificate. If the client > is tunnelling its

[PATCH 5/5] iotests: fortify compare_images() against crashes

2022-03-03 Thread John Snow
Fority compare_images() to be more discerning about the status codes it receives. If qemu_img() returns an exit code that implies it didn't actually perform the comparison, treat that as an exceptional circumstance and force the caller to be aware of the peril. If a negative test is desired

[PATCH 3/5] iotests: Remove explicit checks for qemu_img() == 0

2022-03-03 Thread John Snow
qemu_img() returning zero ought to be the rule, not the exception. Remove all explicit checks against the condition in preparation for making non-zero returns an Exception. Signed-off-by: John Snow Reviewed-by: Eric Blake --- tests/qemu-iotests/163 | 9 +++--

[PATCH 2/5] python/utils: add VerboseProcessError

2022-03-03 Thread John Snow
This adds an Exception that extends the Python stdlib subprocess.CalledProcessError. The difference is that the str() method of this exception also adds the stdout/stderr logs. In effect, if this exception goes unhandled, Python will print the output in a visually distinct wrapper to the terminal

[PATCH 1/5] python/utils: add add_visual_margin() text decoration utility

2022-03-03 Thread John Snow
>>> print(add_visual_margin(msg, width=72, name="Commit Message")) ┏━ Commit Message ━━ ┃ add_visual_margin() takes a chunk of text and wraps it in a visual ┃ container that force-wraps to a specified width. An optional title ┃ label may be

[PATCH 4/5] iotests: make qemu_img raise on non-zero rc by default

2022-03-03 Thread John Snow
re-write qemu_img() as a function that will by default raise a VerboseProcessException (extended from CalledProcessException) on non-zero return codes. This will produce a stack trace that will show the command line arguments and return code from the failed process run. Users that want something

[PATCH 0/5] iotests: add enhanced debugging info to qemu-img failures

2022-03-03 Thread John Snow
This is kinda-sorta V3-ish of a series I started in response to Thomas Huth's encountering a failure in qemu-img because of missing zstd support. This series changes the qemu_img() function in iotests.py to one that raises an Exception on non-zero return code by default. Alongside this, the

Re: [PATCH v2 1/4] os-posix: Add os_set_daemonize()

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 05:48:11PM +0100, Hanna Reitz wrote: > The daemonizing functions in os-posix (os_daemonize() and > os_setup_post()) only daemonize the process if the static `daemonize` > variable is set. Right now, it can only be set by os_parse_cmd_args(). > > In order to use

Re: [PATCH 02/12] block: pass desired TLS hostname through from block driver client

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:20PM +, Daniel P. Berrangé wrote: > In > > commit a71d597b989fd701b923f09b3c20ac4fcaa55e81 > Author: Vladimir Sementsov-Ogievskiy > Date: Thu Jun 10 13:08:00 2021 +0300 > > block/nbd: reuse nbd_co_do_establish_connection() in nbd_open() > > the use

Re: [PATCH 01/12] crypto: mandate a hostname when checking x509 creds on a client

2022-03-03 Thread Eric Blake
On Thu, Mar 03, 2022 at 04:03:19PM +, Daniel P. Berrangé wrote: > Currently the TLS session object assumes that the caller will always > provide a hostname when using x509 creds on a client endpoint. This > relies on the caller to detect and report an error if the user has > configured QEMU

[PATCH v6 16/16] iotests/image-fleecing: test push backup with fleecing

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 125 +++- tests/qemu-iotests/tests/image-fleecing.out | 63 ++ 2 files changed, 156 insertions(+), 32 deletions(-) diff --git a/tests/qemu-iotests/tests/image-fleecing

[PATCH v6 11/16] block: introduce snapshot-access block driver

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
The new block driver simply utilizes snapshot-access API of underlying block node. In further patches we want to use it like this: [guest] [NBD export] || | root | root v file v

[PATCH v6 06/16] block: intoduce reqlist

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Split intersecting-requests functionality out of block-copy to be reused in copy-before-write filter. Note: while being here, fix tiny typo in MAINTAINERS. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/reqlist.h | 67 +++

[PATCH v6 15/16] iotests/image-fleecing: add test case with bitmap

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Note that reads zero areas (not dirty in the bitmap) fails, that's correct. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 38 +++--- tests/qemu-iotests/tests/image-fleecing.out | 84 + 2 files changed, 113 insertions(+), 9

[PATCH v6 10/16] block/io: introduce block driver snapshot-access API

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Add new block driver handlers and corresponding generic wrappers. It will be used to allow copy-before-write filter to provide reach fleecing interface in further commit. In future this approach may be used to allow reading qcow2 internal snapshots, for example to export them through NBD.

[PATCH v6 05/16] block/block-copy: add block_copy_reset()

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Split block_copy_reset() out of block_copy_reset_unallocated() to be used separately later. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/block-copy.h | 1 + block/block-copy.c | 21 + 2 files changed, 14 insertions(+), 8

[PATCH v6 07/16] block/reqlist: reqlist_find_conflict(): use ranges_overlap()

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Let's reuse convenient helper. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- block/reqlist.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/block/reqlist.c b/block/reqlist.c index 5e320ba649..09fecbd48c 100644 --- a/block/reqlist.c +++

[PATCH v6 14/16] iotests.py: add qemu_io_pipe_and_status()

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Add helper that returns both status and output, to be used in the following commit Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index

[PATCH v6 08/16] block/dirty-bitmap: introduce bdrv_dirty_bitmap_status()

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Add a convenient function similar with bdrv_block_status() to get status of dirty bitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 12 block/dirty-bitmap.c | 6 ++

[PATCH v6 12/16] block: copy-before-write: realize snapshot-access API

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Current scheme of image fleecing looks like this: [guest][NBD export] | | |root | root v v [copy-before-write] -> [temp.qcow2] | target | |file

[PATCH v6 13/16] iotests/image-fleecing: add test-case for fleecing format node

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- tests/qemu-iotests/tests/image-fleecing | 64 - tests/qemu-iotests/tests/image-fleecing.out | 76 - 2 files changed, 120 insertions(+), 20 deletions(-) diff --git

[PATCH v6 04/16] block/copy-before-write: add bitmap open parameter

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
This brings "incremental" mode to copy-before-write filter: user can specify bitmap so that filter will copy only "dirty" areas. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 10 +++- block/copy-before-write.c | 51 ++- 2

[PATCH v6 03/16] block/block-copy: block_copy_state_new(): add bitmap parameter

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
This will be used in the following commit to bring "incremental" mode to copy-before-write filter. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/block-copy.h | 1 + block/block-copy.c | 14 +- block/copy-before-write.c | 2 +- 3

[PATCH v6 02/16] block/dirty-bitmap: bdrv_merge_dirty_bitmap(): add return value

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
That simplifies handling failure in existing code and in further new usage of bdrv_merge_dirty_bitmap(). Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/dirty-bitmap.h| 2 +- block/dirty-bitmap.c| 9 +++--

[PATCH v6 09/16] block/reqlist: add reqlist_wait_all()

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
Add function to wait for all intersecting requests. To be used in the further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Nikita Lapshin Reviewed-by: Hanna Reitz --- include/block/reqlist.h | 8 block/reqlist.c | 8 2 files changed, 16

[PATCH v6 01/16] block/block-copy: move copy_bitmap initialization to block_copy_state_new()

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
We are going to complicate bitmap initialization in the further commit. And in future, backup job will be able to work without filter (when source is immutable), so we'll need same bitmap initialization in copy-before-write filter and in backup job. So, it's reasonable to do it in block-copy.

[PATCH v6 00/16] Make image fleecing more usable

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
v6: 11: add comment 15: limit to qcow2 with unsupported compat fix style 16: fix style change log('Backup finished ...') to assertion and comment Vladimir Sementsov-Ogievskiy (16): block/block-copy: move copy_bitmap initialization to block_copy_state_new() block/dirty-bitmap:

Re: [PATCH v5 16/16] iotests/image-fleecing: test push backup with fleecing

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
03.03.2022 13:58, Hanna Reitz wrote: On 28.02.22 12:39, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy ---   tests/qemu-iotests/tests/image-fleecing | 120 ++--   tests/qemu-iotests/tests/image-fleecing.out |  63 ++   2 files changed,

Re: [PATCH v5 11/16] block: introduce snapshot-access block driver

2022-03-03 Thread Vladimir Sementsov-Ogievskiy
03.03.2022 14:11, Hanna Reitz wrote: On 03.03.22 12:05, Hanna Reitz wrote: On 28.02.22 12:39, Vladimir Sementsov-Ogievskiy wrote: The new block driver simply utilizes snapshot-access API of underlying block node. In further patches we want to use it like this: [guest]   [NBD

Re: [PATCH v2 1/3] block: Make bdrv_refresh_limits() non-recursive

2022-03-03 Thread Kevin Wolf
Am 16.02.2022 um 11:53 hat Hanna Reitz geschrieben: > bdrv_refresh_limits() recurses down to the node's children. That does > not seem necessary: When we refresh limits on some node, and then > recurse down and were to change one of its children's BlockLimits, then > that would mean we noticed

Re: [PATCH 4/9] util/oslib-win32: Return NULL on qemu_try_memalign() with zero size

2022-03-03 Thread Peter Maydell
On Sun, 27 Feb 2022 at 18:36, Richard Henderson wrote: > > On 2/27/22 02:54, Peter Maydell wrote: > >>> +if (size) { > >>> +ptr = _aligned_malloc(size, alignment); > >>> +} else { > >>> +ptr = NULL; > >>> +} > >> > >> Oh, should we set errno to something here? > >>

[PATCH v2 4/4] iotests/185: Add post-READY quit tests

2022-03-03 Thread Hanna Reitz
185 tests quitting qemu while a block job is active. It does not specifically test quitting qemu while a mirror or active commit job is in its READY phase. Add two test cases for this, where we respectively mirror or commit to an external QSD instance, which provides a throttled block device.

[PATCH v2 3/4] qsd: Add --daemonize

2022-03-03 Thread Hanna Reitz
To implement this, we reuse the existing daemonizing functions from the system emulator, which mainly do the following: - Fork off a child process, and set up a pipe between parent and child - The parent process waits until the child sends a status byte over the pipe (0 means that the child was

[PATCH v2 2/4] qsd: Add pre-init argument parsing pass

2022-03-03 Thread Hanna Reitz
In contrast to qemu-nbd (where it is called --fork) and the system emulator, QSD does not have a --daemonize switch yet. Just like them, QSD allows setting up block devices and exports on the command line. When doing so, it is often necessary for whoever invoked the QSD to wait until these

[PATCH v2 1/4] os-posix: Add os_set_daemonize()

2022-03-03 Thread Hanna Reitz
The daemonizing functions in os-posix (os_daemonize() and os_setup_post()) only daemonize the process if the static `daemonize` variable is set. Right now, it can only be set by os_parse_cmd_args(). In order to use os_daemonize() and os_setup_post() from the storage daemon to have it be

[PATCH v2 0/4] qsd: Add --daemonize; and add job quit tests

2022-03-03 Thread Hanna Reitz
Hi, v1 cover letter: https://lists.nongnu.org/archive/html/qemu-block/2021-12/msg00499.html In v2, I followed Vladimir’s suggestion to look into whether we could reuse os_daemonize(). Indeed we can, and it makes patch 3 (formerly 2) much simpler! I decided to leave patch 2 (formerly 1)

[PATCH 09/12] tests/qemu-iotests: convert NBD TLS test to use standard filters

2022-03-03 Thread Daniel P . Berrangé
Using standard filters is more future proof than rolling our own. Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/233 | 29 - tests/qemu-iotests/233.out | 9 - 2 files changed, 16 insertions(+), 22 deletions(-) diff --git

[PATCH 12/12] tests/qemu-iotests: validate NBD TLS with UNIX sockets and PSK

2022-03-03 Thread Daniel P . Berrangé
This validates that connections to an NBD server running on a UNIX socket can use TLS with pre-shared keys (PSK). Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/233| 28 tests/qemu-iotests/233.out| 17 +

[PATCH 05/12] block/nbd: don't restrict TLS usage to IP sockets

2022-03-03 Thread Daniel P . Berrangé
The TLS usage for NBD was restricted to IP sockets because validating x509 certificates requires knowledge of the hostname that the client is connecting to. TLS does not have to use x509 certificates though, as PSK (pre-shared keys) provide an alternative credential option. These have no

[PATCH 04/12] qemu-nbd: add --tls-hostname option for TLS certificate validation

2022-03-03 Thread Daniel P . Berrangé
When using the --list option, qemu-nbd acts as an NBD client rather than a server. As such when using TLS, it has a need to validate the server certificate. This adds a --tls-hostname option which can be used to override the default hostname used for certificate validation. Signed-off-by: Daniel

[PATCH 03/12] block/nbd: support override of hostname for TLS certificate validation

2022-03-03 Thread Daniel P . Berrangé
When connecting to an NBD server with TLS and x509 credentials, the client must validate the hostname it uses for the connection, against that published in the server's certificate. If the client is tunnelling its connection over some other channel, however, the hostname it uses may not match the

[PATCH 02/12] block: pass desired TLS hostname through from block driver client

2022-03-03 Thread Daniel P . Berrangé
In commit a71d597b989fd701b923f09b3c20ac4fcaa55e81 Author: Vladimir Sementsov-Ogievskiy Date: Thu Jun 10 13:08:00 2021 +0300 block/nbd: reuse nbd_co_do_establish_connection() in nbd_open() the use of the 'hostname' field from the BDRVNBDState struct was lost, and 'nbd_connect' just

[PATCH 08/12] tests/qemu-iotests: introduce filter for qemu-nbd export list

2022-03-03 Thread Daniel P . Berrangé
Introduce a filter for the output of qemu-nbd export list so it can be reused in multiple tests. The filter is a bit more permissive that what test 241 currently uses, as its allows printing of the export count, along with any possible error messages that might be emitted. Signed-off-by: Daniel

[PATCH 11/12] tests/qemu-iotests: validate NBD TLS with UNIX sockets

2022-03-03 Thread Daniel P . Berrangé
This validates that connections to an NBD server running on a UNIX socket can use TLS, and require a TLS hostname override to pass certificate validation. Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/233 | 24 tests/qemu-iotests/233.out | 15

[PATCH 10/12] tests/qemu-iotests: validate NBD TLS with hostname mismatch

2022-03-03 Thread Daniel P . Berrangé
This validates that connections to an NBD server where the certificate hostname does not match will fail. It further validates that using the new 'tls-hostname' override option can solve the failure. Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/233| 18 ++

[PATCH 07/12] tests/qemu-iotests: expand _filter_nbd rules

2022-03-03 Thread Daniel P . Berrangé
Some tests will want to use 'localhost' instead of '127.0.0.1', and some will use the image options syntax rather than the classic URI syntax. Signed-off-by: Daniel P. Berrangé --- tests/qemu-iotests/common.filter | 4 1 file changed, 4 insertions(+) diff --git

[PATCH 06/12] tests/qemu-iotests: add QEMU_IOTESTS_REGEN=1 to update reference file

2022-03-03 Thread Daniel P . Berrangé
When developing an I/O test it is typical to add some logic to the test script, run it to view the output diff, and then apply the output diff to the reference file. This can be drastically simplified by letting the test runner update the reference file in place. By setting

[PATCH 01/12] crypto: mandate a hostname when checking x509 creds on a client

2022-03-03 Thread Daniel P . Berrangé
Currently the TLS session object assumes that the caller will always provide a hostname when using x509 creds on a client endpoint. This relies on the caller to detect and report an error if the user has configured QEMU with x509 credentials on a UNIX socket. The migration code has such a check,

[PATCH 00/12] nbd: enable use of TLS on non-TCP transports and other TLS improvements

2022-03-03 Thread Daniel P . Berrangé
This series was principally motivated by a desire to enabl use of TLS on non-TCP transports. For x509 certificates this means we need a way to set the hostname to use for validation. This also lets us override the hostname when connecting on a TCP transport that is tunnelled or port-forwarded. It

[PATCH v8 28/31] block_int-common.h: assertions in the callers of BdrvChildClass function pointers

2022-03-03 Thread Emanuele Giuseppe Esposito
Signed-off-by: Emanuele Giuseppe Esposito --- block.c | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index 5afdbe3060..01811d6519 100644 --- a/block.c +++ b/block.c @@ -1497,7 +1497,7 @@ const BdrvChildClass child_of_bds = { AioContext

[PATCH v8 27/31] block_int-common.h: split function pointers in BdrvChildClass

2022-03-03 Thread Emanuele Giuseppe Esposito
Signed-off-by: Emanuele Giuseppe Esposito --- include/block/block_int-common.h | 81 ++-- 1 file changed, 47 insertions(+), 34 deletions(-) diff --git a/include/block/block_int-common.h b/include/block/block_int-common.h index f05ebb0da3..5a04c778e4 100644 ---

[PATCH v8 31/31] job.h: assertions in the callers of JobDriver function pointers

2022-03-03 Thread Emanuele Giuseppe Esposito
Signed-off-by: Emanuele Giuseppe Esposito --- job.c | 10 ++ 1 file changed, 10 insertions(+) diff --git a/job.c b/job.c index 54db80df66..075c6f3a20 100644 --- a/job.c +++ b/job.c @@ -381,6 +381,8 @@ void job_ref(Job *job) void job_unref(Job *job) { +GLOBAL_STATE_CODE(); +

[PATCH v8 26/31] block_int-common.h: assertions in the callers of BlockDriver function pointers

2022-03-03 Thread Emanuele Giuseppe Esposito
Signed-off-by: Emanuele Giuseppe Esposito --- block.c| 17 + block/create.c | 2 ++ 2 files changed, 19 insertions(+) diff --git a/block.c b/block.c index 4a3447b2a0..5afdbe3060 100644 --- a/block.c +++ b/block.c @@ -529,6 +529,7 @@ static void coroutine_fn

[PATCH v8 29/31] block-backend-common.h: split function pointers in BlockDevOps

2022-03-03 Thread Emanuele Giuseppe Esposito
Assertions in the callers of the function pointrs are already added by previous patches. Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Stefan Hajnoczi Reviewed-by: Philippe Mathieu-Daudé --- include/sysemu/block-backend-common.h | 28 ++- 1 file changed, 23

[PATCH v8 25/31] block_int-common.h: split function pointers in BlockDriver

2022-03-03 Thread Emanuele Giuseppe Esposito
Similar to the header split, also the function pointers in BlockDriver can be split in I/O and global state. Signed-off-by: Emanuele Giuseppe Esposito --- include/block/block_int-common.h | 445 --- 1 file changed, 237 insertions(+), 208 deletions(-) diff --git

[PATCH v8 23/31] block/copy-before-write.h: global state API + assertions

2022-03-03 Thread Emanuele Giuseppe Esposito
copy-before-write functions always run under BQL. Signed-off-by: Emanuele Giuseppe Esposito --- block/copy-before-write.c | 2 ++ block/copy-before-write.h | 7 +++ 2 files changed, 9 insertions(+) diff --git a/block/copy-before-write.c b/block/copy-before-write.c index

[PATCH v8 19/31] assertions for blockjob.h global state API

2022-03-03 Thread Emanuele Giuseppe Esposito
Signed-off-by: Emanuele Giuseppe Esposito --- blockjob.c | 11 +++ 1 file changed, 11 insertions(+) diff --git a/blockjob.c b/blockjob.c index d79a52d204..4868453d74 100644 --- a/blockjob.c +++ b/blockjob.c @@ -62,6 +62,7 @@ static bool is_block_job(Job *job) BlockJob

[PATCH v8 20/31] include/sysemu/blockdev.h: global state API

2022-03-03 Thread Emanuele Giuseppe Esposito
blockdev functions run always under the BQL lock. Signed-off-by: Emanuele Giuseppe Esposito --- include/sysemu/blockdev.h | 13 ++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/include/sysemu/blockdev.h b/include/sysemu/blockdev.h index f9fb54d437..3211b16513 100644

[PATCH v8 18/31] include/block/blockjob.h: global state API

2022-03-03 Thread Emanuele Giuseppe Esposito
blockjob functions run always under the BQL lock. Signed-off-by: Emanuele Giuseppe Esposito --- include/block/blockjob.h | 29 ++--- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/include/block/blockjob.h b/include/block/blockjob.h index

[PATCH v8 10/31] block.c: assertions to the block layer permissions API

2022-03-03 Thread Emanuele Giuseppe Esposito
Now that we "covered" the three main cases where the permission API was being used under BQL (fuse, amend and invalidate_cache), we can safely assert for the permission functions implemented in block.c Signed-off-by: Emanuele Giuseppe Esposito --- block.c | 12 1 file changed, 12

[PATCH v8 22/31] include/block/snapshot: global state API + assertions

2022-03-03 Thread Emanuele Giuseppe Esposito
Snapshots run also under the BQL, so they all are in the global state API. The aiocontext lock that they hold is currently an overkill and in future could be removed. Signed-off-by: Emanuele Giuseppe Esposito --- block/snapshot.c | 28

[PATCH v8 30/31] job.h: split function pointers in JobDriver

2022-03-03 Thread Emanuele Giuseppe Esposito
The job API will be handled separately in another serie. Signed-off-by: Emanuele Giuseppe Esposito --- include/qemu/job.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/include/qemu/job.h b/include/qemu/job.h index 6e67b6977f..c105b31076 100644 ---

[PATCH v8 16/31] GS and IO CODE macros for blockjob_int.h

2022-03-03 Thread Emanuele Giuseppe Esposito
Signed-off-by: Emanuele Giuseppe Esposito --- blockjob.c | 5 + 1 file changed, 5 insertions(+) diff --git a/blockjob.c b/blockjob.c index 10815a89fe..d79a52d204 100644 --- a/blockjob.c +++ b/blockjob.c @@ -84,6 +84,7 @@ BlockJob *block_job_get(const char *id) void block_job_free(Job *job)

[PATCH v8 17/31] block.c: add assertions to static functions

2022-03-03 Thread Emanuele Giuseppe Esposito
Following the assertion derived from the API split, propagate the assertion also in the static functions. Signed-off-by: Emanuele Giuseppe Esposito --- block.c | 46 ++- block/block-backend.c | 3 +++ 2 files changed, 48 insertions(+), 1

[PATCH v8 14/31] block: introduce assert_bdrv_graph_writable

2022-03-03 Thread Emanuele Giuseppe Esposito
We want to be sure that the functions that write the child and parent list of a bs are under BQL and drain. BQL prevents from concurrent writings from the GS API, while drains protect from I/O. TODO: drains are missing in some functions using this assert. Therefore a proper assertion will fail.

[PATCH v8 24/31] block/coroutines: I/O and "I/O or GS" API

2022-03-03 Thread Emanuele Giuseppe Esposito
block coroutines functions run in different aiocontext, and are not protected by the BQL. Therefore are I/O. On the other side, generated_co_wrapper functions use BDRV_POLL_WHILE, meaning the caller can either be the main loop or a specific iothread. Signed-off-by: Emanuele Giuseppe Esposito

[PATCH v8 21/31] assertions for blockdev.h global state API

2022-03-03 Thread Emanuele Giuseppe Esposito
Signed-off-by: Emanuele Giuseppe Esposito --- block/block-backend.c | 3 +++ blockdev.c| 16 2 files changed, 19 insertions(+) diff --git a/block/block-backend.c b/block/block-backend.c index 2ab1274dfe..bf77c4a8fa 100644 --- a/block/block-backend.c +++

[PATCH v8 15/31] include/block/blockjob_int.h: split header into I/O and GS API

2022-03-03 Thread Emanuele Giuseppe Esposito
Since the I/O functions are not many, keep a single file. Also split the function pointers in BlockJobDriver. Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Stefan Hajnoczi --- include/block/blockjob_int.h | 28 1 file changed, 28 insertions(+) diff --git

[PATCH v8 08/31] block/block-backend.c: assertions for block-backend

2022-03-03 Thread Emanuele Giuseppe Esposito
All the global state (GS) API functions will check that qemu_in_main_thread() returns true. If not, it means that the safety of BQL cannot be guaranteed, and they need to be moved to I/O. Signed-off-by: Emanuele Giuseppe Esposito --- block/block-backend.c | 78

[PATCH v8 01/31] main-loop.h: introduce qemu_in_main_thread()

2022-03-03 Thread Emanuele Giuseppe Esposito
When invoked from the main loop, this function is the same as qemu_mutex_iothread_locked, and returns true if the BQL is held. When invoked from iothreads or tests, it returns true only if the current AioContext is the Main Loop. This essentially just extends qemu_mutex_iothread_locked to work

[PATCH v8 11/31] include/block/block_int: split header into I/O and global state API

2022-03-03 Thread Emanuele Giuseppe Esposito
Similarly to the previous patch, split block_int.h in block_int-io.h and block_int-global-state.h block_int-common.h contains the structures shared between the two headers, and the functions that can't be categorized as I/O or global state. Assertions are added in the next patch. Signed-off-by:

[PATCH v8 12/31] assertions for block_int global state API

2022-03-03 Thread Emanuele Giuseppe Esposito
Signed-off-by: Emanuele Giuseppe Esposito --- block.c | 15 +++ block/backup.c | 1 + block/block-backend.c | 3 +++ block/commit.c | 2 ++ block/dirty-bitmap.c| 1 + block/io.c |

[PATCH v8 13/31] IO_CODE and IO_OR_GS_CODE for block_int I/O API

2022-03-03 Thread Emanuele Giuseppe Esposito
Mark all I/O functions with IO_CODE, and all "I/O OR GS" with IO_OR_GS_CODE. Signed-off-by: Emanuele Giuseppe Esposito --- block.c | 14 +- block/block-backend.c| 2 ++ block/dirty-bitmap.c | 3 +++ block/io.c | 13

[PATCH v8 03/31] include/block/block: split header into I/O and global state API

2022-03-03 Thread Emanuele Giuseppe Esposito
block.h currently contains a mix of functions: some of them run under the BQL and modify the block layer graph, others are instead thread-safe and perform I/O in iothreads. Some others can only be called by either the main loop or the iothread running the AioContext (and not other iothreads), and

[PATCH v8 05/31] IO_CODE and IO_OR_GS_CODE for block I/O API

2022-03-03 Thread Emanuele Giuseppe Esposito
Mark all I/O functions with IO_CODE, and all "I/O OR GS" with IO_OR_GS_CODE. Signed-off-by: Emanuele Giuseppe Esposito --- block.c | 38 ++- block/dirty-bitmap.c | 1 + block/io.c | 43 ++--

[PATCH v8 09/31] IO_CODE and IO_OR_GS_CODE for block-backend I/O API

2022-03-03 Thread Emanuele Giuseppe Esposito
Mark all I/O functions with IO_CODE, and all "I/O OR GS" with IO_OR_GS_CODE. Signed-off-by: Emanuele Giuseppe Esposito --- block/block-backend.c | 58 +++ include/sysemu/block-backend-io.h | 2 ++ 2 files changed, 60 insertions(+) diff --git

[PATCH v8 07/31] include/sysemu/block-backend: split header into I/O and global state (GS) API

2022-03-03 Thread Emanuele Giuseppe Esposito
Similarly to the previous patches, split block-backend.h in block-backend-io.h and block-backend-global-state.h In addition, remove "block/block.h" include as it seems it is not necessary anymore, together with "qemu/iov.h" block-backend-common.h contains the structures shared between the two

[PATCH v8 00/31] block layer: split block APIs in global state and I/O

2022-03-03 Thread Emanuele Giuseppe Esposito
Currently, block layer APIs like block.h contain a mix of functions that are either running in the main loop and under the BQL, or are thread-safe functions and run in iothreads performing I/O. The functions running under BQL also take care of modifying the block graph, by using drain and/or

[PATCH v8 06/31] block/export/fuse.c: allow writable exports to take RESIZE permission

2022-03-03 Thread Emanuele Giuseppe Esposito
Allow writable exports to get BLK_PERM_RESIZE permission from creation, in fuse_export_create(). In this way, there is no need to give the permission in fuse_do_truncate(), which might be run in an iothread. Permissions should be set only in the main thread, so in any case if an iothread tries to

[PATCH v8 04/31] assertions for block global state API

2022-03-03 Thread Emanuele Giuseppe Esposito
All the global state (GS) API functions will check that qemu_in_main_thread() returns true. If not, it means that the safety of BQL cannot be guaranteed, and they need to be moved to I/O. Signed-off-by: Emanuele Giuseppe Esposito --- block.c| 124

[PATCH v8 02/31] main loop: macros to mark GS and I/O functions

2022-03-03 Thread Emanuele Giuseppe Esposito
Righ now, IO_CODE and IO_OR_GS_CODE are nop, as there isn't really a way to check that a function is only called in I/O. On the other side, we can use qemu_in_main_thread() to check if we are in the main loop. The usage of macros makes easy to extend them in the future without making changes in

[PATCH v2 4/4] util/event-loop-base: Introduce options to set the thread pool size

2022-03-03 Thread Nicolas Saenz Julienne
The thread pool regulates itself: when idle, it kills threads until empty, when in demand, it creates new threads until full. This behaviour doesn't play well with latency sensitive workloads where the price of creating a new thread is too high. For example, when paired with qemu's '-mlock', or

[PATCH v2 2/4] Introduce event-loop-base abstract class

2022-03-03 Thread Nicolas Saenz Julienne
Introduce the 'event-loop-base' abstract class, it'll hold the properties common to all event loops and provide the necessary hooks for their creation and maintenance. Then have iothread inherit from it. EventLoopBaseClass is defined as user creatable and provides a hook for its children to

[PATCH v2 1/4] util/thread-pool: Fix thread pool freeing locking

2022-03-03 Thread Nicolas Saenz Julienne
Upon freeing a thread pool we need to get rid of any remaining worker. This is achieved by setting the thread pool's topping flag, waking the workers up, and waiting for them to exit one by one. The problem is that currently all this process happens with the thread pool lock held, effectively

  1   2   >