Re: [PATCH 0/6] hw/nvme: enhanced protection information (64-bit guard)

2022-02-16 Thread Keith Busch
On Mon, Feb 14, 2022 at 01:30:23PM +0100, Klaus Jensen wrote: > From: Klaus Jensen > > This adds support for one possible new protection information format > introduced in TP4068 (and integrated in NVMe 2.0): the 64-bit CRC guard > and 48-bit reference tag. This version does not support storage t

Re: [PATCH 6/6] hw/nvme: 64-bit pi support

2022-02-16 Thread Keith Busch
On Mon, Feb 14, 2022 at 01:30:29PM +0100, Klaus Jensen wrote: > @@ -384,6 +389,12 @@ static int nvme_ns_check_constraints(NvmeNamespace *ns, > Error **errp) > return -1; > } > > +if (ns->params.pif != NVME_PI_GUARD_16 && > +ns->params.pif != NVME_PI_GUARD_64) { > +

Re: [PATCH v2] tests/qemu-iotests: Rework the checks and spots using GNU sed

2022-02-16 Thread Eric Blake
On Wed, Feb 16, 2022 at 01:54:54PM +0100, Thomas Huth wrote: > Instead of failing the iotests if GNU sed is not available (or skipping > them completely in the check-block.sh script), it would be better to > simply skip the bash-based tests that rely on GNU sed, so that the other > tests could stil

[PATCH v4 13/18] iotests/image-fleecing: add test-case for fleecing format node

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 64 - tests/qemu-iotests/tests/image-fleecing.out | 76 - 2 files changed, 120 insertions(+), 20 deletions(-) diff --git a/tests/qemu-iotests/tests/image-fleecing b/

[PATCH v4 11/18] block: introduce snapshot-access filter

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
The filter simply utilizes snapshot-access API of underlying block node. In further patches we want to use it like this: [guest] [NBD export] || | root | root v file v [copy-before-write]<--[sna

[PATCH v4 09/18] block/reqlist: add reqlist_wait_all()

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Add function to wait for all intersecting requests. To be used in the further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Nikita Lapshin Reviewed-by: Hanna Reitz --- include/block/reqlist.h | 8 block/reqlist.c | 8 2 files changed, 16 insertions(

[PATCH v4 08/18] block/dirty-bitmap: introduce bdrv_dirty_bitmap_status()

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Add a convenient function similar with bdrv_block_status() to get status of dirty bitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 12 block/dirty-bitmap.c | 6 ++ util/hbitmap.c |

[PATCH v4 12/18] block: copy-before-write: realize snapshot-access API

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Current scheme of image fleecing looks like this: [guest][NBD export] | | |root | root v v [copy-before-write] -> [temp.qcow2] | target | |file

[PATCH v4 16/18] block: blk_root(): return non-const pointer

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
In the following patch we'll want to pass blk children to block-copy. Const pointers are not enough. So, return non const pointer from blk_root(). Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/sysemu/block-backend.h | 2 +- block/block-backend.c | 2 +- 2 files changed, 2 inse

[PATCH v4 18/18] iotests/image-fleecing: test push backup with fleecing

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 121 ++-- tests/qemu-iotests/tests/image-fleecing.out | 63 ++ 2 files changed, 152 insertions(+), 32 deletions(-) diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/

[PATCH v4 17/18] qapi: backup: add immutable-source parameter

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
We are on the way to implement internal-backup with fleecing scheme, which includes backup job copying from fleecing block driver node (which is target of copy-before-write filter) to final target of backup. This job doesn't need own filter, as fleecing block driver node is a kind of snapshot, it's

[PATCH v4 05/18] block/block-copy: add block_copy_reset()

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Split block_copy_reset() out of block_copy_reset_unallocated() to be used separately later. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/block-copy.h | 1 + block/block-copy.c | 21 + 2 files changed, 14 insertions(+), 8 del

[PATCH v4 10/18] block/io: introduce block driver snapshot-access API

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Add new block driver handlers and corresponding generic wrappers. It will be used to allow copy-before-write filter to provide reach fleecing interface in further commit. In future this approach may be used to allow reading qcow2 interanal snaphots, for example to export them through NBD. Signed-

[PATCH v4 04/18] block/copy-before-write: add bitmap open parameter

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
This brings "incremental" mode to copy-before-write filter: user can specify bitmap so that filter will copy only "dirty" areas. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 10 +++- block/copy-before-write.c | 51 ++- 2 file

[PATCH v4 15/18] iotests/image-fleecing: add test case with bitmap

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Note that reads zero areas (not dirty in the bitmap) fails, that's correct. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 32 ++-- tests/qemu-iotests/tests/image-fleecing.out | 84 + 2 files changed, 108 insertions(+), 8 del

[PATCH v4 14/18] iotests.py: add qemu_io_pipe_and_status()

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Add helper that returns both status and output, to be used in the following commit Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 4 1 file changed, 4 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 6ba65eb1ff.

[PATCH v4 07/18] block/reqlist: reqlist_find_conflict(): use ranges_overlap()

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Let's reuse convenient helper. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/reqlist.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/block/reqlist.c b/block/reqlist.c index 5e320ba649..09fecbd48c 100644 --- a/block/reqlist.c +++ b/block/reqlist.c @@ -13,6 +13,7 @@

[PATCH v4 06/18] block: intoduce reqlist

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
Split intersecting-requests functionality out of block-copy to be reused in copy-before-write filter. Note: while being here, fix tiny typo in MAINTAINERS. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/reqlist.h | 67 +++ block/block-copy.c | 116 ++

[PATCH v4 02/18] block/dirty-bitmap: bdrv_merge_dirty_bitmap(): add return value

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
That simplifies handling failure in existing code and in further new usage of bdrv_merge_dirty_bitmap(). Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/dirty-bitmap.h| 2 +- block/dirty-bitmap.c| 9 +++-- block/monitor/bitmap-qmp-cmds.

[PATCH v4 03/18] block/block-copy: block_copy_state_new(): add bitmap parameter

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
This will be used in the following commit to bring "incremental" mode to copy-before-write filter. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block-copy.h | 1 + block/block-copy.c | 14 +- block/copy-before-write.c | 2 +- 3 files changed, 15 insertions

[PATCH v4 01/18] block/block-copy: move copy_bitmap initialization to block_copy_state_new()

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
We are going to complicate bitmap initialization in the further commit. And in future, backup job will be able to work without filter (when source is immutable), so we'll need same bitmap initialization in copy-before-write filter and in backup job. So, it's reasonable to do it in block-copy. Note

[PATCH v4 00/18] Make image fleecing more usable

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
v4: Switch to new fleecing scheme, more native for Qemu's block-layer, see new patches 10-12 for details. 01,02: add Hanna's r-b 03: add const, add forgotten bdrv_release_dirty_bitmap() 04: rewrite bitmap parameter parsing 05: add Hanna's r-b 06: tiny wording fixes 07: new 08: fix comments, im

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
16.02.2022 20:14, Nir Soffer wrote: On Wed, Feb 16, 2022 at 10:08 AM Vladimir Sementsov-Ogievskiy wrote: 16.02.2022 02:24, Eric Blake wrote: On Tue, Feb 15, 2022 at 09:23:36PM +0200, Nir Soffer wrote: On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: According to the NBD spec, a server ad

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Nir Soffer
On Wed, Feb 16, 2022 at 10:08 AM Vladimir Sementsov-Ogievskiy wrote: > > 16.02.2022 02:24, Eric Blake wrote: > > On Tue, Feb 15, 2022 at 09:23:36PM +0200, Nir Soffer wrote: > >> On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: > >> > >>> According to the NBD spec, a server advertising > >>> NBD_

Re: [PATCH 1/4] python/utils: add enboxify() text decoration utility

2022-02-16 Thread Hanna Reitz
On 16.02.22 17:16, John Snow wrote: On Tue, Feb 15, 2022, 6:57 PM Philippe Mathieu-Daudé wrote: On 16/2/22 00:53, John Snow wrote: > On Tue, Feb 15, 2022 at 5:55 PM Eric Blake wrote: >> >> On Tue, Feb 15, 2022 at 05:08:50PM -0500, John Snow wrote: >> print(enboxi

Re: [PATCH 1/4] python/utils: add enboxify() text decoration utility

2022-02-16 Thread John Snow
On Tue, Feb 15, 2022, 6:57 PM Philippe Mathieu-Daudé wrote: > On 16/2/22 00:53, John Snow wrote: > > On Tue, Feb 15, 2022 at 5:55 PM Eric Blake wrote: > >> > >> On Tue, Feb 15, 2022 at 05:08:50PM -0500, John Snow wrote: > >> print(enboxify(msg, width=72, name="commit message")) > >>> ┏━ comm

Re: [PATCH] tests/qemu-iotests: Rework the checks and spots using GNU sed

2022-02-16 Thread Eric Blake
On Wed, Feb 16, 2022 at 12:39:06PM +0100, Thomas Huth wrote: > > > -$SED -re 's/[0-9]{4}-[0-9]{2}-[0-9]{2} > > > [0-9]{2}:[0-9]{2}:[0-9]{2}/-mm-dd hh:mm:ss/' > > > +gsed -re 's/[0-9]{4}-[0-9]{2}-[0-9]{2} > > > [0-9]{2}:[0-9]{2}:[0-9]{2}/-mm-dd hh:mm:ss/' > > > > GNU sed recommend

Re: [PATCH v2] tests/qemu-iotests: Rework the checks and spots using GNU sed

2022-02-16 Thread Philippe Mathieu-Daudé via
On 16/2/22 13:54, Thomas Huth wrote: Instead of failing the iotests if GNU sed is not available (or skipping them completely in the check-block.sh script), it would be better to simply skip the bash-based tests that rely on GNU sed, so that the other tests could still be run. Thus we now explicit

Re: [PATCH v4 00/15] hw/nvme: SR-IOV with Virtualization Enhancements

2022-02-16 Thread Lukasz Maniak
On Fri, Feb 11, 2022 at 08:26:10AM +0100, Klaus Jensen wrote: > On Jan 26 18:11, Lukasz Maniak wrote: > > Changes since v3: > > - Addressed comments to review on pcie: Add support for Single Root I/O > > Virtualization (SR/IOV) > > - Fixed issues reported by checkpatch.pl > > > > Knut Omang (2):

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Markus Armbruster
Eric Blake writes: > According to the NBD spec, a server advertising > NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will > not see any cache inconsistencies: when properly separated by a single > flush, actions performed by one client will be visible to another > client, rega

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Nir Soffer
On Wed, Feb 16, 2022 at 12:13 PM Richard W.M. Jones wrote: > On Tue, Feb 15, 2022 at 05:24:14PM -0600, Eric Blake wrote: > > Oh. The QMP command (which is immediately visible through > > nbd-server-add/block-storage-add to qemu and qemu-storage-daemon) > > gains "multi-conn":"on", but you may be

[PATCH v2] tests/qemu-iotests: Rework the checks and spots using GNU sed

2022-02-16 Thread Thomas Huth
Instead of failing the iotests if GNU sed is not available (or skipping them completely in the check-block.sh script), it would be better to simply skip the bash-based tests that rely on GNU sed, so that the other tests could still be run. Thus we now explicitely use "gsed" (either as direct progra

Re: [PATCH] tests/qemu-iotests: Rework the checks and spots using GNU sed

2022-02-16 Thread Thomas Huth
On 15/02/2022 23.10, Eric Blake wrote: On Tue, Feb 15, 2022 at 02:20:31PM +0100, Thomas Huth wrote: Instead of failing the iotests if GNU sed is not available (or skipping them completely in the check-block.sh script), it would be better to simply skip the bash-based tests that rely on GNU sed,

[PATCH v2 3/3] iotests/graph-changes-while-io: New test

2022-02-16 Thread Hanna Reitz
Test the following scenario: 1. Some block node (null-co) attached to a user (here: NBD server) that performs I/O and keeps the node in an I/O thread 2. Repeatedly run blockdev-add/blockdev-del to add/remove an overlay to/from that node Each blockdev-add triggers bdrv_refresh_limits(), and b

[PATCH v2 2/3] iotests: Allow using QMP with the QSD

2022-02-16 Thread Hanna Reitz
Add a parameter to optionally open a QMP connection when creating a QemuStorageDaemon instance. Signed-off-by: Hanna Reitz --- tests/qemu-iotests/iotests.py | 32 +++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/

[PATCH v2 1/3] block: Make bdrv_refresh_limits() non-recursive

2022-02-16 Thread Hanna Reitz
bdrv_refresh_limits() recurses down to the node's children. That does not seem necessary: When we refresh limits on some node, and then recurse down and were to change one of its children's BlockLimits, then that would mean we noticed the changed limits by pure chance. The fact that we refresh th

[PATCH v2 0/3] block: Make bdrv_refresh_limits() non-recursive

2022-02-16 Thread Hanna Reitz
Hi, v1 with detailed reasoning: https://lists.nongnu.org/archive/html/qemu-block/2022-02/msg00508.html This series makes bdrv_refresh_limits() non-recursive so that it is sufficient for callers to ensure that the node on which they call it will not receive concurrent I/O requests (instead of ensu

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Richard W.M. Jones
On Tue, Feb 15, 2022 at 05:24:14PM -0600, Eric Blake wrote: > Oh. The QMP command (which is immediately visible through > nbd-server-add/block-storage-add to qemu and qemu-storage-daemon) > gains "multi-conn":"on", but you may be right that qemu-nbd would want > a command line option (either that,

Re: [PATCH 2/3] iotests: Allow using QMP with the QSD

2022-02-16 Thread Hanna Reitz
On 15.02.22 23:19, Eric Blake wrote: On Tue, Feb 15, 2022 at 02:57:26PM +0100, Hanna Reitz wrote: Add a parameter to optionally open a QMP connection when creating a QemuStorageDaemon instance. Signed-off-by: Hanna Reitz --- tests/qemu-iotests/iotests.py | 29 -

Re: [PATCH 3/3] iotests/graph-changes-while-io: New test

2022-02-16 Thread Hanna Reitz
On 15.02.22 23:22, Eric Blake wrote: On Tue, Feb 15, 2022 at 02:57:27PM +0100, Hanna Reitz wrote: Test the following scenario: 1. Some block node (null-co) attached to a user (here: NBD server) that performs I/O and keeps the node in an I/O thread 2. Repeatedly run blockdev-add/blockdev-del

Re: [PATCH 1/4] block: bdrv_merge_dirty_bitmap: add return value

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
I forget that I already sent it in other series: [PATCH v3 02/19] block/dirty-bitmap: bdrv_merge_dirty_bitmap(): add return value "[PATCH v3 02/19] block/dirty-bitmap: bdrv_merge_dirty_bitmap(): add return value" is a bit better as it adds a comment. And has Hanna's r-b 15.02.2022 20:53, Vladi

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Vladimir Sementsov-Ogievskiy
16.02.2022 02:24, Eric Blake wrote: On Tue, Feb 15, 2022 at 09:23:36PM +0200, Nir Soffer wrote: On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: According to the NBD spec, a server advertising NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsiste