Re: [Qemu-block] [PATCH] nvme: add Get/Set Feature Timestamp support

2019-05-17 Thread Heitke, Kenneth
On 5/17/2019 12:24 AM, Klaus Birkelund wrote: On Fri, May 17, 2019 at 07:35:04AM +0200, Klaus Birkelund wrote: Hi Kenneth, On Thu, May 16, 2019 at 05:24:47PM -0600, Heitke, Kenneth wrote: Hi Klaus, thank you for you review. I have one comment inline On 5/14/2019 12:02 AM, Klaus Birkelund w

Re: [Qemu-block] [PATCH v2 2/2] iotests: test external snapshot with bitmap copying

2019-05-17 Thread John Snow
On 5/17/19 11:21 AM, Vladimir Sementsov-Ogievskiy wrote: > This test shows that external snapshots and incremental backups are > friends. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > tests/qemu-iotests/254 | 52 ++ > tests/qemu-iotests/254.ou

Re: [Qemu-block] [PATCH v2 1/2] qapi: support external bitmaps in block-dirty-bitmap-merge

2019-05-17 Thread John Snow
On 5/17/19 11:21 AM, Vladimir Sementsov-Ogievskiy wrote: > Add new optional parameter making possible to merge bitmaps from > different nodes. It is needed to maintain external snapshots during > incremental backup chain history. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > qapi/bl

Re: [Qemu-block] [Qemu-devel] [PATCH v1] [RFC] qcow2: add compression type feature

2019-05-17 Thread John Snow
On 5/17/19 4:05 AM, Denis Plotnikov wrote: > > > On 17.05.2019 2:25, John Snow wrote: >> >> >> On 5/16/19 9:48 AM, Denis Plotnikov wrote: >>> The patch adds some preparation parts for incompatible compression type >>> feature into QCOW2 header that indicates that *all* compressed clusters >>>

Re: [Qemu-block] [PATCH v2 4/4] iotests: add iotest 250 for testing blockdev-backup across iothread contexts

2019-05-17 Thread John Snow
On 5/17/19 7:18 AM, Max Reitz wrote: > On 10.05.19 21:03, John Snow wrote: >> Signed-off-by: John Snow >> --- >> tests/qemu-iotests/250 | 129 + >> tests/qemu-iotests/250.out | 119 ++ >> tests/qemu-iotests/group | 1 +

Re: [Qemu-block] [Qemu-devel] [PATCH 1/2] qapi: support external bitmaps in block-dirty-bitmap-merge

2019-05-17 Thread John Snow
On 5/17/19 10:09 AM, Vladimir Sementsov-Ogievskiy wrote: > 17.05.2019 16:50, Eric Blake wrote: >> On 5/16/19 7:32 PM, John Snow wrote: >>> >>> >>> On 5/16/19 8:27 AM, Vladimir Sementsov-Ogievskiy wrote: Add new optional parameter making possible to merge bitmaps from different nodes. I

Re: [Qemu-block] [Qemu-devel] [PATCH 1/4] qapi: Support features for structs

2019-05-17 Thread Markus Armbruster
Peter Krempa writes: > On Wed, May 15, 2019 at 15:48:29 +0200, Markus Armbruster wrote: >> Kevin Wolf writes: >> > Am 18.04.2019 um 22:03 hat Markus Armbruster geschrieben: >> >> Kevin Wolf writes: > > [...] > >> > Do you expect libvirt to check a full list of all QMP commands, types, >> > etc.

Re: [Qemu-block] [PATCH] nvme: add Get/Set Feature Timestamp support

2019-05-17 Thread Heitke, Kenneth
On 5/17/2019 12:24 AM, Klaus Birkelund wrote: On Fri, May 17, 2019 at 07:35:04AM +0200, Klaus Birkelund wrote: Hi Kenneth, On Thu, May 16, 2019 at 05:24:47PM -0600, Heitke, Kenneth wrote: Hi Klaus, thank you for you review. I have one comment inline On 5/14/2019 12:02 AM, Klaus Birkelund w

[Qemu-block] [PATCH v2 0/2] bitmaps: merge bitmaps from different nodes

2019-05-17 Thread Vladimir Sementsov-Ogievskiy
Hi all! We need to copy bitmaps to new top node on external snapshot, to not break incremental backup chain. The only thing to do is to allow block-dirty-bitmap-merge to work with different nodes, here it is. v2: use 'alternate' type in qapi for specifying source bitmap instead of adding new

[Qemu-block] [PATCH v2 2/2] iotests: test external snapshot with bitmap copying

2019-05-17 Thread Vladimir Sementsov-Ogievskiy
This test shows that external snapshots and incremental backups are friends. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/254 | 52 ++ tests/qemu-iotests/254.out | 52 ++ tests/qemu-iotests/group

[Qemu-block] [PATCH v2 1/2] qapi: support external bitmaps in block-dirty-bitmap-merge

2019-05-17 Thread Vladimir Sementsov-Ogievskiy
Add new optional parameter making possible to merge bitmaps from different nodes. It is needed to maintain external snapshots during incremental backup chain history. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 22 --- block/dirty-bitmap.c | 9 +---

Re: [Qemu-block] [Qemu-devel] [PATCH v4 02/11] block: Filtered children access functions

2019-05-17 Thread Vladimir Sementsov-Ogievskiy
10.04.2019 23:20, Max Reitz wrote: > What bs->file and bs->backing mean depends on the node. For filter > nodes, both signify a node that will eventually receive all R/W > accesses. For format nodes, bs->file contains metadata and data, and > bs->backing will not receive writes -- instead, writes

[Qemu-block] [PATCH v2 6/6] file-posix: Add dynamic-auto-read-only QAPI feature

2019-05-17 Thread Kevin Wolf
In commit 23dece19da4 ('file-posix: Make auto-read-only dynamic') , auto-read-only=on changed its behaviour in file-posix for the 4.0 release. This change cannot be detected through the usual mechanisms like schema introspection. Add a new feature flag to the schema to allow libvirt to detect the p

[Qemu-block] [PATCH v2 3/6] tests/qapi-schema: Error case tests for features in structs

2019-05-17 Thread Kevin Wolf
Signed-off-by: Kevin Wolf --- tests/qapi-schema/features-bad-type.json | 3 +++ tests/qapi-schema/features-duplicate-name.json | 3 +++ tests/qapi-schema/features-missing-name.json | 3 +++ tests/qapi-schema/features-name-bad-type.json | 3 +++ tests/qapi-schema/features-no-list.json

[Qemu-block] [PATCH v2 4/6] qapi: Disentangle QAPIDoc code

2019-05-17 Thread Kevin Wolf
Documentation comment follow a certain structure: First, we have a text with a general description (called QAPIDoc.body). After this, descriptions of the arguments follow. Finally, we have part that contains various named sections. The code doesn't show this structure but just checks the right sid

[Qemu-block] [PATCH v2 0/6] file-posix: Add dynamic-auto-read-only QAPI feature

2019-05-17 Thread Kevin Wolf
This series adds optional feature lists to struct definitions in the QAPI schema and makes use of them to advertise the new behaviour of auto-read-only=on in file-posix. v2: - Check that features have well-formed names instead of just checking that they are strings - Use QAPISchemaFeature object

[Qemu-block] [PATCH v2 5/6] qapi: Allow documentation for features

2019-05-17 Thread Kevin Wolf
Features will be documented in a new part introduced by a "Features:" line, after arguments and before named sections. Signed-off-by: Kevin Wolf --- scripts/qapi/common.py | 43 ++ scripts/qapi/doc.py| 11 +++ 2 files changed, 50 insertions(+),

[Qemu-block] [PATCH v2 1/6] qapi: Support features for structs

2019-05-17 Thread Kevin Wolf
Sometimes, the behaviour of QEMU changes compatibly, but without a change in the QMP syntax (usually by allowing values or operations that previously resulted in an error). QMP clients may still need to know whether the extension is available. This allows to add a list of features to struct defini

[Qemu-block] [PATCH v2 2/6] tests/qapi-schema: Test for good feature lists in structs

2019-05-17 Thread Kevin Wolf
Signed-off-by: Kevin Wolf --- tests/qapi-schema/qapi-schema-test.json | 30 +++ tests/qapi-schema/qapi-schema-test.out | 32 + tests/qapi-schema/test-qapi.py | 4 3 files changed, 66 insertions(+) diff --git a/tests/qapi-schema/qapi-sch

Re: [Qemu-block] [Qemu-devel] [PATCH 1/2] qapi: support external bitmaps in block-dirty-bitmap-merge

2019-05-17 Thread Vladimir Sementsov-Ogievskiy
17.05.2019 16:50, Eric Blake wrote: > On 5/16/19 7:32 PM, John Snow wrote: >> >> >> On 5/16/19 8:27 AM, Vladimir Sementsov-Ogievskiy wrote: >>> Add new optional parameter making possible to merge bitmaps from >>> different nodes. It is needed to maintain external snapshots during >>> incremental ba

Re: [Qemu-block] [Qemu-devel] [PATCH 1/2] qapi: support external bitmaps in block-dirty-bitmap-merge

2019-05-17 Thread Eric Blake
On 5/16/19 7:32 PM, John Snow wrote: > > > On 5/16/19 8:27 AM, Vladimir Sementsov-Ogievskiy wrote: >> Add new optional parameter making possible to merge bitmaps from >> different nodes. It is needed to maintain external snapshots during >> incremental backup chain history. >> >> Signed-off-by: V

Re: [Qemu-block] [Qemu-devel] [PATCH 1/4] qapi: Support features for structs

2019-05-17 Thread Peter Krempa
On Wed, May 15, 2019 at 15:48:29 +0200, Markus Armbruster wrote: > Kevin Wolf writes: > > Am 18.04.2019 um 22:03 hat Markus Armbruster geschrieben: > >> Kevin Wolf writes: [...] > > Do you expect libvirt to check a full list of all QMP commands, types, > > etc. it ever uses against the schema a

Re: [Qemu-block] [Qemu-devel] [PATCH v4 02/11] block: Filtered children access functions

2019-05-17 Thread Vladimir Sementsov-Ogievskiy
07.05.2019 18:13, Max Reitz wrote: > On 07.05.19 15:30, Vladimir Sementsov-Ogievskiy wrote: >> 10.04.2019 23:20, Max Reitz wrote: >>> What bs->file and bs->backing mean depends on the node. For filter >>> nodes, both signify a node that will eventually receive all R/W >>> accesses. For format nod

Re: [Qemu-block] [PATCH 3/4] iotests: Test driver whitelisting in 093

2019-05-17 Thread Max Reitz
On 17.05.19 13:00, Kevin Wolf wrote: > Am 17.05.2019 um 11:56 hat Max Reitz geschrieben: >> null-aio may not be whitelisted. If it is not, fall back to null-co. >> This may run tests twice in the same configuration, but this is the >> simplest way to effectively skip the tests in setUp() (without

Re: [Qemu-block] [PATCH v2 4/4] iotests: add iotest 250 for testing blockdev-backup across iothread contexts

2019-05-17 Thread Max Reitz
On 10.05.19 21:03, John Snow wrote: > Signed-off-by: John Snow > --- > tests/qemu-iotests/250 | 129 + > tests/qemu-iotests/250.out | 119 ++ > tests/qemu-iotests/group | 1 + > 3 files changed, 249 insertions(+) > creat

Re: [Qemu-block] [PATCH 3/4] iotests: Test driver whitelisting in 093

2019-05-17 Thread Kevin Wolf
Am 17.05.2019 um 11:56 hat Max Reitz geschrieben: > null-aio may not be whitelisted. If it is not, fall back to null-co. > This may run tests twice in the same configuration, but this is the > simplest way to effectively skip the tests in setUp() (without changing > the output, and while having th

Re: [Qemu-block] [Qemu-devel] [PATCH 1/4] iotests: Add -display none to the qemu options

2019-05-17 Thread Alex Bennée
Max Reitz writes: > Without this argument, qemu will print an angry message about not being > able to connect to a display server if $DISPLAY is not set. For me, > that breaks iotests.supported_formats() because it thus only sees > ["Could", "not", "connect"] as the supported formats. > > Sign

Re: [Qemu-block] [Qemu-devel] [PATCH v2] migration/dirty-bitmaps: change bitmap enumeration method

2019-05-17 Thread Vladimir Sementsov-Ogievskiy
16.05.2019 22:03, John Snow wrote: > > > On 5/16/19 6:12 AM, Vladimir Sementsov-Ogievskiy wrote: >> 14.05.2019 23:19, John Snow wrote: >>> Shift from looking at every root BDS to *every* BDS. This will migrate >>> bitmaps that are attached to blockdev created nodes instead of just ones >>> attach

Re: [Qemu-block] [Qemu-devel] [PATCH 1/4] iotests: Add -display none to the qemu options

2019-05-17 Thread Thomas Huth
On 17/05/2019 11.56, Max Reitz wrote: > Without this argument, qemu will print an angry message about not being > able to connect to a display server if $DISPLAY is not set. For me, > that breaks iotests.supported_formats() because it thus only sees > ["Could", "not", "connect"] as the supported f

[Qemu-block] [PATCH 3/4] iotests: Test driver whitelisting in 093

2019-05-17 Thread Max Reitz
null-aio may not be whitelisted. If it is not, fall back to null-co. This may run tests twice in the same configuration, but this is the simplest way to effectively skip the tests in setUp() (without changing the output, and while having the respective driver in a class attribute). Signed-off-by:

[Qemu-block] [PATCH 0/4] iotests: Selfish patches

2019-05-17 Thread Max Reitz
Hi, These are some rather selfish iotests patches. The first patch helps me personally because I tend to run the tests over SSH and forget to set $DISPLAY. That makes test 139 skip the tests annotated with skip_if_unsupprted(), because iotests.py can no longer determine the list of whitelisted f

[Qemu-block] [PATCH 1/4] iotests: Add -display none to the qemu options

2019-05-17 Thread Max Reitz
Without this argument, qemu will print an angry message about not being able to connect to a display server if $DISPLAY is not set. For me, that breaks iotests.supported_formats() because it thus only sees ["Could", "not", "connect"] as the supported formats. Signed-off-by: Max Reitz --- tests/

[Qemu-block] [PATCH 2/4] iotests: Prefer null-co over null-aio

2019-05-17 Thread Max Reitz
We use null-co basically everywhere in the iotests. Unless we want to test null-aio specifically, we should use it instead (for consistency). Signed-off-by: Max Reitz --- tests/qemu-iotests/093 | 7 +++ tests/qemu-iotests/245 | 2 +- 2 files changed, 4 insertions(+), 5 deletions(-) diff --

[Qemu-block] [PATCH 4/4] iotests: Test driver whitelisting in 136

2019-05-17 Thread Max Reitz
null-aio may not be whitelisted. If it is not, fall back to null-co. This may run tests twice in the same configuration, but this is the simplest way to effectively skip the tests in setUp() (without changing the output, and while having the respective driver in a class attribute). Signed-off-by:

Re: [Qemu-block] [PATCH v1] [RFC] qcow2: add compression type feature

2019-05-17 Thread Denis Plotnikov
On 16.05.2019 17:42, Eric Blake wrote: > On 5/16/19 8:48 AM, Denis Plotnikov wrote: >> The patch adds some preparation parts for incompatible compression type >> feature into QCOW2 header that indicates that *all* compressed clusters >> must be (de)compressed using a certain compression type. >>

[Qemu-block] [PATCH 2/8] nvme: bump supported spec to 1.3

2019-05-17 Thread Klaus Birkelund Jensen
Bump the supported NVMe version to 1.3. To do so, this patch adds a number of missing 'Mandatory' features from the spec: * Support for returning a Namespace Identification Descriptor List in the Identify command (CNS 03h). * Support for the Asynchronous Event Request command. * Support

[Qemu-block] [PATCH 6/8] nvme: add support for scatter gather lists

2019-05-17 Thread Klaus Birkelund Jensen
Add partial SGL support. For now, only support a single data block or last segment descriptor. This is in line with what, for instance, SPDK currently supports. Signed-off-by: Klaus Birkelund Jensen --- block/nvme.c | 18 ++-- hw/block/nvme.c | 242 +++

[Qemu-block] [PATCH 5/8] nvme: add support for metadata

2019-05-17 Thread Klaus Birkelund Jensen
The new `ms` parameter may be used to indicate the number of metadata bytes provided per LBA. Signed-off-by: Klaus Birkelund Jensen --- hw/block/nvme.c | 31 +-- hw/block/nvme.h | 11 ++- 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/hw/block

[Qemu-block] [PATCH 3/8] nvme: simplify PRP mappings

2019-05-17 Thread Klaus Birkelund Jensen
Instead of handling both QSGs and IOVs in multiple places, simply use QSGs everywhere by assuming that the request does not involve the controller memory buffer (CMB). If the request is found to involve the CMB, convert the QSG to an IOV and issue the I/O. The QSG is converted to an IOV by the dma

[Qemu-block] [PATCH 7/8] nvme: keep a copy of the NVMe command in request

2019-05-17 Thread Klaus Birkelund Jensen
Signed-off-by: Klaus Birkelund Jensen --- hw/block/nvme.c | 4 ++-- hw/block/nvme.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index 81201a8b4834..5cd593806701 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -184,7 +184,7 @@ st

[Qemu-block] [PATCH 1/8] nvme: move device parameters to separate struct

2019-05-17 Thread Klaus Birkelund Jensen
Move device configuration parameters to separate struct to make it explicit what is configurable and what is set internally. Also, clean up some includes. Signed-off-by: Klaus Birkelund Jensen --- hw/block/nvme.c | 53 +++-- hw/block/nvme.h | 16 +

[Qemu-block] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device

2019-05-17 Thread Klaus Birkelund Jensen
Hi, This series of patches contains a number of refactorings to the emulated nvme device, adds additional features, such as support for metadata and scatter gather lists, and bumps the supported NVMe version to 1.3. Lastly, it contains a new 'ocssd' device. The motivation for the first seven patc

[Qemu-block] [PATCH 4/8] nvme: allow multiple i/o's per request

2019-05-17 Thread Klaus Birkelund Jensen
Introduce a new NvmeBlockBackendRequest and move the QEMUSGList and QEMUIOVector from the NvmeRequest. This is in preparation for metadata support and makes it easier to handle multiple block backend requests to different offsets. Signed-off-by: Klaus Birkelund Jensen --- hw/block/nvme.c

Re: [Qemu-block] [PATCH v1] [RFC] qcow2: add compression type feature

2019-05-17 Thread Denis Plotnikov
On 17.05.2019 2:25, John Snow wrote: > > > On 5/16/19 9:48 AM, Denis Plotnikov wrote: >> The patch adds some preparation parts for incompatible compression type >> feature into QCOW2 header that indicates that *all* compressed clusters >> must be (de)compressed using a certain compression type.