Re: [Qemu-block] [Qemu-devel] [PATCH v4 17/18] iotests: add test 257 for bitmap-mode backups

2019-07-10 Thread John Snow
On 7/9/19 7:25 PM, John Snow wrote: > Signed-off-by: John Snow > --- > tests/qemu-iotests/257 | 416 +++ > tests/qemu-iotests/257.out | 2247 > tests/qemu-iotests/group |1 + > 3 files changed, 2664 insertions(+) > create mode 100755

Re: [Qemu-block] [PATCH 0/4] virtio: handle zoned backing devices

2019-07-10 Thread Dmitry Fomichev
On Wed, 2019-07-10 at 23:09 +0200, Kevin Wolf wrote: > Am 10.07.2019 um 13:33 hat Paolo Bonzini geschrieben: > > On 10/07/19 13:02, Kevin Wolf wrote: > > > Hm... Actually, file-posix implements .bdrv_check_perm and could just > > > refuse attaching a parent there if it doesn't request a specific >

Re: [Qemu-block] [PATCH] LUKS: support preallocation in qemu-img

2019-07-10 Thread Max Reitz
On 10.07.19 23:24, Max Reitz wrote: > On 10.07.19 19:03, Maxim Levitsky wrote: >> preallocation=off and preallocation=metadata >> both allocate luks header only, and preallocation=falloc/full >> is passed to underlying file, with the given image size. >> >> Note that the actual preallocated size

Re: [Qemu-block] [PATCH v2 00/12] block: Fixes for concurrent block jobs

2019-07-10 Thread Max Reitz
Ping I suppose I’ll apply the patches I had in v1 if I don’t get reviews for the new patches, because some fixes are better than none. (And probably patch 2, because it’s obvious.) Max On 03.07.19 19:28, Max Reitz wrote: > This is a v2 to “block: Add BDS.never_freeze”. > > It depends on my

Re: [Qemu-block] [PATCH] LUKS: support preallocation in qemu-img

2019-07-10 Thread Max Reitz
On 10.07.19 19:03, Maxim Levitsky wrote: > preallocation=off and preallocation=metadata > both allocate luks header only, and preallocation=falloc/full > is passed to underlying file, with the given image size. > > Note that the actual preallocated size is a bit smaller due > to luks header.

Re: [Qemu-block] [PATCH 0/4] virtio: handle zoned backing devices

2019-07-10 Thread Kevin Wolf
Am 10.07.2019 um 13:33 hat Paolo Bonzini geschrieben: > On 10/07/19 13:02, Kevin Wolf wrote: > > Hm... Actually, file-posix implements .bdrv_check_perm and could just > > refuse attaching a parent there if it doesn't request a specific > > permission like BLK_PERM_SUPPORT_ZONED. That should give

Re: [Qemu-block] [PATCH] nvme: Set number of queues later in nvme_init()

2019-07-10 Thread Max Reitz
On 10.07.19 16:57, Michal Privoznik wrote: > When creating the admin queue in nvme_init() the variable that > holds the number of queues created is modified before actual > queue creation. This is a problem because if creating the queue > fails then the variable is left in inconsistent state. This

Re: [Qemu-block] [PATCH 6/8] block/backup: issue progress updates for skipped regions

2019-07-10 Thread Max Reitz
On 10.07.19 22:47, John Snow wrote: > > > On 7/10/19 4:30 PM, Max Reitz wrote: >> On 10.07.19 20:20, John Snow wrote: >>> >>> >>> On 7/10/19 12:36 PM, Max Reitz wrote: On 10.07.19 03:05, John Snow wrote: > The way bitmap backups work is by starting at 75% if it needs > to copy just

Re: [Qemu-block] [PATCH 8/8] iotests/257: test traditional sync modes

2019-07-10 Thread Max Reitz
On 10.07.19 21:00, John Snow wrote: > > > On 7/10/19 1:14 PM, Max Reitz wrote: >> On 10.07.19 03:05, John Snow wrote: >>> Signed-off-by: John Snow >>> --- >>> tests/qemu-iotests/257 | 31 + >>> tests/qemu-iotests/257.out | 3089 >>> 2 files changed,

Re: [Qemu-block] [PATCH 6/8] block/backup: issue progress updates for skipped regions

2019-07-10 Thread John Snow
On 7/10/19 4:30 PM, Max Reitz wrote: > On 10.07.19 20:20, John Snow wrote: >> >> >> On 7/10/19 12:36 PM, Max Reitz wrote: >>> On 10.07.19 03:05, John Snow wrote: The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk. >>> >>> Although there is this

Re: [Qemu-block] [PATCH 7/8] block/backup: support bitmap sync modes for non-bitmap backups

2019-07-10 Thread Max Reitz
On 10.07.19 20:32, John Snow wrote: > > > On 7/10/19 12:48 PM, Max Reitz wrote: >> On 10.07.19 03:05, John Snow wrote: >>> Accept bitmaps and sync policies for the other backup modes. >>> This allows us to do things like create a bitmap synced to a full backup >>> without a transaction, or start

Re: [Qemu-block] [PATCH 6/8] block/backup: issue progress updates for skipped regions

2019-07-10 Thread Max Reitz
On 10.07.19 20:20, John Snow wrote: > > > On 7/10/19 12:36 PM, Max Reitz wrote: >> On 10.07.19 03:05, John Snow wrote: >>> The way bitmap backups work is by starting at 75% if it needs >>> to copy just 25% of the disk. >> >> Although there is this comment: >> >>> /* TODO

Re: [Qemu-block] [PATCH 4/8] block/backup: hoist bitmap check into QMP interface

2019-07-10 Thread Max Reitz
On 10.07.19 19:57, John Snow wrote: > > > On 7/10/19 12:11 PM, Max Reitz wrote: >> On 10.07.19 03:05, John Snow wrote: >>> This is nicer to do in the unified QMP interface that we have now, >>> because it lets us use the right terminology back at the user. >>> >>> Signed-off-by: John Snow >>>

Re: [Qemu-block] [PATCH 3/8] iotests/257: Refactor backup helpers

2019-07-10 Thread Max Reitz
On 10.07.19 19:52, John Snow wrote: > > > On 7/10/19 12:04 PM, Max Reitz wrote: >> On 10.07.19 03:05, John Snow wrote: >>> This test needs support for non-bitmap backups and missing or >>> unspecified bitmap sync modes, so rewrite the helpers to be a little >>> more generic. >>> >>>

[Qemu-block] [PULL 2/3] sphinx: add qmp_lexer

2019-07-10 Thread John Snow
Sphinx, through Pygments, does not like annotated json examples very much. In some versions of Sphinx (1.7), it will render the non-json portions of code blocks in red, but in newer versions (2.0) it will throw an exception and not highlight the block at all. Though we can suppress this warning,

[Qemu-block] [PULL 1/3] docs/interop/bitmaps.rst: Fix typos

2019-07-10 Thread John Snow
Pygments and Sphinx get pickier all the time; Sphinx 2.1+ now catches these errors. Signed-off-by: John Snow Reported-by: Aarushi Mehta Reviewed-by: Vladimir Sementsov-Ogievskiy Message-id: 20190603214653.29369-2-js...@redhat.com Signed-off-by: John Snow --- docs/interop/bitmaps.rst | 4 ++--

[Qemu-block] [PULL 0/3] Bitmaps patches

2019-07-10 Thread John Snow
The following changes since commit 6df2cdf44a82426f7a59dcb03f0dd2181ed7fdfa: Update version for v4.1.0-rc0 release (2019-07-09 17:21:53 +0100) are available in the Git repository at: https://github.com/jnsnow/qemu.git tags/bitmaps-pull-request for you to fetch changes up to

[Qemu-block] [PULL 3/3] docs/bitmaps: use QMP lexer instead of json

2019-07-10 Thread John Snow
The annotated style json we use in QMP documentation is not strict json and depending on the version of Sphinx (2.0+) or Pygments installed, might cause the build to fail. Use the new QMP lexer. Further, some versions of Sphinx can not apply custom lexers to "code" directives and require the use

Re: [Qemu-block] [PATCH 8/8] iotests/257: test traditional sync modes

2019-07-10 Thread John Snow
On 7/10/19 1:14 PM, Max Reitz wrote: > On 10.07.19 03:05, John Snow wrote: >> Signed-off-by: John Snow >> --- >> tests/qemu-iotests/257 | 31 + >> tests/qemu-iotests/257.out | 3089 >> 2 files changed, 3120 insertions(+) > > Oof. > Yeah, it's... a

Re: [Qemu-block] [PATCH 7/8] block/backup: support bitmap sync modes for non-bitmap backups

2019-07-10 Thread John Snow
On 7/10/19 12:48 PM, Max Reitz wrote: > On 10.07.19 03:05, John Snow wrote: >> Accept bitmaps and sync policies for the other backup modes. >> This allows us to do things like create a bitmap synced to a full backup >> without a transaction, or start a resumable backup process. >> >> Some

Re: [Qemu-block] [PATCH 6/8] block/backup: issue progress updates for skipped regions

2019-07-10 Thread John Snow
On 7/10/19 12:36 PM, Max Reitz wrote: > On 10.07.19 03:05, John Snow wrote: >> The way bitmap backups work is by starting at 75% if it needs >> to copy just 25% of the disk. > > Although there is this comment: > >> /* TODO job_progress_set_remaining() would make more sense */ > > So... > >>

Re: [Qemu-block] [PATCH 5/8] iotests/257: test API failures

2019-07-10 Thread John Snow
On 7/10/19 12:22 PM, Max Reitz wrote: > On 10.07.19 03:05, John Snow wrote: >> Signed-off-by: John Snow >> --- >> tests/qemu-iotests/257 | 69 +++ >> tests/qemu-iotests/257.out | 85 ++ >> 2 files changed, 154 insertions(+)

Re: [Qemu-block] [PATCH 4/8] block/backup: hoist bitmap check into QMP interface

2019-07-10 Thread John Snow
On 7/10/19 12:11 PM, Max Reitz wrote: > On 10.07.19 03:05, John Snow wrote: >> This is nicer to do in the unified QMP interface that we have now, >> because it lets us use the right terminology back at the user. >> >> Signed-off-by: John Snow >> --- >> block/backup.c | 13 - >>

Re: [Qemu-block] [PATCH 3/8] iotests/257: Refactor backup helpers

2019-07-10 Thread John Snow
On 7/10/19 12:04 PM, Max Reitz wrote: > On 10.07.19 03:05, John Snow wrote: >> This test needs support for non-bitmap backups and missing or >> unspecified bitmap sync modes, so rewrite the helpers to be a little >> more generic. >> >> Signed-off-by: John Snow >> --- >> tests/qemu-iotests/257

Re: [Qemu-block] [PATCH 2/8] iotests/257: add EmulatedBitmap class

2019-07-10 Thread John Snow
On 7/10/19 11:47 AM, Max Reitz wrote: > On 10.07.19 03:05, John Snow wrote: >> Represent a bitmap with an object that we can mark and clear bits in. >> This makes it easier to manage partial writes when we don't write a >> full group's worth of patterns before an error. >> >> Signed-off-by:

Re: [Qemu-block] [PATCH 1/8] iotests/257: add Pattern class

2019-07-10 Thread John Snow
On 7/10/19 12:26 PM, Max Reitz wrote: > On 10.07.19 03:05, John Snow wrote: >> Just kidding, this is easier to manage with a full class instead of a >> namedtuple. >> >> Signed-off-by: John Snow >> --- >> tests/qemu-iotests/257 | 58 +++--- >> 1 file

Re: [Qemu-block] [PATCH v4 00/18] bitmaps: introduce 'bitmap' sync mode

2019-07-10 Thread John Snow
On 7/10/19 10:48 AM, Max Reitz wrote: > On 10.07.19 01:25, John Snow wrote: >> This series adds a new "BITMAP" sync mode that is meant to replace the >> existing "INCREMENTAL" sync mode. > > So who’s going to take it? :-) > > get_maintainer.pl says I’m responsible for 14/18 patches, and you

Re: [Qemu-block] [PATCH 8/8] iotests/257: test traditional sync modes

2019-07-10 Thread Max Reitz
On 10.07.19 03:05, John Snow wrote: > Signed-off-by: John Snow > --- > tests/qemu-iotests/257 | 31 + > tests/qemu-iotests/257.out | 3089 > 2 files changed, 3120 insertions(+) Oof. > diff --git a/tests/qemu-iotests/257 b/tests/qemu-iotests/257 >

Re: [Qemu-block] [Qemu-devel] [PATCH] nvme: Set number of queues later in nvme_init()

2019-07-10 Thread Maxim Levitsky
On Wed, 2019-07-10 at 17:34 +0200, Philippe Mathieu-Daudé wrote: > On 7/10/19 4:57 PM, Michal Privoznik wrote: > > When creating the admin queue in nvme_init() the variable that > > holds the number of queues created is modified before actual > > queue creation. This is a problem because if

[Qemu-block] [PATCH] LUKS: support preallocation in qemu-img

2019-07-10 Thread Maxim Levitsky
preallocation=off and preallocation=metadata both allocate luks header only, and preallocation=falloc/full is passed to underlying file, with the given image size. Note that the actual preallocated size is a bit smaller due to luks header. Fixes:

Re: [Qemu-block] [PATCH 7/8] block/backup: support bitmap sync modes for non-bitmap backups

2019-07-10 Thread Max Reitz
On 10.07.19 03:05, John Snow wrote: > Accept bitmaps and sync policies for the other backup modes. > This allows us to do things like create a bitmap synced to a full backup > without a transaction, or start a resumable backup process. > > Some combinations don't make sense, though: > > - NEVER

Re: [Qemu-block] [PATCH 6/8] block/backup: issue progress updates for skipped regions

2019-07-10 Thread Max Reitz
On 10.07.19 03:05, John Snow wrote: > The way bitmap backups work is by starting at 75% if it needs > to copy just 25% of the disk. Although there is this comment: > /* TODO job_progress_set_remaining() would make more sense */ So... > The way sync=top currently works, however, is to start at

Re: [Qemu-block] [PATCH 5/8] iotests/257: test API failures

2019-07-10 Thread Max Reitz
On 10.07.19 03:05, John Snow wrote: > Signed-off-by: John Snow > --- > tests/qemu-iotests/257 | 69 +++ > tests/qemu-iotests/257.out | 85 ++ > 2 files changed, 154 insertions(+) > > diff --git a/tests/qemu-iotests/257

Re: [Qemu-block] [PATCH 1/8] iotests/257: add Pattern class

2019-07-10 Thread Max Reitz
On 10.07.19 03:05, John Snow wrote: > Just kidding, this is easier to manage with a full class instead of a > namedtuple. > > Signed-off-by: John Snow > --- > tests/qemu-iotests/257 | 58 +++--- > 1 file changed, 32 insertions(+), 26 deletions(-) > > diff

Re: [Qemu-block] [PATCH 4/8] block/backup: hoist bitmap check into QMP interface

2019-07-10 Thread Max Reitz
On 10.07.19 03:05, John Snow wrote: > This is nicer to do in the unified QMP interface that we have now, > because it lets us use the right terminology back at the user. > > Signed-off-by: John Snow > --- > block/backup.c | 13 - > blockdev.c | 10 ++ > 2 files changed,

Re: [Qemu-block] [PATCH 3/8] iotests/257: Refactor backup helpers

2019-07-10 Thread Max Reitz
On 10.07.19 03:05, John Snow wrote: > This test needs support for non-bitmap backups and missing or > unspecified bitmap sync modes, so rewrite the helpers to be a little > more generic. > > Signed-off-by: John Snow > --- > tests/qemu-iotests/257 | 46 + >

Re: [Qemu-block] [PATCH 2/8] iotests/257: add EmulatedBitmap class

2019-07-10 Thread Max Reitz
On 10.07.19 03:05, John Snow wrote: > Represent a bitmap with an object that we can mark and clear bits in. > This makes it easier to manage partial writes when we don't write a > full group's worth of patterns before an error. > > Signed-off-by: John Snow > --- > tests/qemu-iotests/257 | 125

Re: [Qemu-block] [Qemu-devel] [PATCH] nvme: Set number of queues later in nvme_init()

2019-07-10 Thread Philippe Mathieu-Daudé
On 7/10/19 4:57 PM, Michal Privoznik wrote: > When creating the admin queue in nvme_init() the variable that > holds the number of queues created is modified before actual > queue creation. This is a problem because if creating the queue > fails then the variable is left in inconsistent state.

Re: [Qemu-block] [PATCH v3] block/rbd: implement .bdrv_get_allocated_file_size callback

2019-07-10 Thread Stefano Garzarella
On Tue, Jul 09, 2019 at 09:42:14PM -0400, Jason Dillaman wrote: > On Tue, Jul 9, 2019 at 11:32 AM Max Reitz wrote: > > On 09.07.19 15:09, Stefano Garzarella wrote: > > > > > > Max, Jason, thanks for the details! > > > > > > If you agree, I'll try to implement something similar, iterating on all >

Re: [Qemu-block] [PATCH 1/8] iotests/257: add Pattern class

2019-07-10 Thread Max Reitz
On 10.07.19 03:05, John Snow wrote: > Just kidding, this is easier to manage with a full class instead of a > namedtuple. > > Signed-off-by: John Snow > --- > tests/qemu-iotests/257 | 58 +++--- > 1 file changed, 32 insertions(+), 26 deletions(-)

[Qemu-block] [PATCH] nvme: Set number of queues later in nvme_init()

2019-07-10 Thread Michal Privoznik
When creating the admin queue in nvme_init() the variable that holds the number of queues created is modified before actual queue creation. This is a problem because if creating the queue fails then the variable is left in inconsistent state. This was actually observed when I tried to hotplug a

Re: [Qemu-block] [PATCH v4 00/18] bitmaps: introduce 'bitmap' sync mode

2019-07-10 Thread Max Reitz
On 10.07.19 01:25, John Snow wrote: > This series adds a new "BITMAP" sync mode that is meant to replace the > existing "INCREMENTAL" sync mode. So who’s going to take it? :-) get_maintainer.pl says I’m responsible for 14/18 patches, and you are for 9/18. But you did prefix the title with

Re: [Qemu-block] [PATCH v4 17/18] iotests: add test 257 for bitmap-mode backups

2019-07-10 Thread Max Reitz
On 10.07.19 01:25, John Snow wrote: > Signed-off-by: John Snow > --- > tests/qemu-iotests/257 | 416 +++ > tests/qemu-iotests/257.out | 2247 > tests/qemu-iotests/group |1 + > 3 files changed, 2664 insertions(+) > create mode 100755

Re: [Qemu-block] [PATCH v4 11/18] block/backup: upgrade copy_bitmap to BdrvDirtyBitmap

2019-07-10 Thread Max Reitz
On 10.07.19 01:25, John Snow wrote: > This simplifies some interface matters; namely the initialization and > (later) merging the manifest back into the sync_bitmap if it was > provided. > > Signed-off-by: John Snow > --- > block/backup.c | 81 ++

Re: [Qemu-block] [PATCH v4 09/18] block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal

2019-07-10 Thread Max Reitz
On 10.07.19 01:25, John Snow wrote: > I'm surprised it didn't come up sooner, but sometimes we have a +busy > bitmap as a source. This is dangerous from the QMP API, but if we are > the owner that marked the bitmap busy, it's safe to merge it using it as > a read only source. > > It is not safe

Re: [Qemu-block] [Qemu-devel] [PATCH v2 1/1] raw-posix.c - use max transfer length / max segement count only for SCSI passthrough

2019-07-10 Thread Maxim Levitsky
On Thu, 2019-07-04 at 15:43 +0300, Maxim Levitsky wrote: > Regular kernel block devices (/dev/sda*, /dev/nvme*, etc) don't have > max segment size/max segment count hardware requirements exposed > to the userspace, but rather the kernel block layer > takes care to split the incoming requests that

Re: [Qemu-block] [PATCH 0/4] virtio: handle zoned backing devices

2019-07-10 Thread Paolo Bonzini
On 10/07/19 13:02, Kevin Wolf wrote: > Hm... Actually, file-posix implements .bdrv_check_perm and could just > refuse attaching a parent there if it doesn't request a specific > permission like BLK_PERM_SUPPORT_ZONED. That should give us the > whitelist semantics through existing infrastructure.

Re: [Qemu-block] [PATCH 0/4] virtio: handle zoned backing devices

2019-07-10 Thread Kevin Wolf
Am 10.07.2019 um 12:09 hat Paolo Bonzini geschrieben: > On 09/07/19 22:38, Dmitry Fomichev wrote: > > Currently, attaching zoned block devices (i.e. storage devices > > compliant to ZAC/ZBC standards) using several virtio methods doesn't > > work - the zoned devices appear as regular block devices

Re: [Qemu-block] [PATCH 0/4] virtio: handle zoned backing devices

2019-07-10 Thread Paolo Bonzini
On 09/07/19 22:38, Dmitry Fomichev wrote: > Currently, attaching zoned block devices (i.e. storage devices > compliant to ZAC/ZBC standards) using several virtio methods doesn't > work - the zoned devices appear as regular block devices at the guest. > This may cause unexpected i/o errors and,

Re: [Qemu-block] ANNOUNCE: Tool for diffing and backing up images and block devices

2019-07-10 Thread Kevin Wolf
Am 09.07.2019 um 18:23 hat Dan Shearer geschrieben: > ddb is now available at https://github.com/gladserv/ddb and is intended for > comparing block devices. The definition of "device" is very broad and can be > custom-defined. > > ddb also knows about sparse files. How does this compare to