Re: [Qemu-block] [PATCH v6 3/3] block/stream: introduce a bottom node

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
28.05.2019 20:33, Max Reitz wrote: > On 06.05.19 17:34, Vladimir Sementsov-Ogievskiy wrote: >> From: Andrey Shinkevich >> >> The bottom node is the intermediate block device that has the base as its >> backing image. It is used instead of the base node while a block stream >> job is running to avo

Re: [Qemu-block] [PATCH] pr-manager-helper: fix pr process been killed when reconectting

2019-05-29 Thread Michal Privoznik
On 5/28/19 7:45 PM, Paolo Bonzini wrote: On 28/05/19 15:06, Jie Wang wrote: if pr-helper been killed and qemu send disconnect event to libvirt and libvirt started a new pr-helper process, the new pr-heleper been killed again when qemu is connectting to the new pr-helper, qemu will never send dis

Re: [Qemu-block] [PATCH v0] qemu-io: add pattern file for write command

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
29.05.2019 9:48, Denis Plotnikov wrote: > The patch allows to provide a pattern file for write > command. There was no similar ability before. > > Signed-off-by: Denis Plotnikov > --- > qemu-io-cmds.c | 58 ++ > 1 file changed, 54 insertions(+),

Re: [Qemu-block] [PATCH] pr-manager-helper: fix pr process been killed when reconectting

2019-05-29 Thread Jie Wang
Hi, Paolo and Michal: Let me add some details about this problem. reappear steps: 1. in Host, execute the following command many times quickly: "ps aux|grep helper|grep -v grep|grep -v qemu-kvm|awk '{print $2}';ps aux|grep helper|grep -v grep|grep -v qemu-kvm|awk '{print $2}'|xargs -n1 kill -

Re: [Qemu-block] [PATCH v0] qemu-io: add pattern file for write command

2019-05-29 Thread Denis Plotnikov
On 29.05.2019 11:11, Vladimir Sementsov-Ogievskiy wrote: > 29.05.2019 9:48, Denis Plotnikov wrote: >> The patch allows to provide a pattern file for write >> command. There was no similar ability before. >> >> Signed-off-by: Denis Plotnikov >> --- >>qemu-io-cmds.c | 58 ++

[Qemu-block] [PATCH v1] qemu-io: add pattern file for write command

2019-05-29 Thread Denis Plotnikov
The patch allows to provide a pattern file for write command. There was no similar ability before. Signed-off-by: Denis Plotnikov --- qemu-io-cmds.c | 58 ++ 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/qemu-io-cmds.c b/qemu-io-cm

Re: [Qemu-block] [Qemu-devel] [PATCH 3/3] block/qcow2-bitmap: rewrite bitmap reopening logic

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
29.05.2019 2:24, John Snow wrote: > > > On 5/23/19 11:47 AM, Vladimir Sementsov-Ogievskiy wrote: >> Current logic >> = >> >> Reopen rw -> ro: >> >> Store bitmaps and release BdrvDirtyBitmap's. >> >> Reopen ro -> rw: >> >> Load bitmap list >> Skip bitmaps which for which we don't have

Re: [Qemu-block] [PATCH] pr-manager-helper: fix pr process been killed when reconectting

2019-05-29 Thread Paolo Bonzini
On 29/05/19 09:33, Michal Privoznik wrote: > On 5/28/19 7:45 PM, Paolo Bonzini wrote: >> On 28/05/19 15:06, Jie Wang wrote: >>> if pr-helper been killed and qemu send disconnect event to libvirt >>> and libvirt started a new pr-helper process, the new pr-heleper >>> been killed again when qemu is c

Re: [Qemu-block] [PATCH] pr-manager-helper: fix pr process been killed when reconectting

2019-05-29 Thread Paolo Bonzini
On 29/05/19 10:37, Jie Wang wrote: > when the problem appeared: > > 1. qemu will initialize pr-helper and connect to it cyclically, but > always failed because no running pr-helper process to connect. > > 2. libvirt will always waiting for connected event, but will never to > start new pr-helper

Re: [Qemu-block] [PATCH v0 2/3] qcow2: add compression type processing

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
28.05.2019 17:37, Denis Plotnikov wrote: > With the patch, qcow2 is able to process image compression type > defined in the image header and choose the corresponding method > for clusters compressing. > > Also, it rework the cluster compression code for adding more > compression types. > > Signed

Re: [Qemu-block] [PATCH v6 3/3] block/stream: introduce a bottom node

2019-05-29 Thread Max Reitz
On 29.05.19 09:34, Vladimir Sementsov-Ogievskiy wrote: > 28.05.2019 20:33, Max Reitz wrote: >> On 06.05.19 17:34, Vladimir Sementsov-Ogievskiy wrote: >>> From: Andrey Shinkevich >>> >>> The bottom node is the intermediate block device that has the base as its >>> backing image. It is used instead

Re: [Qemu-block] [PATCH v0 1/3] qcow2: introduce compression type feature

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
28.05.2019 17:37, Denis Plotnikov wrote: > The patch adds some preparation parts for incompatible compression type > feature to QCOW2 header that indicates that *all* compressed clusters > must be (de)compressed using a certain compression type. > > It is implied that the compression type is set o

Re: [Qemu-block] [PATCH v6 3/3] block/stream: introduce a bottom node

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
29.05.2019 14:23, Max Reitz wrote: > On 29.05.19 09:34, Vladimir Sementsov-Ogievskiy wrote: >> 28.05.2019 20:33, Max Reitz wrote: >>> On 06.05.19 17:34, Vladimir Sementsov-Ogievskiy wrote: From: Andrey Shinkevich The bottom node is the intermediate block device that has the base as

[Qemu-block] [PATCH v2] qemu-io: add pattern file for write command

2019-05-29 Thread Denis Plotnikov
The patch allows to provide a pattern file for write command. There was no similar ability before. --- qemu-io-cmds.c | 58 ++ 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c index 09750a23ce..148f2ff92a

Re: [Qemu-block] [Qemu-devel] [PATCH v2] qemu-io: add pattern file for write command

2019-05-29 Thread no-reply
Patchew URL: https://patchew.org/QEMU/20190529114624.23107-1-dplotni...@virtuozzo.com/ Hi, This series seems to have some coding style problems. See output below for more information: Subject: [Qemu-devel] [PATCH v2] qemu-io: add pattern file for write command Type: series Message-id: 2019052

Re: [Qemu-block] [PATCH v6 3/3] block/stream: introduce a bottom node

2019-05-29 Thread Max Reitz
On 29.05.19 13:44, Vladimir Sementsov-Ogievskiy wrote: > 29.05.2019 14:23, Max Reitz wrote: >> On 29.05.19 09:34, Vladimir Sementsov-Ogievskiy wrote: >>> 28.05.2019 20:33, Max Reitz wrote: On 06.05.19 17:34, Vladimir Sementsov-Ogievskiy wrote: > From: Andrey Shinkevich > > The bot

[Qemu-block] [PATCH] hw/block/fdc: floppy command FIFO memory initialization

2019-05-29 Thread Andrey Shinkevich
The uninitialized memory allocated for the command FIFO of the floppy controller during the VM hardware initialization incurs many unwanted reports by Valgrind when VM state is being saved. That verbosity hardens a search for the real memory issues when the iotests run. Particularly, the patch elim

Re: [Qemu-block] [PATCH v2] qemu-io: add pattern file for write command

2019-05-29 Thread Max Reitz
On 29.05.19 13:46, Denis Plotnikov wrote: > The patch allows to provide a pattern file for write > command. There was no similar ability before. > --- > qemu-io-cmds.c | 58 ++ > 1 file changed, 54 insertions(+), 4 deletions(-) > > diff --git a/qemu

Re: [Qemu-block] [Qemu-devel] [PULL 0/3] Bitmaps patches

2019-05-29 Thread no-reply
Patchew URL: https://patchew.org/QEMU/20190528235842.29453-1-js...@redhat.com/ Hi, This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. === TEST SCRIPT BEGIN === #!/bin/bash time ma

Re: [Qemu-block] [Qemu-devel] [PATCH v2 0/3] Update documentation and help related to the preallocation parameter

2019-05-29 Thread Max Reitz
On 28.05.19 08:15, Markus Armbruster wrote: > Kevin, Max, this series looks ready to me. Feel free to ask me to take > it through my tree. *ask* :-) signature.asc Description: OpenPGP digital signature

Re: [Qemu-block] [PATCH v3] iotests: Filter 175's allocation information

2019-05-29 Thread Max Reitz
On 16.05.19 16:43, Max Reitz wrote: > It is possible for an empty file to take up blocks on a filesystem, for > example: > > $ qemu-img create -f raw test.img 1G > Formatting 'test.img', fmt=raw size=1073741824 > $ mkfs.ext4 -I 128 -q test.img > $ mkdir test-mount > $ sudo mount -o loop test.img t

Re: [Qemu-block] [Qemu-devel] [PATCH] hw/block/fdc: floppy command FIFO memory initialization

2019-05-29 Thread John Snow
On 5/29/19 8:22 AM, Andrey Shinkevich wrote: > The uninitialized memory allocated for the command FIFO of the > floppy controller during the VM hardware initialization incurs > many unwanted reports by Valgrind when VM state is being saved. > That verbosity hardens a search for the real memory i

Re: [Qemu-block] [Qemu-devel] [PULL 0/3] Bitmaps patches

2019-05-29 Thread John Snow
On 5/28/19 8:54 PM, no-re...@patchew.org wrote: > Patchew URL: https://patchew.org/QEMU/20190528235842.29453-1-js...@redhat.com/ > > > > Hi, > > This series failed the asan build test. Please find the testing commands and > their output below. If you have Docker installed, you can probably r

Re: [Qemu-block] [Qemu-devel] [PATCH] hw/block/fdc: floppy command FIFO memory initialization

2019-05-29 Thread Andrey Shinkevich
On 29/05/2019 16:40, John Snow wrote: > > > On 5/29/19 8:22 AM, Andrey Shinkevich wrote: >> The uninitialized memory allocated for the command FIFO of the >> floppy controller during the VM hardware initialization incurs >> many unwanted reports by Valgrind when VM state is being saved. >> That

Re: [Qemu-block] [Qemu-devel] [PATCH] hw/block/fdc: floppy command FIFO memory initialization

2019-05-29 Thread John Snow
On 5/29/19 9:56 AM, Andrey Shinkevich wrote: > > > On 29/05/2019 16:40, John Snow wrote: >> >> >> On 5/29/19 8:22 AM, Andrey Shinkevich wrote: >>> The uninitialized memory allocated for the command FIFO of the >>> floppy controller during the VM hardware initialization incurs >>> many unwanted

[Qemu-block] [PATCH v3] qemu-io: add pattern file for write command

2019-05-29 Thread Denis Plotnikov
The patch allows to provide a pattern file for write command. There was no similar ability before. --- qemu-io-cmds.c | 80 ++ 1 file changed, 74 insertions(+), 6 deletions(-) diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c index 09750a23ce..b75ffaa739

Re: [Qemu-block] [PATCH] iotests: Fix intermittent failure in 219

2019-05-29 Thread Max Reitz
On 16.05.19 18:11, Max Reitz wrote: > In 219, we wait for the job to make progress before we emit its status. > This makes the output reliable. > > Unfortunately, there is a bug: We do not wait for any more progress if > the job has already reached its total-progress. Right after the job has > be

Re: [Qemu-block] [Qemu-devel] [PATCH v2 0/3] Update documentation and help related to the preallocation parameter

2019-05-29 Thread Markus Armbruster
Stefano Garzarella writes: > Following Markus' advice, I updated the documentation of preallocation > parameter in qapi/block-core.json adding default and allowed values (patch 1). > I also updated the help related to BLOCK_OPT_PREALLOC in the QemuOptsList of > file-posix (patch 2) and gluster (p

Re: [Qemu-block] [PATCH 3/3] block/qcow2-bitmap: rewrite bitmap reopening logic

2019-05-29 Thread Max Reitz
On 23.05.19 17:47, Vladimir Sementsov-Ogievskiy wrote: > Current logic > = > > Reopen rw -> ro: > > Store bitmaps and release BdrvDirtyBitmap's. > > Reopen ro -> rw: > > Load bitmap list > Skip bitmaps which for which we don't have BdrvDirtyBitmap [this is >the worst thing] > A

[Qemu-block] [PATCH v8 5/7] block/io: refactor wait_serialising_requests

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
Split out do_wait_serialising_requests with additional possibility to not actually wait but just check, that there is something to wait for. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- block/io.c | 24 1 file changed, 16 insertions(+), 8 delet

[Qemu-block] [PATCH v8 2/7] block: swap operation order in bdrv_append

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
bs_top parents may conflict with bs_new backing child permissions, so let's do bdrv_replace_node first, it covers more possible cases. It is needed for further implementation of backup-top filter, which don't want to share write permission on its backing child. Side effect is that we may set back

[Qemu-block] [PATCH v8 1/7] block: teach bdrv_debug_breakpoint skip filters with backing

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
Teach bdrv_debug_breakpoint and bdrv_debug_remove_breakpoint skip filters with backing. This is needed to implement and use in backup job it's own backup_top filter driver (like mirror already has one), and without this improvement, breakpoint removal will fail at least in 55 iotest. Signed-off-by

[Qemu-block] [PATCH v8 4/7] block: introduce backup-top filter driver

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
Backup-top filter does copy-before-write operation. It should be inserted above active disk and has a target node for CBW, like the following: +---+ | Guest | +---+ |r,w v ++ target +---+ | backup_top |-->| target(qcow2

[Qemu-block] [PATCH v8 3/7] block: allow not one child for implicit node

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
Upcoming backup-top filter wants to operate like usual implicit filter node with fall-through to backing child. But also needs additional target child, let's support that. On the other hand, after backup completion (before job dismiss) filter is still attached to job blk, but don't have any childr

[Qemu-block] [PATCH v8 0/7] backup-top filter driver for backup

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
Hi all! These series introduce backup-top driver. It's a filter-node, which do copy-before-write operation. Mirror uses filter-node for handling guest writes, let's move to filter-node (from write-notifiers) for backup too v8: 01-03: new 05: add Max's r-b others changed, change description in eac

[Qemu-block] [PATCH v8 6/7] block: add lock/unlock range functions

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
From: Vladimir Sementsov-Ogievskiy Introduce lock/unlock range functionality, based on serialized requests. This is needed to refactor backup, dropping local tracked-request-like synchronization. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block_int.h | 4 block/io.c

[Qemu-block] [PATCH v8 7/7] block/backup: use backup-top instead of write notifiers

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
Drop write notifiers and use filter node instead. Changes: 1. copy-before-writes now handled by filter node, so, drop all is_write_notifier arguments. 2. we don't have intersecting requests, so their handling is dropped. Instead, synchronization works as follows: when backup or backup-top star

Re: [Qemu-block] [PATCH 3/3] block/qcow2-bitmap: rewrite bitmap reopening logic

2019-05-29 Thread Vladimir Sementsov-Ogievskiy
29.05.2019 18:33, Max Reitz wrote: > On 23.05.19 17:47, Vladimir Sementsov-Ogievskiy wrote: >> Current logic >> = >> >> Reopen rw -> ro: >> >> Store bitmaps and release BdrvDirtyBitmap's. >> >> Reopen ro -> rw: >> >> Load bitmap list >> Skip bitmaps which for which we don't have BdrvDir

[Qemu-block] [RFC PATCH] qemu-io-cmds: use clock_gettime for benchmarking

2019-05-29 Thread Alex Bennée
The previous use of gettimeofday() ran into undefined behaviour when we ended up doing a div 0 for a very short operation. This is because gettimeofday only works at the microsecond level as well as being prone to discontinuous jumps in system time. Using clock_gettime with CLOCK_MONOTONIC gives gr

[Qemu-block] [PATCH v7 2/3] block/stream: refactor stream_run: drop goto

2019-05-29 Thread Andrey Shinkevich
The goto is unnecessary in the stream_run() since the common exit code was removed in the commit eb23654dbe43b549ea2a9ebff9d8e: "jobs: utilize job_exit shim". Signed-off-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Alberto Garcia Reviewed-by: Max Reitz --- bl

[Qemu-block] [PATCH v7 3/3] block/stream: introduce a bottom node

2019-05-29 Thread Andrey Shinkevich
The bottom node is the intermediate block device that has the base as its backing image. It is used instead of the base node while a block stream job is running to avoid dependency on the base that may change due to the parallel jobs. The change may take place due to a filter node as well that is i

[Qemu-block] [PATCH v7 0/3] block/stream: get rid of the base

2019-05-29 Thread Andrey Shinkevich
This series introduces a bottom intermediate node that eliminates the dependency on the base that may change while stream job is running. It happens when stream/commit parallel jobs are running on the same backing chain. The base node of the stream job may be a top node of the parallel commit job a

[Qemu-block] [PATCH v7 1/3] block: include base when checking image chain for block allocation

2019-05-29 Thread Andrey Shinkevich
This patch is used in the 'block/stream: introduce a bottom node' that is following. Instead of the base node, the caller may pass the node that has the base as its backing image to the function bdrv_is_allocated_above() with a new parameter include_base = true and get rid of the dependency on the

Re: [Qemu-block] [Qemu-devel] [PATCH 3/3] block/qcow2-bitmap: rewrite bitmap reopening logic

2019-05-29 Thread John Snow
Max has picked this thread up for block discussion, so I'm going to stick to slightly more bitmap related discussion here; we'll resume block discussion in the other tail of this thread. On 5/29/19 5:10 AM, Vladimir Sementsov-Ogievskiy wrote: > 29.05.2019 2:24, John Snow wrote: >> >> >> On 5/23/19

Re: [Qemu-block] [PATCH 3/3] block/qcow2-bitmap: rewrite bitmap reopening logic

2019-05-29 Thread Max Reitz
On 29.05.19 17:58, Vladimir Sementsov-Ogievskiy wrote: > 29.05.2019 18:33, Max Reitz wrote: >> On 23.05.19 17:47, Vladimir Sementsov-Ogievskiy wrote: >>> Current logic >>> = >>> >>> Reopen rw -> ro: >>> >>> Store bitmaps and release BdrvDirtyBitmap's. >>> >>> Reopen ro -> rw: >>> >>> Lo

[Qemu-block] [PATCH v2] hw/block/fdc: floppy command FIFO memory initialization

2019-05-29 Thread Andrey Shinkevich
The uninitialized memory allocated for the command FIFO of the floppy controller during the VM hardware initialization incurs many unwanted reports by Valgrind when VM state is being saved. That verbosity hardens a search for the real memory issues when the iotests run. Particularly, the patch elim

Re: [Qemu-block] [PATCH v2] hw/block/fdc: floppy command FIFO memory initialization

2019-05-29 Thread John Snow
On 5/29/19 2:20 PM, Andrey Shinkevich wrote: > The uninitialized memory allocated for the command FIFO of the > floppy controller during the VM hardware initialization incurs > many unwanted reports by Valgrind when VM state is being saved. > That verbosity hardens a search for the real memory i

Re: [Qemu-block] [PATCH v2] hw/block/fdc: floppy command FIFO memory initialization

2019-05-29 Thread John Snow
On 5/29/19 2:20 PM, Andrey Shinkevich wrote: > The uninitialized memory allocated for the command FIFO of the > floppy controller during the VM hardware initialization incurs > many unwanted reports by Valgrind when VM state is being saved. > That verbosity hardens a search for the real memory i

Re: [Qemu-block] [PATCH v2] hw/block/fdc: floppy command FIFO memory initialization

2019-05-29 Thread Max Reitz
On 29.05.19 20:20, Andrey Shinkevich wrote: > The uninitialized memory allocated for the command FIFO of the > floppy controller during the VM hardware initialization incurs > many unwanted reports by Valgrind when VM state is being saved. > That verbosity hardens a search for the real memory issue

Re: [Qemu-block] [Qemu-devel] [PATCH v2 4/6] qapi: Disentangle QAPIDoc code

2019-05-29 Thread Kevin Wolf
Am 24.05.2019 um 18:11 hat Markus Armbruster geschrieben: > Kevin Wolf writes: > > > Documentation comment follow a certain structure: First, we have a text > > with a general description (called QAPIDoc.body). After this, > > descriptions of the arguments follow. Finally, we have part that > > c

Re: [Qemu-block] [RFC 1/3] block: Add ImageRotationalInfo

2019-05-29 Thread Kevin Wolf
Am 24.05.2019 um 19:28 hat Max Reitz geschrieben: > This enum indicates whether a file is stored on a rotating disk or a > solid-state drive. Drivers report it via the .bdrv_get_info() callback, > and if they do not, the global bdrv_get_info() implementation > automatically takes it from bs->file

Re: [Qemu-block] [Qemu-devel] [PATCH v2 2/6] tests/qapi-schema: Test for good feature lists in structs

2019-05-29 Thread Kevin Wolf
Am 24.05.2019 um 15:29 hat Markus Armbruster geschrieben: > Let's add > >{ 'command': 'test-features', > 'data': { 'fs0': 'FeatureStruct0', >'fs1': 'FeatureStruct1', >'fs2': 'FeatureStruct2', >'fs3': 'FeatureStruct3', >'cfs1'

Re: [Qemu-block] [PATCH v2] hw/block/fdc: floppy command FIFO memory initialization

2019-05-29 Thread Andrey Shinkevich
On 30/05/2019 00:38, Max Reitz wrote: > On 29.05.19 20:20, Andrey Shinkevich wrote: >> The uninitialized memory allocated for the command FIFO of the >> floppy controller during the VM hardware initialization incurs >> many unwanted reports by Valgrind when VM state is being saved. >> That verbosit