Re: [Qemu-block] [Qemu-devel] [PATCH v5 06/16] block/mirror: conservative mirror_exit refactor

2018-09-06 Thread John Snow
On 09/06/2018 12:57 PM, Jeff Cody wrote: > On Thu, Sep 06, 2018 at 09:02:15AM -0400, John Snow wrote: >> For purposes of minimum code movement, refactor the mirror_exit >> callback to use the post-finalization callbacks in a trivial way. >> >> Signed-off-by: John Snow >> --- >> block/mirror.c

[Qemu-block] forward of qemu bug regarding random qemu hangs at shutdown

2018-09-06 Thread Christian Borntraeger
Kevin, for reference, it seems that his bug report somehow got lost. https://bugs.launchpad.net/qemu/+bug/1788582

Re: [Qemu-block] [PATCH v5 14/16] qapi/block-stream: expose new job properties

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:23AM -0400, John Snow wrote: > Signed-off-by: John Snow > Reviewed-by: Max Reitz Reviewed-by: Jeff Cody > --- > blockdev.c | 9 + > hmp.c| 5 +++-- > qapi/block-core.json | 16 +++- > 3 files changed, 27

Re: [Qemu-block] [PATCH v5 13/16] qapi/block-mirror: expose new job properties

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:22AM -0400, John Snow wrote: > Signed-off-by: John Snow > Reviewed-by: Max Reitz Reviewed-by: Jeff Cody > --- > blockdev.c | 14 ++ > qapi/block-core.json | 30 -- > 2 files changed, 42 insertions(+), 2

Re: [Qemu-block] [PATCH v5 16/16] blockdev: document transactional shortcomings

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:25AM -0400, John Snow wrote: > Presently only the backup job really guarantees what one would consider > transactional semantics. To guard against someone helpfully adding them > in the future, document that there are shortcomings in the model that > would need to be

Re: [Qemu-block] [PATCH v5 15/16] block/backup: qapi documentation fixup

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:24AM -0400, John Snow wrote: > Fix documentation to match the other jobs amended for 3.1. > > Signed-off-by: John Snow > Reviewed-by: Max Reitz Reviewed-by: Jeff Cody > --- > qapi/block-core.json | 18 ++ > 1 file changed, 10 insertions(+), 8

Re: [Qemu-block] [PATCH v5 12/16] qapi/block-commit: expose new job properties

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:21AM -0400, John Snow wrote: > Signed-off-by: John Snow > Reviewed-by: Max Reitz Reviewed-by: Jeff Cody > --- > blockdev.c | 8 > qapi/block-core.json | 16 +++- > 2 files changed, 23 insertions(+), 1 deletion(-) > > diff --git

Re: [Qemu-block] [PATCH v5 11/16] jobs: remove .exit callback

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:20AM -0400, John Snow wrote: > Now that all of the jobs use the component finalization callbacks, > there's no use for the heavy-hammer .exit callback anymore. > > job_exit becomes a glorified type shim so that we can call > job_completed from

Re: [Qemu-block] [PATCH v5 10/16] tests/test-blockjob-txn: move .exit to .clean

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:19AM -0400, John Snow wrote: > The exit callback in this test actually only performs cleanup. > > Signed-off-by: John Snow > Reviewed-by: Max Reitz Reviewed-by: Jeff Cody > --- > tests/test-blockjob-txn.c | 4 ++-- > 1 file changed, 2 insertions(+), 2

Re: [Qemu-block] [PATCH v5 09/16] tests/test-blockjob: remove exit callback

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:18AM -0400, John Snow wrote: > We remove the exit callback and the completed boolean along with it. > We can simulate it just fine by waiting for the job to defer to the > main loop, and then giving it one final kick to get the main loop > portion to run. > >

Re: [Qemu-block] [PATCH v5 08/16] tests/blockjob: replace Blockjob with Job

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:17AM -0400, John Snow wrote: > These tests don't actually test blockjobs anymore, they test > generic Job lifetimes. Change the types accordingly. > > Signed-off-by: John Snow > Reviewed-by: Max Reitz Reviewed-by: Jeff Cody > --- > tests/test-blockjob.c | 98 >

Re: [Qemu-block] [PATCH v5 07/16] block/stream: refactor stream to use job callbacks

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:16AM -0400, John Snow wrote: > Signed-off-by: John Snow > Reviewed-by: Max Reitz Reviewed-by: Jeff Cody > --- > block/stream.c | 23 +++ > 1 file changed, 15 insertions(+), 8 deletions(-) > > diff --git a/block/stream.c b/block/stream.c >

Re: [Qemu-block] [PATCH v5 06/16] block/mirror: conservative mirror_exit refactor

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:15AM -0400, John Snow wrote: > For purposes of minimum code movement, refactor the mirror_exit > callback to use the post-finalization callbacks in a trivial way. > > Signed-off-by: John Snow > --- > block/mirror.c | 39 --- > 1

Re: [Qemu-block] [PATCH v5 05/16] block/mirror: don't install backing chain on abort

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:14AM -0400, John Snow wrote: > In cases where we abort the block/mirror job, there's no point in > installing the new backing chain before we finish aborting. > > Signed-off-by: John Snow Reviewed-by: Jeff Cody > --- > block/mirror.c | 2 +- > 1 file changed, 1

Re: [Qemu-block] [PATCH v5 04/16] block/commit: refactor commit to use job callbacks

2018-09-06 Thread Jeff Cody
On Thu, Sep 06, 2018 at 09:02:13AM -0400, John Snow wrote: > Use the component callbacks; prepare, abort, and clean. > > NB: prepare is only called when the job has not yet failed; > and abort can be called after prepare. > > complete -> prepare -> abort -> clean > complete -> abort -> clean >

Re: [Qemu-block] backend for blk or fs with guaranteed blocking/synchronous I/O

2018-09-06 Thread Michael S. Tsirkin
On Thu, Sep 06, 2018 at 04:24:12PM +0600, Artem Pisarenko wrote: > Hi all, > > I'm developing paravirtualized target linux system which runs multiple linux > containers (LXC) inside itself. (For those, who unfamiliar with LXC, simply > put, it's an isolated group of userspace processes with their

[Qemu-block] [PATCH] block: Fix use after free error in bdrv_open_inherit()

2018-09-06 Thread Alberto Garcia
When a block device is opened with BDRV_O_SNAPSHOT and the bdrv_append_temp_snapshot() call fails then the error code path tries to unref the already destroyed 'options' QDict. This can be reproduced easily by setting TMPDIR to a location where the QEMU process can't write: $

[Qemu-block] [PATCH v5 12/16] qapi/block-commit: expose new job properties

2018-09-06 Thread John Snow
Signed-off-by: John Snow Reviewed-by: Max Reitz --- blockdev.c | 8 qapi/block-core.json | 16 +++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/blockdev.c b/blockdev.c index ec90eb1cf9..98b91e75a7 100644 --- a/blockdev.c +++ b/blockdev.c @@

[Qemu-block] [PATCH v5 15/16] block/backup: qapi documentation fixup

2018-09-06 Thread John Snow
Fix documentation to match the other jobs amended for 3.1. Signed-off-by: John Snow Reviewed-by: Max Reitz --- qapi/block-core.json | 18 ++ 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index f877e9e414..c0b3d33dbb

[Qemu-block] [PATCH v5 09/16] tests/test-blockjob: remove exit callback

2018-09-06 Thread John Snow
We remove the exit callback and the completed boolean along with it. We can simulate it just fine by waiting for the job to defer to the main loop, and then giving it one final kick to get the main loop portion to run. Signed-off-by: John Snow Reviewed-by: Max Reitz --- tests/test-blockjob.c |

[Qemu-block] [PATCH v5 08/16] tests/blockjob: replace Blockjob with Job

2018-09-06 Thread John Snow
These tests don't actually test blockjobs anymore, they test generic Job lifetimes. Change the types accordingly. Signed-off-by: John Snow Reviewed-by: Max Reitz --- tests/test-blockjob.c | 98 ++- 1 file changed, 50 insertions(+), 48

[Qemu-block] [PATCH v5 16/16] blockdev: document transactional shortcomings

2018-09-06 Thread John Snow
Presently only the backup job really guarantees what one would consider transactional semantics. To guard against someone helpfully adding them in the future, document that there are shortcomings in the model that would need to be audited at that time. Signed-off-by: John Snow --- blockdev.c |

[Qemu-block] [PATCH v5 11/16] jobs: remove .exit callback

2018-09-06 Thread John Snow
Now that all of the jobs use the component finalization callbacks, there's no use for the heavy-hammer .exit callback anymore. job_exit becomes a glorified type shim so that we can call job_completed from aio_bh_schedule_oneshot. Move these three functions down into job.c to eliminate a forward

[Qemu-block] [PATCH v5 13/16] qapi/block-mirror: expose new job properties

2018-09-06 Thread John Snow
Signed-off-by: John Snow Reviewed-by: Max Reitz --- blockdev.c | 14 ++ qapi/block-core.json | 30 -- 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/blockdev.c b/blockdev.c index 98b91e75a7..429cdf9901 100644 --- a/blockdev.c

[Qemu-block] [PATCH v5 10/16] tests/test-blockjob-txn: move .exit to .clean

2018-09-06 Thread John Snow
The exit callback in this test actually only performs cleanup. Signed-off-by: John Snow Reviewed-by: Max Reitz --- tests/test-blockjob-txn.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test-blockjob-txn.c b/tests/test-blockjob-txn.c index

[Qemu-block] [PATCH v5 04/16] block/commit: refactor commit to use job callbacks

2018-09-06 Thread John Snow
Use the component callbacks; prepare, abort, and clean. NB: prepare is only called when the job has not yet failed; and abort can be called after prepare. complete -> prepare -> abort -> clean complete -> abort -> clean During refactor, a potential problem with bdrv_drop_intermediate was

[Qemu-block] [PATCH v5 02/16] block/mirror: add block job creation flags

2018-09-06 Thread John Snow
Add support for taking and passing forward job creation flags. Signed-off-by: John Snow Reviewed-by: Max Reitz Reviewed-by: Jeff Cody --- block/mirror.c| 5 +++-- blockdev.c| 3 ++- include/block/block_int.h | 5 - 3 files changed, 9 insertions(+), 4

[Qemu-block] [PATCH v5 07/16] block/stream: refactor stream to use job callbacks

2018-09-06 Thread John Snow
Signed-off-by: John Snow Reviewed-by: Max Reitz --- block/stream.c | 23 +++ 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/block/stream.c b/block/stream.c index 700eb239e4..81a7ec8ece 100644 --- a/block/stream.c +++ b/block/stream.c @@ -54,16 +54,16 @@

[Qemu-block] [PATCH v5 03/16] block/stream: add block job creation flags

2018-09-06 Thread John Snow
Add support for taking and passing forward job creation flags. Signed-off-by: John Snow Reviewed-by: Max Reitz Reviewed-by: Jeff Cody --- block/stream.c| 5 +++-- blockdev.c| 3 ++- include/block/block_int.h | 5 - 3 files changed, 9 insertions(+), 4

[Qemu-block] [PATCH v5 14/16] qapi/block-stream: expose new job properties

2018-09-06 Thread John Snow
Signed-off-by: John Snow Reviewed-by: Max Reitz --- blockdev.c | 9 + hmp.c| 5 +++-- qapi/block-core.json | 16 +++- 3 files changed, 27 insertions(+), 3 deletions(-) diff --git a/blockdev.c b/blockdev.c index 429cdf9901..0cf8febe6c 100644 ---

[Qemu-block] [PATCH v5 05/16] block/mirror: don't install backing chain on abort

2018-09-06 Thread John Snow
In cases where we abort the block/mirror job, there's no point in installing the new backing chain before we finish aborting. Signed-off-by: John Snow --- block/mirror.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/mirror.c b/block/mirror.c index

[Qemu-block] [PATCH v5 00/16] jobs: Job Exit Refactoring Pt 2

2018-09-06 Thread John Snow
This is part two of a two part series that refactors the exit logic of jobs. This series forces all jobs to use the "finalize" semantics that were introduced previously, but only exposed via the backup jobs. Patches 1-3 add plumbing for the auto-dismiss and auto-finalize flags but do not expose

[Qemu-block] [PATCH v5 06/16] block/mirror: conservative mirror_exit refactor

2018-09-06 Thread John Snow
For purposes of minimum code movement, refactor the mirror_exit callback to use the post-finalization callbacks in a trivial way. Signed-off-by: John Snow --- block/mirror.c | 39 --- 1 file changed, 28 insertions(+), 11 deletions(-) diff --git

[Qemu-block] backend for blk or fs with guaranteed blocking/synchronous I/O

2018-09-06 Thread Artem Pisarenko
Hi all, I'm developing paravirtualized target linux system which runs multiple linux containers (LXC) inside itself. (For those, who unfamiliar with LXC, simply put, it's an isolated group of userspace processes with their own rootfs.) Each container should be provided access to its rootfs

Re: [Qemu-block] [PATCH] block/linux-aio: acquire AioContext before qemu_laio_process_completions

2018-09-06 Thread Kevin Wolf
Am 05.09.2018 um 13:23 hat Sergio Lopez geschrieben: > In qemu_laio_process_completions_and_submit, the AioContext is acquired > before the ioq_submit iteration and after qemu_laio_process_completions, > but the latter is not thread safe either. > > This change avoids a number of random crashes

[Qemu-block] [PATCH v3 06/10] block: Forbid trying to change unsupported options during reopen

2018-09-06 Thread Alberto Garcia
The bdrv_reopen_prepare() function checks all options passed to each BlockDriverState (in the reopen_state->options QDict) and makes all necessary preparations to apply the option changes requested by the user. Options are removed from the QDict as they are processed, so at the end of

[Qemu-block] [PATCH v3 03/10] block: Remove child references from bs->{options, explicit_options}

2018-09-06 Thread Alberto Garcia
Block drivers allow opening their children using a reference to an existing BlockDriverState. These references remain stored in the 'options' and 'explicit_options' QDicts, but we don't need to keep them once everything is open. What is more important, these values can become wrong if the

[Qemu-block] [PATCH v3 01/10] qemu-io: Fix writethrough check in reopen

2018-09-06 Thread Alberto Garcia
"qemu-io reopen" doesn't allow changing the writethrough setting of the cache, but the check is wrong, causing an error even on a simple reopen with the default parameters: $ qemu-img create -f qcow2 hd.qcow2 1M $ qemu-system-x86_64 -monitor stdio -drive if=virtio,file=hd.qcow2 (qemu)

[Qemu-block] [PATCH v3 10/10] block: Allow changing 'force-share' on reopen

2018-09-06 Thread Alberto Garcia
'force-share' is one of the basic BlockdevOptions available for all drivers, but it's not handled by bdrv_reopen_prepare() so any attempt to change it results in a "Cannot change the option" error: (qemu) qemu-io virtio0 "reopen -o force-share=on" Cannot change the option 'force-share'

[Qemu-block] [PATCH v3 07/10] file-posix: Forbid trying to change unsupported options during reopen

2018-09-06 Thread Alberto Garcia
The file-posix code is used for the "file", "host_device" and "host_cdrom" drivers, and it allows reopening images. However the only option that is actually processed is "x-check-cache-dropped", and changes in all other options (e.g. "filename") are silently ignored: (qemu) qemu-io virtio0

[Qemu-block] [PATCH v3 09/10] block: Allow changing 'detect-zeroes' on reopen

2018-09-06 Thread Alberto Garcia
'detect-zeroes' is one of the basic BlockdevOptions available for all drivers, but it's not handled by bdrv_reopen_prepare(), so any attempt to change it results in an error: (qemu) qemu-io virtio0 "reopen -o detect-zeroes=on" Cannot change the option 'detect-zeroes' Since there's no

[Qemu-block] [PATCH v3 05/10] block: Allow child references on reopen

2018-09-06 Thread Alberto Garcia
In the previous patches we removed all child references from bs->{options,explicit_options} because keeping them is useless and wrong. Because of this, any attempt to reopen a BlockDriverState using a child reference as one of its options would result in a failure, because bdrv_reopen_prepare()

[Qemu-block] [PATCH v3 08/10] block: Allow changing 'discard' on reopen

2018-09-06 Thread Alberto Garcia
'discard' is one of the basic BlockdevOptions available for all drivers, but it's not handled by bdrv_reopen_prepare() so any attempt to change it results in an error: (qemu) qemu-io virtio0 "reopen -o discard=on" Cannot change the option 'discard' Since there's no reason why we shouldn't

[Qemu-block] [PATCH v3 04/10] block: Don't look for child references in append_open_options()

2018-09-06 Thread Alberto Garcia
In the previous patch we removed child references from bs->options, so there's no need to look for them here anymore. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block.c | 13 + 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/block.c b/block.c index

[Qemu-block] [PATCH v3 02/10] file-posix: x-check-cache-dropped should default to false on reopen

2018-09-06 Thread Alberto Garcia
The default value of x-check-cache-dropped is false. There's no reason to use the previous value as a default in raw_reopen_prepare() because bdrv_reopen_queue_child() already takes care of putting the old options in the BDRVReopenState.options QDict. If x-check-cache-dropped was previously set

[Qemu-block] [PATCH v3 00/10] Misc reopen-related patches

2018-09-06 Thread Alberto Garcia
Hi, as part of my blockdev-reopen work here's a new set of patches. This doesn't implement yet the core functionality of the new reopen command, but it does fix a few things that help us pave the way. I believe that the next series after this one will be the last. The main change is the removal