Re: [Qemu-block] [PATCH] vl: introduce vm_shutdown()

2018-02-28 Thread Stefan Hajnoczi
On Wed, Feb 28, 2018 at 09:58:13AM +0800, Fam Zheng wrote: > On Tue, 02/27 15:30, Stefan Hajnoczi wrote: > > On Fri, Feb 23, 2018 at 04:20:44PM +0800, Fam Zheng wrote: > > > On Tue, 02/20 13:10, Stefan Hajnoczi wrote: > > > > 1. virtio_scsi_handle_cmd_vq() racing with iothread_stop_all() hits the >

Re: [Qemu-block] [PATCH v4 2/5] qcow2: Document some maximum size constraints

2018-02-28 Thread Alberto Garcia
On Tue 27 Feb 2018 05:29:41 PM CET, Eric Blake wrote: > +The refcount table has implications on the maximum host file size; a > +larger cluster size is required for the refcount table to cover > larger +offsets. Why is this? Because of the refcount_table_clusters field ? I think the maximum offse

Re: [Qemu-block] [PATCH v4 3/5] qcow2: Reduce REFT_OFFSET_MASK

2018-02-28 Thread Alberto Garcia
On Tue 27 Feb 2018 05:29:42 PM CET, Eric Blake wrote: > Match our code to the spec change in the previous patch - there's > no reason for the refcount table to allow larger offsets than the > L1/L2 tables. In practice, no image has more than 64PB of > allocated clusters anyways, as anything beyond

[Qemu-block] [PATCH 2/2] iotests: Test preallocated truncate of 2G image

2018-02-28 Thread Max Reitz
Signed-off-by: Max Reitz --- tests/qemu-iotests/106 | 24 tests/qemu-iotests/106.out | 10 ++ 2 files changed, 34 insertions(+) diff --git a/tests/qemu-iotests/106 b/tests/qemu-iotests/106 index bfe71f4e60..5e51f88a78 100755 --- a/tests/qemu-iotests/106 +++ b

[Qemu-block] [PATCH 0/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Max Reitz
Fully preallocated truncation has a 50 % chance of working on images files over file-posix. It works if $SIZE % 4G < 2G, and it fails otherwise. To make things even more interesting, often you would not even notice because qemu reported success even though it did nothing (because after the succes

[Qemu-block] [PATCH 1/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Max Reitz
Storing the lseek() result in an int results in it overflowing when the file is at least 2 GB big. Then, we have a 50 % chance of the result being "negative" and thus thinking an error occurred when actually everything went just fine. So we should use the correct type for storing the result: off_

Re: [Qemu-block] [PATCH 1/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Daniel P . Berrangé
On Wed, Feb 28, 2018 at 02:13:14PM +0100, Max Reitz wrote: > Storing the lseek() result in an int results in it overflowing when the > file is at least 2 GB big. Then, we have a 50 % chance of the result > being "negative" and thus thinking an error occurred when actually > everything went just fi

Re: [Qemu-block] [PATCH 1/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Max Reitz
On 2018-02-28 14:34, Daniel P. Berrangé wrote: > On Wed, Feb 28, 2018 at 02:13:14PM +0100, Max Reitz wrote: >> Storing the lseek() result in an int results in it overflowing when the >> file is at least 2 GB big. Then, we have a 50 % chance of the result >> being "negative" and thus thinking an er

Re: [Qemu-block] [PATCH 1/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Daniel P . Berrangé
On Wed, Feb 28, 2018 at 02:45:49PM +0100, Max Reitz wrote: > On 2018-02-28 14:34, Daniel P. Berrangé wrote: > > On Wed, Feb 28, 2018 at 02:13:14PM +0100, Max Reitz wrote: > >> Storing the lseek() result in an int results in it overflowing when the > >> file is at least 2 GB big. Then, we have a 50

Re: [Qemu-block] [PATCH 1/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Max Reitz
On 2018-02-28 14:53, Daniel P. Berrangé wrote: > On Wed, Feb 28, 2018 at 02:45:49PM +0100, Max Reitz wrote: >> On 2018-02-28 14:34, Daniel P. Berrangé wrote: >>> On Wed, Feb 28, 2018 at 02:13:14PM +0100, Max Reitz wrote: Storing the lseek() result in an int results in it overflowing when the >

Re: [Qemu-block] [PATCH 1/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Daniel P . Berrangé
On Wed, Feb 28, 2018 at 02:13:14PM +0100, Max Reitz wrote: > Storing the lseek() result in an int results in it overflowing when the > file is at least 2 GB big. Then, we have a 50 % chance of the result > being "negative" and thus thinking an error occurred when actually > everything went just fi

Re: [Qemu-block] [PATCH 1/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Daniel P . Berrangé
On Wed, Feb 28, 2018 at 02:55:22PM +0100, Max Reitz wrote: > On 2018-02-28 14:53, Daniel P. Berrangé wrote: > > On Wed, Feb 28, 2018 at 02:45:49PM +0100, Max Reitz wrote: > >> On 2018-02-28 14:34, Daniel P. Berrangé wrote: > >>> On Wed, Feb 28, 2018 at 02:13:14PM +0100, Max Reitz wrote: > Stor

Re: [Qemu-block] [PATCH v4 2/5] qcow2: Document some maximum size constraints

2018-02-28 Thread Eric Blake
On 02/28/2018 04:26 AM, Alberto Garcia wrote: On Tue 27 Feb 2018 05:29:41 PM CET, Eric Blake wrote: +The refcount table has implications on the maximum host file size; a +larger cluster size is required for the refcount table to cover larger +offsets. Why is this? Because of the refcount_table

Re: [Qemu-block] [PATCH 2/2] iotests: Test preallocated truncate of 2G image

2018-02-28 Thread Daniel P . Berrangé
On Wed, Feb 28, 2018 at 02:13:15PM +0100, Max Reitz wrote: > Signed-off-by: Max Reitz > --- > tests/qemu-iotests/106 | 24 > tests/qemu-iotests/106.out | 10 ++ > 2 files changed, 34 insertions(+) > > diff --git a/tests/qemu-iotests/106 b/tests/qemu-iotests/1

Re: [Qemu-block] [PATCH v2 05/16] block/mirror: Convert to coroutines

2018-02-28 Thread Max Reitz
On 2018-02-27 08:44, Fam Zheng wrote: > On Mon, 01/22 23:07, Max Reitz wrote: >> @@ -101,7 +105,7 @@ static BlockErrorAction >> mirror_error_action(MirrorBlockJob *s, bool read, >> } >> } >> >> -static void mirror_iteration_done(MirrorOp *op, int ret) >> +static void coroutine_fn mirror_it

Re: [Qemu-block] [Qemu-devel] [PATCH 1/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Eric Blake
On 02/28/2018 07:53 AM, Daniel P. Berrangé wrote: @@ -1711,8 +1712,8 @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, buf = g_malloc0(65536); -result = lseek(fd, current_length, SEEK_SET); -if (result < 0) { +seek_result

Re: [Qemu-block] [PATCH v4 2/5] qcow2: Document some maximum size constraints

2018-02-28 Thread Alberto Garcia
On Wed 28 Feb 2018 03:01:33 PM CET, Eric Blake wrote: >>> The refcount table has implications on the maximum host file size; a >>> larger cluster size is required for the refcount table to cover >>> larger offsets. >> >> Why is this? Because of the refcount_table_clusters field ? >> >> I think t

Re: [Qemu-block] [Qemu-devel] [PATCH 0/2] block/file-posix: Fix fully preallocated truncate

2018-02-28 Thread Eric Blake
On 02/28/2018 07:13 AM, Max Reitz wrote: Fully preallocated truncation has a 50 % chance of working on images files over file-posix. It works if $SIZE % 4G < 2G, and it fails otherwise. To make things even more interesting, often you would not even notice because qemu reported success even thou

Re: [Qemu-block] [PATCH v4 5/5] qcow2: Avoid memory over-allocation on compressed images

2018-02-28 Thread Alberto Garcia
On Tue 27 Feb 2018 05:29:44 PM CET, Eric Blake wrote: > So, it's time to cut back on the waste. A compressed cluster > written by qemu will NEVER occupy more than an uncompressed > cluster, but based on mid-sector alignment, we may still need > to read 1 cluster + 1 sector in order to recover enou

Re: [Qemu-block] [PATCH v2 10/16] hbitmap: Add @advance param to hbitmap_iter_next()

2018-02-28 Thread Max Reitz
On 2018-02-27 09:59, Fam Zheng wrote: > On Mon, 01/22 23:08, Max Reitz wrote: >> diff --git a/tests/test-hbitmap.c b/tests/test-hbitmap.c >> index 9091c639b3..2a2aa5bd43 100644 >> --- a/tests/test-hbitmap.c >> +++ b/tests/test-hbitmap.c >> @@ -46,7 +46,7 @@ static void hbitmap_test_check(TestHBitma

Re: [Qemu-block] [RFC v4 08/21] blockjobs: add ABORTING state

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > Add a new state ABORTING. > > This makes transitions from normative states to error states explicit > in the STM, and serves as a disambiguation for which states may complete > normally when normal end-states (CONCLUDED) are added in future commi

Re: [Qemu-block] [PATCH v2 11/16] block/dirty-bitmap: Add bdrv_dirty_iter_next_area

2018-02-28 Thread Max Reitz
On 2018-02-27 10:06, Fam Zheng wrote: > On Mon, 01/22 23:08, Max Reitz wrote: >> This new function allows to look for a consecutively dirty area in a >> dirty bitmap. >> >> Signed-off-by: Max Reitz >> --- >> include/block/dirty-bitmap.h | 2 ++ >> block/dirty-bitmap.c | 51 >> ++

Re: [Qemu-block] [PATCH v2 12/16] block/mirror: Distinguish active from passive ops

2018-02-28 Thread Max Reitz
On 2018-02-27 10:12, Fam Zheng wrote: > On Mon, 01/22 23:08, Max Reitz wrote: >> Currently, the mirror block job only knows passive operations. But once >> we introduce active writes, we need to distinguish between the two; for >> example, mirror_wait_for_free_in_flight_slot() should wait for a pa

Re: [Qemu-block] [PATCH v2 14/16] block/mirror: Add active mirroring

2018-02-28 Thread Max Reitz
On 2018-02-27 10:34, Fam Zheng wrote: > On Mon, 01/22 23:08, Max Reitz wrote: >> @@ -1151,7 +1285,48 @@ static int coroutine_fn >> bdrv_mirror_top_preadv(BlockDriverState *bs, >> static int coroutine_fn bdrv_mirror_top_pwritev(BlockDriverState *bs, >> uint64_t offset, uint64_t bytes, QEMUIOV

Re: [Qemu-block] [RFC v4 09/21] blockjobs: add CONCLUDED state

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > add a new state "CONCLUDED" that identifies a job that has ceased all > operations. The wording was chosen to avoid any phrasing that might > imply success, error, or cancellation. The task has simply ceased all > operation and can never again per

Re: [Qemu-block] [RFC v4 10/21] blockjobs: add NULL state

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > Add a new state that specifically demarcates when we begin to permanently > demolish a job after it has performed all work. This makes the transition > explicit in the STM table and highlights conditions under which a job may > be demolished. > >

Re: [Qemu-block] [RFC v4 11/21] blockjobs: add block_job_dismiss

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > For jobs that have reached their CONCLUDED state, prior to having their > last reference put down (meaning jobs that have completed successfully, > unsuccessfully, or have been canceled), allow the user to dismiss the > job's lingering status repo

Re: [Qemu-block] [PATCH v2 15/16] block/mirror: Add copy mode QAPI interface

2018-02-28 Thread Max Reitz
On 2018-02-27 10:38, Fam Zheng wrote: > On Mon, 01/22 23:08, Max Reitz wrote: >> This patch allows the user to specify whether to use active or only >> passive mode for mirror block jobs. Currently, this setting will remain > > I think you want s/passive/background/ in the whole patch. Errr, yes

Re: [Qemu-block] [RFC v4 12/21] blockjobs: ensure abort is called for cancelled jobs

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > Presently, even if a job is canceled post-completion as a result of > a failing peer in a transaction, it will still call .commit because > nothing has updated or changed its return code. > > The reason why this does not cause problems currently

Re: [Qemu-block] [RFC v4 13/21] blockjobs: add commit, abort, clean helpers

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > The completed_single function is getting a little mucked up with > checking to see which callbacks exist, so let's factor them out. > > Signed-off-by: John Snow Reviewed-by: Kevin Wolf

Re: [Qemu-block] [RFC v4 14/21] blockjobs: add block_job_txn_apply function

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > Simply apply a function transaction-wide. > A few more uses of this in forthcoming patches. > > Signed-off-by: John Snow > --- > blockjob.c | 24 +++- > 1 file changed, 15 insertions(+), 9 deletions(-) > > diff --git a/bloc

Re: [Qemu-block] [RFC v4 15/21] blockjobs: add prepare callback

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > Some jobs upon finalization may need to perform some work that can > still fail. If these jobs are part of a transaction, it's important > that these callbacks fail the entire transaction. > > We allow for a new callback in addition to commit/abo

Re: [Qemu-block] [PATCH v2 05/16] block/mirror: Convert to coroutines

2018-02-28 Thread Max Reitz
On 2018-02-28 15:13, Max Reitz wrote: > On 2018-02-27 08:44, Fam Zheng wrote: >> On Mon, 01/22 23:07, Max Reitz wrote: >>> @@ -101,7 +105,7 @@ static BlockErrorAction >>> mirror_error_action(MirrorBlockJob *s, bool read, >>> } >>> } >>> >>> -static void mirror_iteration_done(MirrorOp *op,

Re: [Qemu-block] [Qemu-devel] [RFC v4 16/21] blockjobs: add waiting status

2018-02-28 Thread Kevin Wolf
Am 27.02.2018 um 21:50 hat John Snow geschrieben: > > > On 02/27/2018 03:00 PM, Eric Blake wrote: > > On 02/23/2018 05:51 PM, John Snow wrote: > >> For jobs that are stuck waiting on others in a transaction, it would > >> be nice to know that they are no longer "running" in that sense, but > >> i

Re: [Qemu-block] [PATCH v3] iotests: Fix CID for VMDK afl image

2018-02-28 Thread Max Reitz
On 2018-02-02 06:23, Fam Zheng wrote: > This reverts commit 76bf133c4 which updated the reference output, and > fixed the reference image, because the code path we want to exercise is > actually the invalid image size. > > The descriptor block in the image, which includes the CID to verify, has be

Re: [Qemu-block] [RFC v4 17/21] blockjobs: add PENDING status and event

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > For jobs utilizing the new manual workflow, we intend to prohibit > them from modifying the block graph until the management layer provides > an explicit ACK via block-job-finalize to move the process forward. > > To distinguish this runstate fro

[Qemu-block] [PATCH v3 01/16] block: BDS deletion during bdrv_drain_recurse

2018-02-28 Thread Max Reitz
Draining a BDS child may lead to other children of the same parent being detached and/or deleted. We should prepare for the former case (by copying the children list before iterating through it) and prevent the latter (by bdrv_ref()'ing all nodes if we are in the main loop). Signed-off-by: Max Re

[Qemu-block] [PATCH v3 05/16] block/mirror: Convert to coroutines

2018-02-28 Thread Max Reitz
In order to talk to the source BDS (and maybe in the future to the target BDS as well) directly, we need to convert our existing AIO requests into coroutine I/O requests. Signed-off-by: Max Reitz --- block/mirror.c | 152 ++--- 1 file changed,

[Qemu-block] [PATCH v3 00/16] block/mirror: Add active-sync mirroring

2018-02-28 Thread Max Reitz
This series implements an active and synchronous mirroring mode. Currently, the mirror block job is passive an asynchronous: Depending on your start conditions, some part of the source disk starts as "dirty". Then, the block job will (as a background operation) continuously copy dirty parts to the

[Qemu-block] [PATCH v3 02/16] block: BDS deletion in bdrv_do_drained_begin()

2018-02-28 Thread Max Reitz
Draining a BDS (in the main loop) may cause it to go be deleted. That is rather suboptimal if we still plan to access it afterwards, so let us enclose the main body of the function with a bdrv_ref()/bdrv_unref() pair. Signed-off-by: Max Reitz --- block/io.c | 19 +++ 1 file chan

[Qemu-block] [PATCH v3 16/16] iotests: Add test for active mirroring

2018-02-28 Thread Max Reitz
Signed-off-by: Max Reitz --- tests/qemu-iotests/151 | 120 + tests/qemu-iotests/151.out | 5 ++ tests/qemu-iotests/group | 1 + 3 files changed, 126 insertions(+) create mode 100755 tests/qemu-iotests/151 create mode 100644 tests/qemu-iotest

[Qemu-block] [PATCH v3 03/16] tests: Add bdrv-drain test for node deletion

2018-02-28 Thread Max Reitz
This patch adds two bdrv-drain tests for what happens if some BDS goes away during the drainage. The basic idea is that you have a parent BDS with some child nodes. Then, you drain one of the children. Because of that, the party who actually owns the parent decides to (A) delete it, or (B) detach

[Qemu-block] [PATCH v3 06/16] block/mirror: Use CoQueue to wait on in-flight ops

2018-02-28 Thread Max Reitz
Attach a CoQueue to each in-flight operation so if we need to wait for any we can use it to wait instead of just blindly yielding and hoping for some operation to wake us. A later patch will use this infrastructure to allow requests accessing the same area of the virtual disk to specifically wait

[Qemu-block] [PATCH v3 09/16] block: Generalize should_update_child() rule

2018-02-28 Thread Max Reitz
Currently, bdrv_replace_node() refuses to create loops from one BDS to itself if the BDS to be replaced is the backing node of the BDS to replace it: Say there is a node A and a node B. Replacing B by A means making all references to B point to A. If B is a child of A (i.e. A has a reference to B

[Qemu-block] [PATCH v3 07/16] block/mirror: Wait for in-flight op conflicts

2018-02-28 Thread Max Reitz
This patch makes the mirror code differentiate between simply waiting for any operation to complete (mirror_wait_for_free_in_flight_slot()) and specifically waiting for all operations touching a certain range of the virtual disk to complete (mirror_wait_on_conflicts()). Signed-off-by: Max Reitz -

[Qemu-block] [PATCH v3 04/16] block/mirror: Pull out mirror_perform()

2018-02-28 Thread Max Reitz
When converting mirror's I/O to coroutines, we are going to need a point where these coroutines are created. mirror_perform() is going to be that point. Signed-off-by: Max Reitz Reviewed-by: Fam Zheng Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/mirror.c | 51 ++

[Qemu-block] [PATCH v3 10/16] hbitmap: Add @advance param to hbitmap_iter_next()

2018-02-28 Thread Max Reitz
This new parameter allows the caller to just query the next dirty position without moving the iterator. Signed-off-by: Max Reitz --- include/qemu/hbitmap.h | 5 - block/backup.c | 2 +- block/dirty-bitmap.c | 2 +- tests/test-hbitmap.c | 26 +- util/hbi

[Qemu-block] [PATCH v3 13/16] block/mirror: Add MirrorBDSOpaque

2018-02-28 Thread Max Reitz
This will allow us to access the block job data when the mirror block driver becomes more complex. Signed-off-by: Max Reitz --- block/mirror.c | 12 1 file changed, 12 insertions(+) diff --git a/block/mirror.c b/block/mirror.c index addfdff4dd..d7bd1d3195 100644 --- a/block/mirror.

Re: [Qemu-block] [PATCH v3 3/7] qapi: Replace qobject_to_X(o) by qobject_to(o, X)

2018-02-28 Thread Max Reitz
On 2018-02-27 15:47, Eric Blake wrote: > On 02/26/2018 06:01 AM, Max Reitz wrote: > +++ b/block.c @@ -1457,7 +1457,7 @@ static QDict *parse_json_filename(const char *filename, Error **errp)    return NULL;    }    -    options = qobject_to_qdict(options_ob

[Qemu-block] [PATCH v3 08/16] block/mirror: Use source as a BdrvChild

2018-02-28 Thread Max Reitz
With this, the mirror_top_bs is no longer just a technically required node in the BDS graph but actually represents the block job operation. Also, drop MirrorBlockJob.source, as we can reach it through mirror_top_bs->backing. Signed-off-by: Max Reitz --- block/mirror.c | 14 ++ 1 fi

[Qemu-block] [PATCH v2 1/4] block: add aio_wait_bh_oneshot()

2018-02-28 Thread Stefan Hajnoczi
Sometimes it's necessary for the main loop thread to run a BH in an IOThread and wait for its completion. This primitive is useful during startup/shutdown to synchronize and avoid race conditions. Signed-off-by: Stefan Hajnoczi --- include/block/aio-wait.h | 13 + util/aio-wait.c

[Qemu-block] [PATCH v3 11/16] test-hbitmap: Add non-advancing iter_next tests

2018-02-28 Thread Max Reitz
Add a function that wraps hbitmap_iter_next() and always calls it in non-advancing mode first, and in advancing mode next. The result should always be the same. By using this function everywhere we called hbitmap_iter_next() before, we should get good test coverage for non-advancing hbitmap_iter_

[Qemu-block] [PATCH v2 2/4] virtio-blk: fix race between .ioeventfd_stop() and vq handler

2018-02-28 Thread Stefan Hajnoczi
If the main loop thread invokes .ioeventfd_stop() just as the vq handler function begins in the IOThread then the handler may lose the race for the AioContext lock. By the time the vq handler is able to acquire the AioContext lock the ioeventfd has already been removed and the handler isn't suppos

[Qemu-block] [PATCH v3 12/16] block/dirty-bitmap: Add bdrv_dirty_iter_next_area

2018-02-28 Thread Max Reitz
This new function allows to look for a consecutively dirty area in a dirty bitmap. Signed-off-by: Max Reitz --- include/block/dirty-bitmap.h | 2 ++ block/dirty-bitmap.c | 55 2 files changed, 57 insertions(+) diff --git a/include/block/dirt

[Qemu-block] [PATCH v3 15/16] block/mirror: Add copy mode QAPI interface

2018-02-28 Thread Max Reitz
This patch allows the user to specify whether to use active or only background mode for mirror block jobs. Currently, this setting will remain constant for the duration of the entire block job. Signed-off-by: Max Reitz --- qapi/block-core.json | 11 +-- include/block/block_int.h |

[Qemu-block] [PATCH v3 14/16] block/mirror: Add active mirroring

2018-02-28 Thread Max Reitz
This patch implements active synchronous mirroring. In active mode, the passive mechanism will still be in place and is used to copy all initially dirty clusters off the source disk; but every write request will write data both to the source and the target disk, so the source cannot be dirtied fas

[Qemu-block] [PATCH v2 4/4] vl: introduce vm_shutdown()

2018-02-28 Thread Stefan Hajnoczi
Commit 00d09fdbbae5f7864ce754913efc84c12fdf9f1a ("vl: pause vcpus before stopping iothreads") and commit dce8921b2baaf95974af8176406881872067adfa ("iothread: Stop threads before main() quits") tried to work around the fact that emulation was still active during termination by stopping iothreads. T

Re: [Qemu-block] [Qemu-devel] [RFC] qemu-img: Drop BLK_ZERO from convert

2018-02-28 Thread Max Reitz
On 2018-02-27 17:17, Stefan Hajnoczi wrote: > On Mon, Feb 26, 2018 at 06:03:13PM +0100, Max Reitz wrote: >> There are filesystems (among which is tmpfs) that have a hard time >> reporting allocation status. That is definitely a bug in them. >> >> However, there is no good reason why qemu-img conve

Re: [Qemu-block] [RFC v4 19/21] blockjobs: Expose manual property

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > Expose the "manual" property via QAPI for the backup-related jobs. > As of this commit, this allows the management API to request the > "concluded" and "dismiss" semantics for backup jobs. > > Signed-off-by: John Snow > --- > blockdev.c

Re: [Qemu-block] [Qemu-devel] [RFC v4 00/21] blockjobs: add explicit job management

2018-02-28 Thread Kevin Wolf
Am 27.02.2018 um 22:01 hat John Snow geschrieben: > All false positives unless someone really would like to advocate for me > to change the format of the table. Agreed, this isn't going to stop the series. Changing the format of the table would make it much harder to read. Kevin

Re: [Qemu-block] [Qemu-devel] [RFC v4 21/21] blockjobs: add manual_mgmt option to transactions

2018-02-28 Thread Kevin Wolf
Am 27.02.2018 um 21:24 hat Eric Blake geschrieben: > On 02/23/2018 05:51 PM, John Snow wrote: > > This allows us to easily force the option for all jobs belonging > > to a transaction to ensure consistency with how all those jobs > > will be handled. > > > > This is purely a convenience. > > > >

Re: [Qemu-block] [RFC v4 18/21] blockjobs: add block-job-finalize

2018-02-28 Thread Kevin Wolf
Am 24.02.2018 um 00:51 hat John Snow geschrieben: > Instead of automatically transitioning from PENDING to CONCLUDED, gate > the .prepare() and .commit() phases behind an explicit acknowledgement > provided by the QMP monitor if manual completion mode has been requested. > > This allows us to perf

[Qemu-block] [PATCH v2 0/4] vl: introduce vm_shutdown()

2018-02-28 Thread Stefan Hajnoczi
v2: * Tackle the .ioeventfd_stop() vs vq handler race by removing the ioeventfd from a BH in the IOThread [Fam] There are several race conditions in virtio-blk/virtio-scsi dataplane code. This patch series addresses them, see the commit description for details on the individual cases. Stefan

[Qemu-block] [PATCH v2 3/4] virtio-scsi: fix race between .ioeventfd_stop() and vq handler

2018-02-28 Thread Stefan Hajnoczi
If the main loop thread invokes .ioeventfd_stop() just as the vq handler function begins in the IOThread then the handler may lose the race for the AioContext lock. By the time the vq handler is able to acquire the AioContext lock the ioeventfd has already been removed and the handler isn't suppos

Re: [Qemu-block] [Qemu-devel] [RFC v4 19/21] blockjobs: Expose manual property

2018-02-28 Thread Kevin Wolf
Am 27.02.2018 um 22:57 hat John Snow geschrieben: > > > On 02/27/2018 03:16 PM, Eric Blake wrote: > > On 02/23/2018 05:51 PM, John Snow wrote: > >> Expose the "manual" property via QAPI for the backup-related jobs. > >> As of this commit, this allows the management API to request the > >> "conclu

Re: [Qemu-block] [Qemu-devel] [PATCH v3 00/16] block/mirror: Add active-sync mirroring

2018-02-28 Thread no-reply
Hi, This series failed docker-build@min-glib build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. Type: series Message-id: 20180228180507.3964-1-mre...@redhat.com Subject: [Qemu-devel] [PATCH v3 00/16] block/mir

Re: [Qemu-block] [Qemu-devel] [RFC v4 18/21] blockjobs: add block-job-finalize

2018-02-28 Thread John Snow
On 02/28/2018 01:15 PM, Kevin Wolf wrote: > Is it because you want to avoid that the user picks an automatic job for > completing the mixed transaction? I wanted to avoid the case that a job without the manual property would be stuck "pending" -- which I have defined to mean that it is waiting o

Re: [Qemu-block] [Qemu-devel] [RFC v4 19/21] blockjobs: Expose manual property

2018-02-28 Thread John Snow
On 02/28/2018 01:23 PM, Kevin Wolf wrote: > But given how often I already said that and people still don't consider > it as an option, this doesn't appear to have been very convincing. So > whatever... *shrug* > Sorry, I didn't mean to give that impression. I initially *did* use two (or more?)

Re: [Qemu-block] [Qemu-devel] [RFC v4 19/21] blockjobs: Expose manual property

2018-02-28 Thread John Snow
On 02/28/2018 01:25 PM, Kevin Wolf wrote: > Am 24.02.2018 um 00:51 hat John Snow geschrieben: >> Expose the "manual" property via QAPI for the backup-related jobs. >> As of this commit, this allows the management API to request the >> "concluded" and "dismiss" semantics for backup jobs. >> >> Sig

Re: [Qemu-block] [Qemu-devel] [RFC v4 21/21] blockjobs: add manual_mgmt option to transactions

2018-02-28 Thread John Snow
On 02/28/2018 01:29 PM, Kevin Wolf wrote: > Am 27.02.2018 um 21:24 hat Eric Blake geschrieben: >> On 02/23/2018 05:51 PM, John Snow wrote: >>> This allows us to easily force the option for all jobs belonging >>> to a transaction to ensure consistency with how all those jobs >>> will be handled. >

Re: [Qemu-block] [Qemu-devel] [RFC v4 08/21] blockjobs: add ABORTING state

2018-02-28 Thread John Snow
On 02/28/2018 09:54 AM, Kevin Wolf wrote: > Am 24.02.2018 um 00:51 hat John Snow geschrieben: >> Add a new state ABORTING. >> >> This makes transitions from normative states to error states explicit >> in the STM, and serves as a disambiguation for which states may complete >> normally when norma

Re: [Qemu-block] [Qemu-devel] [RFC v4 19/21] blockjobs: Expose manual property

2018-02-28 Thread Eric Blake
On 02/28/2018 01:20 PM, John Snow wrote: +if (!backup->has_manual) { +backup->manual = false; +} I think this is unnecessary these days, NULL/0/false is the default value for QMP/QAPI. That's what I get for cargo cult. Eric, confirm/deny? Should I remove the other auto-false

Re: [Qemu-block] [Qemu-devel] [RFC v4 09/21] blockjobs: add CONCLUDED state

2018-02-28 Thread John Snow
On 02/28/2018 10:37 AM, Kevin Wolf wrote: > Am 24.02.2018 um 00:51 hat John Snow geschrieben: >> add a new state "CONCLUDED" that identifies a job that has ceased all >> operations. The wording was chosen to avoid any phrasing that might >> imply success, error, or cancellation. The task has simp

Re: [Qemu-block] [Qemu-devel] [RFC v4 10/21] blockjobs: add NULL state

2018-02-28 Thread John Snow
On 02/28/2018 10:42 AM, Kevin Wolf wrote: > Am 24.02.2018 um 00:51 hat John Snow geschrieben: >> Add a new state that specifically demarcates when we begin to permanently >> demolish a job after it has performed all work. This makes the transition >> explicit in the STM table and highlights condi

Re: [Qemu-block] [Qemu-devel] [RFC] qemu-img: Drop BLK_ZERO from convert

2018-02-28 Thread Max Reitz
On 2018-02-28 19:08, Max Reitz wrote: > On 2018-02-27 17:17, Stefan Hajnoczi wrote: >> On Mon, Feb 26, 2018 at 06:03:13PM +0100, Max Reitz wrote: >>> There are filesystems (among which is tmpfs) that have a hard time >>> reporting allocation status. That is definitely a bug in them. >>> >>> Howeve

Re: [Qemu-block] [Qemu-devel] [RFC] qemu-img: Drop BLK_ZERO from convert

2018-02-28 Thread Max Reitz
On 2018-02-28 21:11, Max Reitz wrote: > On 2018-02-28 19:08, Max Reitz wrote: [...] > In any case it's interesting to see that even the current qemu-img > convert takes longer to read sparsely allocated qcow2/raw files from xfs > than fully allocated images... (That's because I didn't drop the c

Re: [Qemu-block] [Qemu-devel] [RFC v4 11/21] blockjobs: add block_job_dismiss

2018-02-28 Thread John Snow
On 02/28/2018 10:53 AM, Kevin Wolf wrote: > Am 24.02.2018 um 00:51 hat John Snow geschrieben: >> For jobs that have reached their CONCLUDED state, prior to having their >> last reference put down (meaning jobs that have completed successfully, >> unsuccessfully, or have been canceled), allow the

[Qemu-block] [PATCH] iotests: Skip test for ENOMEM error

2018-02-28 Thread Fam Zheng
The AFL image is to exercise the code validating image size, which doesn't work on 32 bit or when out of memory (there is a large allocation before the interesting point). So check that and skip the test, instead of faking the result. Signed-off-by: Fam Zheng --- tests/qemu-iotests/059 | 5 ++---

Re: [Qemu-block] [PATCH v3] iotests: Fix CID for VMDK afl image

2018-02-28 Thread Fam Zheng
On Wed, 02/28 18:50, Max Reitz wrote: > On 2018-02-02 06:23, Fam Zheng wrote: > > This reverts commit 76bf133c4 which updated the reference output, and > > fixed the reference image, because the code path we want to exercise is > > actually the invalid image size. > > > > The descriptor block in t

Re: [Qemu-block] [PATCH v2 0/4] vl: introduce vm_shutdown()

2018-02-28 Thread Fam Zheng
On Wed, 02/28 18:19, Stefan Hajnoczi wrote: > v2: > * Tackle the .ioeventfd_stop() vs vq handler race by removing the ioeventfd >from a BH in the IOThread [Fam] > > There are several race conditions in virtio-blk/virtio-scsi dataplane code. > This patch series addresses them, see the commit d

Re: [Qemu-block] [PATCH v2 05/16] block/mirror: Convert to coroutines

2018-02-28 Thread Fam Zheng
On Wed, 02/28 18:07, Max Reitz wrote: > On 2018-02-28 15:13, Max Reitz wrote: > > On 2018-02-27 08:44, Fam Zheng wrote: > >> On Mon, 01/22 23:07, Max Reitz wrote: > >>> @@ -101,7 +105,7 @@ static BlockErrorAction > >>> mirror_error_action(MirrorBlockJob *s, bool read, > >>> } > >>> } > >>>

Re: [Qemu-block] [PATCH v3 11/16] test-hbitmap: Add non-advancing iter_next tests

2018-02-28 Thread Fam Zheng
On Wed, 02/28 19:05, Max Reitz wrote: > Add a function that wraps hbitmap_iter_next() and always calls it in > non-advancing mode first, and in advancing mode next. The result should > always be the same. > > By using this function everywhere we called hbitmap_iter_next() before, > we should get

Re: [Qemu-block] [PATCH v3 00/16] block/mirror: Add active-sync mirroring

2018-02-28 Thread Fam Zheng
On Wed, 02/28 19:04, Max Reitz wrote: > v3: [Fam] > - Patch 5: Drop shadowing ret declaration > - Patch 11: Added > - Patch 12: Add comment on how @iter is modified by > bdrv_dirty_iter_next_area() > - Patch 14: > - Squashed old patch 12 into this one > - Don't forget write_zeroes a

Re: [Qemu-block] [PATCH v2] iotests: Test creating overlay when guest running

2018-02-28 Thread Fam Zheng
On Wed, 02/14 09:26, Fam Zheng wrote: > Ping? Ping?

Re: [Qemu-block] [PATCH v8 09/21] null: Switch to .bdrv_co_block_status()

2018-02-28 Thread Vladimir Sementsov-Ogievskiy
26.02.2018 17:05, Kevin Wolf wrote: Am 24.02.2018 um 00:38 hat Eric Blake geschrieben: On 02/23/2018 11:05 AM, Kevin Wolf wrote: Am 23.02.2018 um 17:43 hat Eric Blake geschrieben: OFFSET_VALID | DATA might be excusable because I can see that it's convenient that a protocol driver refers to its