Re: [Qemu-block] [Qemu-devel] [PATCH v2 00/40] Generic background jobs

* Kevin Wolf (kw...@redhat.com) wrote: > Before we can make x-blockdev-create a background job, we need to > generalise the job infrastructure so that it can be used without any > associated block node. Is there any relationship between what this does, and what Marc-André's 'monitor: add

[Qemu-block] [PATCH] sheepdog: Remove unnecessary NULL check in sd_prealloc()

In commit 8b9ad56e9cbfd852a, we removed the code that could result in our getting to sd_prealloc()'s out_with_err_set label with a NULL blk pointer. That makes the NULL check in the error-handling path unnecessary, and Coverity gripes about it (CID 1390636). Delete the redundant check.

Re: [Qemu-block] [PATCH v2 36/40] job: Add lifecycle QMP commands

On 05/18/2018 08:21 AM, Kevin Wolf wrote: This adds QMP commands that control the transition between states of the job lifecycle. Signed-off-by: Kevin Wolf --- qapi/job.json | 99 +++ job-qmp.c | 134

Re: [Qemu-block] [PATCH v2 20/40] job: Move single job finalisation to Job

On 05/18/2018 08:20 AM, Kevin Wolf wrote: This moves the finalisation of a single job from BlockJob to Job. Some part of this code depends on job transactions, and job transactions call this code, we introduce some temporary calls from Job functions to BlockJob ones. This will be fixed once

Re: [Qemu-block] [Qemu-devel] [PATCH v2 12/40] job: Move defer_to_main_loop to Job

On 05/18/2018 09:20 AM, Kevin Wolf wrote: > Move the defer_to_main_loop functionality from BlockJob to Job. > > The code can be simplified because we can use job->aio_context in > job_defer_to_main_loop_bh() now, instead of having to access the > BlockDriverState. > > Probably taking the

Re: [Qemu-block] [PATCH v2 35/40] job: Add JOB_STATUS_CHANGE QMP event

On 05/18/2018 08:21 AM, Kevin Wolf wrote: This adds a QMP event that is emitted whenever a job transitions from one status to another. Signed-off-by: Kevin Wolf --- qapi/job.json | 14 job.c | 10 +++ @@ -157,6 +158,11 @@

Re: [Qemu-block] [Qemu-devel] [PATCH v2 02/40] blockjob: Improve BlockJobInfo.offset/len documentation

On 05/18/2018 09:20 AM, Kevin Wolf wrote: > Clarify that len is just an estimation of the end value of offset, and > that offset increases monotonically while len can change arbitrarily. > > Signed-off-by: Kevin Wolf > --- > qapi/block-core.json | 9 ++--- > 1 file

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

On Fri, May 18, 2018 at 06:09:56PM +0100, Daniel P. Berrangé wrote: > On Fri, May 18, 2018 at 06:30:38PM +0300, Michael S. Tsirkin wrote: > > Hi! > > Right now, QEMU supports multiple machine types within > > a given architecture. This was the case for many architectures > > (like ARM) for a

Re: [Qemu-block] [Qemu-devel] [PATCH v2 01/40] blockjob: Update block-job-pause/resume documentation

On 05/18/2018 09:20 AM, Kevin Wolf wrote: > Commit 0ec4dfb8d changed block-job_pause/resume so that they return an > error if they don't do anything because the job is already > paused/running. It forgot to update the documentation, so do that now. > > Signed-off-by: Kevin Wolf

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

On Fri, May 18, 2018 at 06:30:38PM +0300, Michael S. Tsirkin wrote: > Hi! > Right now, QEMU supports multiple machine types within > a given architecture. This was the case for many architectures > (like ARM) for a while, somewhat more recently this is the case > for x86 with I440FX and Q35

Re: [Qemu-block] storing machine data in qcow images?

On Fri, May 18, 2018 at 06:30:38PM +0300, Michael S. Tsirkin wrote: > Hi! > Right now, QEMU supports multiple machine types within > a given architecture. This was the case for many architectures > (like ARM) for a while, somewhat more recently this is the case > for x86 with I440FX and Q35

Re: [Qemu-block] [PATCH v2 34/40] job: Introduce qapi/job.json

On 05/18/2018 08:21 AM, Kevin Wolf wrote: This adds a separate schema file for all job-related definitions that aren't tied to the block layer. For a start, move the enums JobType, JobStatus and JobVerb. Signed-off-by: Kevin Wolf --- qapi/block-core.json | 90

Re: [Qemu-block] [PATCH v2 08/40] job: Move state transitions to Job

On 05/18/2018 08:20 AM, Kevin Wolf wrote: This moves BlockJob.status and the closely related functions (block_)job_state_transition() and (block_)job_apply_verb to Job. The two QAPI enums are renamed to JobStatus and JobVerb. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz

[Qemu-block] [PATCH v2 2/3] qemu-iotests: Filter NFS paths

NFS paths were only partially filtered in _filter_img_create, _img_info and _filter_img_info, resulting in "nfs://127.0.0.1TEST_DIR/t.IMGFMT". This adds another replacement to the sed calls that matches the test directory not as a host path, but as an NFS URL (the prefix as used for $TEST_IMG).

[Qemu-block] [PATCH v2 0/3] qemu-iotests: Fix NFS + raw

This series fixes the NFS support in qemu-iotests, at least as far as raw is concerned. More fixes would be needed for NFS with qcow2. v2: - Introduce $REMOTE_TEST_DIR to avoid breaking non-NFS cases (and to support additional files in the same directory besides $TEST_IMG) Kevin Wolf (3):

[Qemu-block] [PATCH v2 1/3] qemu-iotests: Fix paths for NFS

Test cases were trying to use nfs:// URLs as local filenames, which made every test fail for NFS. With TEST_IMG and TEST_IMG_FILE set like for the other protocols, NFS tests can pass again. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/common.rc | 4 ++-- 1 file changed, 2

Re: [Qemu-block] [PATCH v2 02/40] blockjob: Improve BlockJobInfo.offset/len documentation

On 05/18/2018 08:20 AM, Kevin Wolf wrote: Clarify that len is just an estimation of the end value of offset, and that offset increases monotonically while len can change arbitrarily. That's tighter than what libvirt promises (and in fact, there are cases where libvirt synthesizes an

Re: [Qemu-block] [Qemu-devel] [PATCH v2 00/40] Generic background jobs

Hi, This series seems to have some coding style problems. See output below for more information: Type: series Message-id: 20180518132114.4070-1-kw...@redhat.com Subject: [Qemu-devel] [PATCH v2 00/40] Generic background jobs === TEST SCRIPT BEGIN === #!/bin/bash BASE=base n=1 total=$(git log

Re: [Qemu-block] [PATCH v2 01/40] blockjob: Update block-job-pause/resume documentation

On 05/18/2018 08:20 AM, Kevin Wolf wrote: Commit 0ec4dfb8d changed block-job_pause/resume so that they return an error if they don't do anything because the job is already paused/running. It forgot to update the documentation, so do that now. Signed-off-by: Kevin Wolf ---

[Qemu-block] [PATCH v2 35/40] job: Add JOB_STATUS_CHANGE QMP event

This adds a QMP event that is emitted whenever a job transitions from one status to another. Signed-off-by: Kevin Wolf --- qapi/job.json | 14 job.c | 10 +++ tests/qemu-iotests/030| 17 +++- tests/qemu-iotests/040

[Qemu-block] [PATCH v2 40/40] qemu-iotests: Test job-* with block jobs

This adds a test case that tests the new job-* QMP commands with mirror and backup block jobs. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/219 | 209 + tests/qemu-iotests/219.out | 327 +

[Qemu-block] [PATCH v2 36/40] job: Add lifecycle QMP commands

This adds QMP commands that control the transition between states of the job lifecycle. Signed-off-by: Kevin Wolf --- qapi/job.json | 99 +++ job-qmp.c | 134 ++ MAINTAINERS

[Qemu-block] [PATCH v2 34/40] job: Introduce qapi/job.json

This adds a separate schema file for all job-related definitions that aren't tied to the block layer. For a start, move the enums JobType, JobStatus and JobVerb. Signed-off-by: Kevin Wolf --- qapi/block-core.json | 90 +---

[Qemu-block] [PATCH v2 30/40] job: Add job_dismiss()

This moves block_job_dismiss() to the Job layer. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz --- include/block/blockjob.h | 9 - include/qemu/job.h | 7 ++- blockdev.c | 10 ++ blockjob.c | 13

[Qemu-block] [PATCH v2 27/40] job: Move completion and cancellation to Job

This moves the top-level job completion and cancellation functions from BlockJob to Job. Signed-off-by: Kevin Wolf --- include/block/blockjob.h | 55 - include/block/blockjob_int.h | 18 -- include/qemu/job.h | 68

[Qemu-block] [PATCH v2 39/40] iotests: Move qmp_to_opts() to VM

qmp_to_opts() used to be a method of QMPTestCase, but recently we started to add more Python test cases that don't make use of QMPTestCase. In order to make the method usable there, move it to VM. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz ---

[Qemu-block] [PATCH v2 37/40] job: Add query-jobs QMP command

This adds a minimal query-jobs implementation that shouldn't pose many design questions. It can later be extended to expose more information, and especially job-specific information. Signed-off-by: Kevin Wolf --- qapi/job.json | 45

[Qemu-block] [PATCH v2 33/40] job: Move progress fields to Job

BlockJob has fields .offset and .len, which are actually misnomers today because they are no longer tied to block device sizes, but just progress counters. As such they make a lot of sense in generic Jobs. This patch moves the fields to Job and renames them to .progress_current and

[Qemu-block] [PATCH v2 31/40] job: Add job_is_ready()

Instead of having a 'bool ready' in BlockJob, add a function that derives its value from the job status. At the same time, this fixes the behaviour to match what the QAPI documentation promises for query-block-job: 'true if the job may be completed'. When the ready flag was introduced in commit

[Qemu-block] [PATCH v2 28/40] block: Cancel job in bdrv_close_all() callers

Now that we cancel all jobs and not only block jobs on shutdown, doing that in bdrv_close_all() isn't really appropriate any more. Move the job_cancel_sync_all() call to the callers, and only assert that there are no job running in bdrv_close_all(). Signed-off-by: Kevin Wolf

[Qemu-block] [PATCH v2 32/40] job: Add job_transition_to_ready()

The transition to the READY state was still performed in the BlockJob layer, in the same function that sent the BLOCK_JOB_READY QMP event. This patch brings the state transition to the Job layer and implements the QMP event using a notifier called from the Job layer, like we already do for other

[Qemu-block] [PATCH v2 29/40] job: Add job_yield()

This moves block_job_yield() to the Job layer. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz --- include/block/blockjob_int.h | 8 include/qemu/job.h | 9 +++-- block/backup.c | 2 +- block/mirror.c |

[Qemu-block] [PATCH v2 25/40] job: Switch transactions to JobTxn

This doesn't actually move any transaction code to Job yet, but it renames the type for transactions from BlockJobTxn to JobTxn and makes them contain Jobs rather than BlockJobs Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz --- include/block/block_int.h

[Qemu-block] [PATCH v2 21/40] job: Convert block_job_cancel_async() to Job

block_job_cancel_async() did two things that were still block job specific: * Setting job->force. This field makes sense on the Job level, so we can just move it. While at it, rename it to job->force_cancel to make its purpose more obvious. * Resetting the I/O status. This can't be moved

[Qemu-block] [PATCH v2 26/40] job: Move transactions to Job

This moves the logic that implements job transactions from BlockJob to Job. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz --- include/block/blockjob.h | 54 -- include/block/blockjob_int.h | 10 -- include/qemu/job.h | 71

[Qemu-block] [PATCH v2 19/40] job: Add job_event_*()

Go through the Job layer in order to send QMP events. For the moment, these functions only call a notifier in the BlockJob layer that sends the existing commands. This uses notifiers rather than JobDriver callbacks because internal users of jobs won't receive QMP events, but might still be

[Qemu-block] [PATCH v2 24/40] job: Move job_finish_sync() to Job

block_job_finish_sync() doesn't contain anything block job specific any more, so it can be moved to Job. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz --- include/qemu/job.h | 9 + block/commit.c | 6 +++--- blockjob.c | 55

[Qemu-block] [PATCH v2 22/40] job: Add job_drain()

block_job_drain() contains a blk_drain() call which cannot be moved to Job, so add a new JobDriver callback JobDriver.drain which has a common implementation for all BlockJobs. In addition to this we keep the existing BlockJobDriver.drain callback that is called by the common drain implementation

[Qemu-block] [PATCH v2 10/40] job: Move cancelled to Job

We cannot yet move the whole logic around job cancelling to Job because it depends on quite a few other things that are still only in BlockJob, but we can move the cancelled field at least. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz Reviewed-by: John

[Qemu-block] [PATCH v2 18/40] blockjob: Split block_job_event_pending()

block_job_event_pending() doesn't only send a QMP event, but it also transitions to the PENDING state. Split the function so that we get one part only sending the event (like other block_job_event_* functions) and another part that does the state transition. Signed-off-by: Kevin Wolf

[Qemu-block] [PATCH v2 11/40] job: Add Job.aio_context

When block jobs need an AioContext, they just take it from their main block node. Generic jobs don't have a main block node, so we need to assign them an AioContext explicitly. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz Reviewed-by: John Snow

[Qemu-block] [PATCH v2 09/40] job: Add reference counting

This moves reference counting from BlockJob to Job. In order to keep calling the BlockJob cleanup code when the job is deleted via job_unref(), introduce a new JobDriver.free callback. Every block job must use block_job_free() for this callback, this is asserted in block_job_create().

[Qemu-block] [PATCH v2 16/40] job: Replace BlockJob.completed with job_is_completed()

Since we introduced an explicit status to block job, BlockJob.completed is redundant because it can be derived from the status. Remove the field from BlockJob and add a function to derive it from the status at the Job level. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz

[Qemu-block] [PATCH v2 15/40] job: Move pause/resume functions to Job

While we already moved the state related to job pausing to Job, the functions to do were still BlockJob only. This commit moves them over to Job. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz Reviewed-by: John Snow ---

[Qemu-block] [PATCH v2 14/40] job: Add job_sleep_ns()

There is nothing block layer specific about block_job_sleep_ns(), so move the function to Job. Signed-off-by: Kevin Wolf Reviewed-by: John Snow Reviewed-by: Max Reitz --- include/block/blockjob_int.h | 11 --- include/qemu/job.h

[Qemu-block] [PATCH v2 08/40] job: Move state transitions to Job

This moves BlockJob.status and the closely related functions (block_)job_state_transition() and (block_)job_apply_verb to Job. The two QAPI enums are renamed to JobStatus and JobVerb. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz Reviewed-by: John Snow

[Qemu-block] [PATCH v2 12/40] job: Move defer_to_main_loop to Job

Move the defer_to_main_loop functionality from BlockJob to Job. The code can be simplified because we can use job->aio_context in job_defer_to_main_loop_bh() now, instead of having to access the BlockDriverState. Probably taking the data->aio_context lock in addition was already unnecessary in

[Qemu-block] [PATCH v2 06/40] job: Add job_delete()

This moves freeing the Job object and its fields from block_job_unref() to job_delete(). Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz Reviewed-by: John Snow --- include/qemu/job.h | 3 +++ blockjob.c | 3 +-- job.c

[Qemu-block] [PATCH v2 03/40] job: Create Job, JobDriver and job_create()

This is the first step towards creating an infrastructure for generic background jobs that aren't tied to a block device. For now, Job only stores its ID and JobDriver, the rest stays in BlockJob. The following patches will move over more parts of BlockJob to Job if they are meaningful outside

[Qemu-block] [PATCH v2 07/40] job: Maintain a list of all jobs

This moves the job list from BlockJob to Job. Now we can check for duplicate IDs in job_create(). Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz Reviewed-by: John Snow --- include/block/blockjob.h | 3 --- include/qemu/job.h | 19

[Qemu-block] [PATCH v2 01/40] blockjob: Update block-job-pause/resume documentation

Commit 0ec4dfb8d changed block-job_pause/resume so that they return an error if they don't do anything because the job is already paused/running. It forgot to update the documentation, so do that now. Signed-off-by: Kevin Wolf --- qapi/block-core.json | 5 ++--- 1 file

[Qemu-block] [PATCH v2 05/40] job: Add JobDriver.job_type

This moves the job_type field from BlockJobDriver to JobDriver. Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz Reviewed-by: John Snow --- include/block/blockjob_int.h | 3 --- include/qemu/job.h | 11 +++

[Qemu-block] [PATCH v2 04/40] job: Rename BlockJobType into JobType

QAPI types aren't externally visible, so we can rename them without causing problems. Before we add a job type to Job, rename the enum so it can be used for more than just block jobs. Signed-off-by: Kevin Wolf Reviewed-by: Eric Blake Reviewed-by: Max Reitz

[Qemu-block] [PATCH v2 02/40] blockjob: Improve BlockJobInfo.offset/len documentation

Clarify that len is just an estimation of the end value of offset, and that offset increases monotonically while len can change arbitrarily. Signed-off-by: Kevin Wolf --- qapi/block-core.json | 9 ++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git

[Qemu-block] [PATCH v2 00/40] Generic background jobs

Before we can make x-blockdev-create a background job, we need to generalise the job infrastructure so that it can be used without any associated block node. This series extracts a Job object from the block job infrastructure, which should contain everything related to jobs that doesn't require

Re: [Qemu-block] [PATCH 0/5] NBD reconnect: preliminary refactoring

17.05.2018 16:48, Eric Blake wrote: On 05/17/2018 04:54 AM, Vladimir Sementsov-Ogievskiy wrote: What about patches 1-4? Still on my list to review (I'm first trying to post an updated proposal on the libvirt list for managing incremental backups); but on first glance, the idea of being able

Re: [Qemu-block] [PATCH v4 5/9] xen_backend: add an emulation of grant copy

On Thu, May 17, 2018 at 04:35:54PM +0100, Paul Durrant wrote: > Not all Xen environments support the xengnttab_grant_copy() operation. > E.g. where the OS is FreeBSD or Xen is older than 4.8.0. > > This patch introduces an emulation of that operation using > xengnttab_map_domain_grant_refs() and

Re: [Qemu-block] [PATCH v4 2/9] xen_backend: add grant table helpers

On Thu, May 17, 2018 at 04:35:51PM +0100, Paul Durrant wrote: > This patch adds grant table helper functions to the xen_backend code to > localize error reporting and use of xen_domid. > > The patch also defers the call to xengnttab_open() until just before the > initialise method in XenDevOps is

Re: [Qemu-block] [PATCH v4 1/9] xen: add a meaningful declaration of grant_copy_segment into xen_common.h

On Thu, May 17, 2018 at 04:35:50PM +0100, Paul Durrant wrote: > Currently the xen_disk source has to carry #ifdef exclusions to compile > against Xen older then 4.8. This is a bit messy so this patch lifts the > definition of struct xengnttab_grant_copy_segment and adds it into the > pre-4.8

Re: [Qemu-block] Ping? Re: [PATCH v6 2/2] iothread: let aio_epoll_disable fit to aio_context_destroy

On Fri, 05/18 16:24, WangJie (Pluto) wrote: > Ping I'll send a pull request with v5. Fam

Re: [Qemu-block] [PATCH 2/3] qemu-iotests: Filter NFS paths

Am 17.05.2018 um 18:52 hat Kevin Wolf geschrieben: > NFS paths were only partially filtered in _filter_img_create, _img_info > and _filter_img_info, resulting in "nfs://127.0.0.1TEST_DIR/t.IMGFMT". > This adds another replacement to the sed calls that matches $TEST_IMG as > a whole. > >

[Qemu-block] Ping? Re: [PATCH v6 2/2] iothread: let aio_epoll_disable fit to aio_context_destroy

Ping On 2018/5/17 14:48, WangJie (Pluto) wrote: > I enjoyed the great benefit of your suggestions, and I will improve next > time. :) > This time, I ask maintainers to touch up the commit message base on version 5 > and merge it, thanks very much. > > On 2018/5/17 14:22, Peter Xu wrote: >> On