Re: [PATCH v3 2/4] qemu-img: make --block-size optional for compare --stat

2021-10-29 Thread Eric Blake
On Thu, Oct 28, 2021 at 12:24:39PM +0200, Vladimir Sementsov-Ogievskiy wrote: > Let's detect block-size automatically if not specified by user: > > If both files define cluster-size, use minimum to be more precise. > If both files don't specify cluster-size, use default of 64K > If only one

Re: [PATCH v3 3/4] qemu-img: add --shallow option for qemu-img compare

2021-10-29 Thread Eric Blake
On Thu, Oct 28, 2021 at 12:24:40PM +0200, Vladimir Sementsov-Ogievskiy wrote: > Allow compare only top images of backing chains. This is useful to Allow the comparison of only the top image of backing chains. > compare images with same backing file or to compare incremental images > from the

Re: [PATCH] nbd/server: Advertise MULTI_CONN for shared writable exports

2021-10-29 Thread Eric Blake
On Thu, Oct 28, 2021 at 04:37:36PM +0200, Kevin Wolf wrote: > Am 27.08.2021 um 17:09 hat Eric Blake geschrieben: > > According to the NBD spec, a server advertising > > NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will > > not see any cache inconsistencies: when properly

Re: [PATCH v3 1/4] qemu-img: implement compare --stat

2021-10-29 Thread Eric Blake
On Thu, Oct 28, 2021 at 12:24:38PM +0200, Vladimir Sementsov-Ogievskiy wrote: > With new option qemu-img compare will not stop at first mismatch, but With the new --stat option, qemu-img compare... [Of course the subject line should be self-contained: think 'git shortlog' and friends. But I

Re: [PATCH v2 8/9] qapi: Factor out compat_policy_input_ok()

2021-10-29 Thread Markus Armbruster
Markus Armbruster writes: > The code to check policy for handling deprecated input is triplicated. > Factor it out into compat_policy_input_ok() before I mess with it in > the next commit. > > Signed-off-by: Markus Armbruster > Reviewed-by: Philippe Mathieu-Daudé > --- >

Re: [PATCH v2 8/9] qapi: Factor out compat_policy_input_ok()

2021-10-29 Thread Philippe Mathieu-Daudé
On 10/29/21 18:55, Markus Armbruster wrote: > Markus Armbruster writes: > >> The code to check policy for handling deprecated input is triplicated. >> Factor it out into compat_policy_input_ok() before I mess with it in >> the next commit. >> >> Signed-off-by: Markus Armbruster >> Reviewed-by:

[RFC PATCH 11/15] jobs: remove aiocontext locks since the functions are under BQL

2021-10-29 Thread Emanuele Giuseppe Esposito
In preparation to the job_lock/unlock patch, remove these aiocontext locks. The main reason these two locks are removed here is because they are inside a loop iterating on the jobs list. Once the job_lock is added, it will have to protect the whole loop, wrapping also the aiocontext

[RFC PATCH 14/15] jobs: add missing job locks to replace aiocontext lock

2021-10-29 Thread Emanuele Giuseppe Esposito
find_block_job() and find_job() cannot be handled like all other functions in the previous commit: in order to avoid unneccessary job lock/unlock, replace the aiocontext lock with the job_lock. However, once we start dropping these aiocontex locks we also break the assumptions of the callees in

[RFC PATCH 08/15] job.c: minor adjustments in preparation to job-driver

2021-10-29 Thread Emanuele Giuseppe Esposito
job_event_* functions can be all static, as they are not used outside job.c Add also missing notifier initialization for the on_idle list in job_create(). Signed-off-by: Emanuele Giuseppe Esposito --- job.c | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git

[RFC PATCH 10/15] aio-wait.h: introduce AIO_WAIT_WHILE_UNLOCKED

2021-10-29 Thread Emanuele Giuseppe Esposito
Same as AIO_WAIT_WHILE macro, but if we are in the Main loop do not release and then acquire ctx_ 's aiocontext. Once all Aiocontext locks go away, this macro will replace AIO_WAIT_WHILE. Signed-off-by: Emanuele Giuseppe Esposito --- include/block/aio-wait.h | 15 +++ 1 file

[RFC PATCH 05/15] job-monitor.h: define the job monitor API

2021-10-29 Thread Emanuele Giuseppe Esposito
These functions assume that the job lock is held by the caller, to avoid TOC/TOU conditions. Introduce also additional helpers that define _locked functions (useful when the job_mutex is globally applied). Signed-off-by: Emanuele Giuseppe Esposito --- include/qemu/job-monitor.h | 61

[RFC PATCH 12/15] jobs: protect jobs with job_lock/unlock

2021-10-29 Thread Emanuele Giuseppe Esposito
Introduce the job locking mechanism through the whole job API, following the comments and requirements of job-monitor (assume lock is held) and job-driver (lock is not held). At this point, we do not care if the job lock is inside or outside the aiocontext. The aiocontext is going away and it is

[RFC PATCH 15/15] jobs: remove all unnecessary AioContext locks

2021-10-29 Thread Emanuele Giuseppe Esposito
Now that we removed the aiocontext in find_* functions, we need to remove it also in all other functions of the job API that assumed the lock was held. Therefore: - when possible, remove completely the aiocontext lock/unlock pair - if it is used by some other funcitons too, just leave the job API

[RFC PATCH 13/15] jobs: use job locks and helpers also in the unit tests

2021-10-29 Thread Emanuele Giuseppe Esposito
Add missing job synchronization in the unit tests, with both explicit locks and helpers. Signed-off-by: Emanuele Giuseppe Esposito --- tests/unit/test-bdrv-drain.c | 40 +++--- tests/unit/test-block-iothread.c | 4 +++ tests/unit/test-blockjob-txn.c | 10 ++

[RFC PATCH 09/15] job.c: move inner aiocontext lock in callbacks

2021-10-29 Thread Emanuele Giuseppe Esposito
Instead of having the lock in job_tnx_apply, move it inside in the callback. This will be helpful for next commits, when we introduce job_lock/unlock pairs. job_transition_to_pending() and job_needs_finalize() do not need to be protected by the aiocontext lock. No functional change intended.

[RFC PATCH 03/15] job-common.h: categorize fields in struct Job

2021-10-29 Thread Emanuele Giuseppe Esposito
Categorize the fields in struct Job to understand which need to be protected by the job muutex and which not. Also move job_type() and job_type_str() there, as they are common helper functions. Signed-off-by: Emanuele Giuseppe Esposito --- include/qemu/job-common.h | 62

[RFC PATCH 07/15] job-driver.h: add helper functions

2021-10-29 Thread Emanuele Giuseppe Esposito
These functions will be useful when job_lock is globally applied, as they will allow drivers to access the job struct fields without worrying about the job lock. Now that we are done with the job API header split, update also the comments in blockjob.c (and move them in job.c). Signed-off-by:

[RFC PATCH 06/15] jobs: add job-driver.h

2021-10-29 Thread Emanuele Giuseppe Esposito
job-driver.h contains all functions of job.h that are used by the drivers (JobDriver, BlockJobDriver). These functions are unaware of the job_mutex, so they all will take and release the lock internally. No functional change intended. Signed-off-by: Emanuele Giuseppe Esposito ---

[RFC PATCH 02/15] job.c: make job_lock/unlock public

2021-10-29 Thread Emanuele Giuseppe Esposito
job mutex will be used to protect the job struct elements and list, replacing AioContext locks. Right now use a shared lock for all jobs, in order to keep things simple. Once the AioContext lock is gone, we can introduce per-job locks. Signed-off-by: Emanuele Giuseppe Esposito ---

[RFC PATCH 04/15] jobs: add job-monitor.h

2021-10-29 Thread Emanuele Giuseppe Esposito
job-monitor.h contains all functions of job.h that are used by the monitor and essentially all functions that do not define a JobDriver/Blockdriver. Right now just move the headers, proper categorization and API definition will come in the next commit. No functional change intended.

[RFC PATCH 00/15] job: replace AioContext lock with job_mutex

2021-10-29 Thread Emanuele Giuseppe Esposito
In this series, we want to remove the AioContext lock and instead use the already existent job_mutex to protect the job structures and list. This is part of the work to get rid of AioContext lock usage in favour of smaller granularity locks. In patches 1-3-5-6-7, we split the job API in two

[RFC PATCH 01/15] jobs: add job-common.h

2021-10-29 Thread Emanuele Giuseppe Esposito
job-common.h contains all struct and common function that currently are in job.h and will be shared by job-monitor and job-driver in the next commits. No functional change intended. Signed-off-by: Emanuele Giuseppe Esposito --- include/qemu/job-common.h | 300

Re: [PATCH v2 5/9] qapi: Generalize struct member policy checking

2021-10-29 Thread Philippe Mathieu-Daudé
On 10/29/21 17:34, Markus Armbruster wrote: > Eric Blake writes: > >> On Thu, Oct 28, 2021 at 12:25:16PM +0200, Markus Armbruster wrote: >>> The generated visitor functions call visit_deprecated_accept() and >>> visit_deprecated() when visiting a struct member with special feature >>> flag

Re: [PATCH v2 5/9] qapi: Generalize struct member policy checking

2021-10-29 Thread Philippe Mathieu-Daudé
On 10/29/21 16:01, Markus Armbruster wrote: > Philippe Mathieu-Daudé writes: > >> On 10/28/21 12:25, Markus Armbruster wrote: >>> The generated visitor functions call visit_deprecated_accept() and >>> visit_deprecated() when visiting a struct member with special feature >>> flag 'deprecated'.

Re: [PATCH v2 6/9] qapi: Generalize command policy checking

2021-10-29 Thread Philippe Mathieu-Daudé
On 10/29/21 17:28, Eric Blake wrote: > On Thu, Oct 28, 2021 at 12:25:17PM +0200, Markus Armbruster wrote: >> The code to check command policy can see special feature flag >> 'deprecated' as command flag QCO_DEPRECATED. I want to make feature >> flag 'unstable' visible there as well, so I can add

Re: [PATCH v2 5/9] qapi: Generalize struct member policy checking

2021-10-29 Thread Markus Armbruster
Eric Blake writes: > On Thu, Oct 28, 2021 at 12:25:16PM +0200, Markus Armbruster wrote: >> The generated visitor functions call visit_deprecated_accept() and >> visit_deprecated() when visiting a struct member with special feature >> flag 'deprecated'. This makes the feature flag visible to the

Re: [PATCH v2 6/9] qapi: Generalize command policy checking

2021-10-29 Thread Eric Blake
On Thu, Oct 28, 2021 at 12:25:17PM +0200, Markus Armbruster wrote: > The code to check command policy can see special feature flag > 'deprecated' as command flag QCO_DEPRECATED. I want to make feature > flag 'unstable' visible there as well, so I can add policy for it. > > To let me make it

Re: [PATCH v2 5/9] qapi: Generalize struct member policy checking

2021-10-29 Thread Eric Blake
On Thu, Oct 28, 2021 at 12:25:16PM +0200, Markus Armbruster wrote: > The generated visitor functions call visit_deprecated_accept() and > visit_deprecated() when visiting a struct member with special feature > flag 'deprecated'. This makes the feature flag visible to the actual > visitors. I

Re: [PATCH v2 9/9] qapi: Extend -compat to set policy for unstable interfaces

2021-10-29 Thread Markus Armbruster
Eric Blake writes: > On Thu, Oct 28, 2021 at 12:25:20PM +0200, Markus Armbruster wrote: >> New option parameters unstable-input and unstable-output set policy >> for unstable interfaces just like deprecated-input and >> deprecated-output set policy for deprecated interfaces (see commit >>

Re: [PATCH v2 9/9] qapi: Extend -compat to set policy for unstable interfaces

2021-10-29 Thread Eric Blake
On Thu, Oct 28, 2021 at 12:25:20PM +0200, Markus Armbruster wrote: > New option parameters unstable-input and unstable-output set policy > for unstable interfaces just like deprecated-input and > deprecated-output set policy for deprecated interfaces (see commit > 6dd75472d5 "qemu-options: New

Re: [PATCH 8/9] qapi: Factor out compat_policy_input_ok()

2021-10-29 Thread Markus Armbruster
Philippe Mathieu-Daudé writes: > On 10/26/21 11:46, Markus Armbruster wrote: >> Philippe Mathieu-Daudé writes: >> >>> On 10/25/21 07:25, Markus Armbruster wrote: The code to check policy for handling deprecated input is triplicated. Factor it out into compat_policy_input_ok() before

Re: [PATCH v2 5/9] qapi: Generalize struct member policy checking

2021-10-29 Thread Markus Armbruster
Philippe Mathieu-Daudé writes: > On 10/28/21 12:25, Markus Armbruster wrote: >> The generated visitor functions call visit_deprecated_accept() and >> visit_deprecated() when visiting a struct member with special feature >> flag 'deprecated'. This makes the feature flag visible to the actual >>

Re: [PATCH v2 5/9] qapi: Generalize struct member policy checking

2021-10-29 Thread Philippe Mathieu-Daudé
On 10/28/21 12:25, Markus Armbruster wrote: > The generated visitor functions call visit_deprecated_accept() and > visit_deprecated() when visiting a struct member with special feature > flag 'deprecated'. This makes the feature flag visible to the actual > visitors. I want to make feature flag

Re: [PATCH v2 5/9] qapi: Generalize struct member policy checking

2021-10-29 Thread Markus Armbruster
Juan Quintela writes: > Markus Armbruster wrote: >> The generated visitor functions call visit_deprecated_accept() and >> visit_deprecated() when visiting a struct member with special feature >> flag 'deprecated'. This makes the feature flag visible to the actual >> visitors. I want to make

Re: [PATCH 2/9] qapi: Mark unstable QMP parts with feature 'unstable'

2021-10-29 Thread Eric Blake
On Mon, Oct 25, 2021 at 07:25:25AM +0200, Markus Armbruster wrote: > Add special feature 'unstable' everywhere the name starts with 'x-', > except for InputBarrierProperties member x-origin and > MemoryBackendProperties member x-use-canonical-path-for-ramblock-id, > because these two are actually

Re: [PATCH v2 7/9] qapi: Generalize enum member policy checking

2021-10-29 Thread Philippe Mathieu-Daudé
On 10/28/21 12:25, Markus Armbruster wrote: > The code to check enumeration value policy can see special feature > flag 'deprecated' in QEnumLookup member flags[value]. I want to make > feature flag 'unstable' visible there as well, so I can add policy for > it. > > Instead of extending flags[],

Re: [PATCH v2 4/9] qapi: Tools for sets of special feature flags in generated code

2021-10-29 Thread Philippe Mathieu-Daudé
On 10/28/21 12:25, Markus Armbruster wrote: > New enum QapiSpecialFeature enumerates the special feature flags. > > New helper gen_special_features() returns code to represent a > collection of special feature flags as a bitset. > > The next few commits will put them to use. > > Signed-off-by:

Re: [PATCH v2 4/9] qapi: Tools for sets of special feature flags in generated code

2021-10-29 Thread Juan Quintela
Markus Armbruster wrote: > New enum QapiSpecialFeature enumerates the special feature flags. > > New helper gen_special_features() returns code to represent a > collection of special feature flags as a bitset. > > The next few commits will put them to use. > > Signed-off-by: Markus Armbruster >

Re: [PATCH v2 6/9] qapi: Generalize command policy checking

2021-10-29 Thread Juan Quintela
Markus Armbruster wrote: > The code to check command policy can see special feature flag > 'deprecated' as command flag QCO_DEPRECATED. I want to make feature > flag 'unstable' visible there as well, so I can add policy for it. > > To let me make it visible, add member @special_features (a

Re: [PATCH v2 5/9] qapi: Generalize struct member policy checking

2021-10-29 Thread Juan Quintela
Markus Armbruster wrote: > The generated visitor functions call visit_deprecated_accept() and > visit_deprecated() when visiting a struct member with special feature > flag 'deprecated'. This makes the feature flag visible to the actual > visitors. I want to make feature flag 'unstable' visible