Re: [Qemu-devel] [PATCH v5 11/14] block/backup: support block job transactions

[Fam Zheng]

On Fri, 09/11 15:26, Eric Blake wrote:
> On 09/07/2015 01:34 AM, Fam Zheng wrote:
> > From: Stefan Hajnoczi 
> > 
> > Join the transaction when the 'transactional-cancel' QMP argument is
> > true.
> > 
> > This ensures that the sync bitmap is not thrown away if another block
> > job in the transaction is cancelled or fails.  This is critical so
> > incremental backup with multiple disks can be retried in case of
> > cancellation/failure.
> > 
> > Signed-off-by: Stefan Hajnoczi 
> > Signed-off-by: Fam Zheng 
> > ---
> 
> Interface review:
> 
> > +void qmp_blockdev_backup(const char *device, const char *target,
> > + enum MirrorSyncMode sync,
> > + bool has_speed, int64_t speed,
> > + bool has_on_source_error,
> > + BlockdevOnError on_source_error,
> > + bool has_on_target_error,
> > + BlockdevOnError on_target_error,
> > + bool has_transactional_cancel,
> > + bool transactional_cancel,
> > + Error **errp)
> > +{
> > +if (has_transactional_cancel && transactional_cancel) {
> > +error_setg(errp, "Transactional cancel can only be used in the "
> > +   "'transaction' command");
> > +return;
> > +}
> 
> Hmm. It might be nicer if we had two separate qapi structs:
> 
> # used in 'blockdev-backup'
> { 'struct':'BlockdevBackup', 'data': { device ... on-target-error } }
> 
> # used in 'transaction'
> { 'struct':'BlockdevBackupTxn', 'base':'BlockdevBackup',
>   'data': { 'transactional-cancel':'bool' } }
> 
> so that the user can't abuse the boolean from the wrong context.

Very good point, will do that in v6.

Fam

Re: [Qemu-devel] [PATCH v5 11/14] block/backup: support block job transactions

[Max Reitz]

On 07.09.2015 09:34, Fam Zheng wrote:
> From: Stefan Hajnoczi 
> 
> Join the transaction when the 'transactional-cancel' QMP argument is
> true.
> 
> This ensures that the sync bitmap is not thrown away if another block
> job in the transaction is cancelled or fails.  This is critical so
> incremental backup with multiple disks can be retried in case of
> cancellation/failure.
> 
> Signed-off-by: Stefan Hajnoczi 
> Signed-off-by: Fam Zheng 
> ---
>  block/backup.c|  25 +++--
>  blockdev.c| 139 
> --
>  hmp.c |   2 +-
>  include/block/block_int.h |   3 +-
>  qapi/block-core.json  |  16 +-
>  5 files changed, 148 insertions(+), 37 deletions(-)
> 
> diff --git a/block/backup.c b/block/backup.c
> index 9776d9c..3a3dccc 100644
> --- a/block/backup.c
> +++ b/block/backup.c

[snip]

> @@ -545,6 +559,7 @@ void backup_start(BlockDriverState *bs, BlockDriverState 
> *target,
> sync_bitmap : NULL;
>  job->common.len = len;
>  job->common.co = qemu_coroutine_create(backup_run);
> +block_job_txn_add_job(txn, >common);

You're calling this for every single job, so the reference count of txn
will probably be job count + 1. However, block_job_txn_unref() is only
called for one block job by block_job_completed_txn_{abort,success}(). I
think you need to call block_job_txn_unref() for every completed block
job, otherwise it will be leaked if there is more than a single job in
the txn.

Max

>  qemu_coroutine_enter(job->common.co, job);
>  return;
> 



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v5 11/14] block/backup: support block job transactions

[Eric Blake]

On 09/07/2015 01:34 AM, Fam Zheng wrote:
> From: Stefan Hajnoczi 
> 
> Join the transaction when the 'transactional-cancel' QMP argument is
> true.
> 
> This ensures that the sync bitmap is not thrown away if another block
> job in the transaction is cancelled or fails.  This is critical so
> incremental backup with multiple disks can be retried in case of
> cancellation/failure.
> 
> Signed-off-by: Stefan Hajnoczi 
> Signed-off-by: Fam Zheng 
> ---

Interface review:

> +void qmp_blockdev_backup(const char *device, const char *target,
> + enum MirrorSyncMode sync,
> + bool has_speed, int64_t speed,
> + bool has_on_source_error,
> + BlockdevOnError on_source_error,
> + bool has_on_target_error,
> + BlockdevOnError on_target_error,
> + bool has_transactional_cancel,
> + bool transactional_cancel,
> + Error **errp)
> +{
> +if (has_transactional_cancel && transactional_cancel) {
> +error_setg(errp, "Transactional cancel can only be used in the "
> +   "'transaction' command");
> +return;
> +}

Hmm. It might be nicer if we had two separate qapi structs:

# used in 'blockdev-backup'
{ 'struct':'BlockdevBackup', 'data': { device ... on-target-error } }

# used in 'transaction'
{ 'struct':'BlockdevBackupTxn', 'base':'BlockdevBackup',
  'data': { 'transactional-cancel':'bool' } }

so that the user can't abuse the boolean from the wrong context.

[Side notes...
Furthermore, with pending changes coming down the qapi pipeline, we will
generate the C struct so that you can safely do
'(BlockdevBackup*)blockdev_backup_txn' to go from the child back to the
base class.
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg02583.html

> +
> +do_blockdev_backup(device, target, sync, has_speed, speed,
> +   has_on_source_error, on_source_error,
> +   has_on_target_error, on_target_error,
> +   NULL, errp);
> +}

And with other changes coming down the pipe, you could write a function as:

void do_blockdev_backup(BlockdevBackup *args, Error **errp)

by adding 'box':true' to 'blockdev-backup' and make it a bit easier to
pass around all the data without breaking it into multiple parameters.
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg02599.html

But we're not there yet - it may have to be a simplification we apply
after the fact. :)
...]

> +++ b/qapi/block-core.json
> @@ -736,6 +736,11 @@
>  #   default 'report' (no limitations, since this applies to
>  #   a different block device than @device).
>  #
> +# @transactional-cancel: #optional whether failure or cancellation of other
> +#block jobs with @transactional-cancel true in the 
> same
> +#transaction causes the whole group to cancel.
> +#(Since 2.5)

Mention default false.

> +#
>  # Note that @on-source-error and @on-target-error only affect background I/O.
>  # If an error occurs during a guest write request, the device's rerror/werror
>  # actions will be used.
> @@ -747,7 +752,8 @@
>  'sync': 'MirrorSyncMode', '*mode': 'NewImageMode',
>  '*speed': 'int', '*bitmap': 'str',
>  '*on-source-error': 'BlockdevOnError',
> -'*on-target-error': 'BlockdevOnError' } }
> +'*on-target-error': 'BlockdevOnError',
> +'*transactional-cancel': 'bool' } }

See my above comments about the idea of using a parent and child class,
rather than exposing this outside of transaction, especially since you
didn't document that it can't be used outside of a transaction.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

[Qemu-devel] [PATCH v5 11/14] block/backup: support block job transactions

[Fam Zheng]

From: Stefan Hajnoczi 

Join the transaction when the 'transactional-cancel' QMP argument is
true.

This ensures that the sync bitmap is not thrown away if another block
job in the transaction is cancelled or fails.  This is critical so
incremental backup with multiple disks can be retried in case of
cancellation/failure.

Signed-off-by: Stefan Hajnoczi 
Signed-off-by: Fam Zheng 
---
 block/backup.c|  25 +++--
 blockdev.c| 139 --
 hmp.c |   2 +-
 include/block/block_int.h |   3 +-
 qapi/block-core.json  |  16 +-
 5 files changed, 148 insertions(+), 37 deletions(-)

diff --git a/block/backup.c b/block/backup.c
index 9776d9c..3a3dccc 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -226,11 +226,29 @@ static void backup_cleanup_sync_bitmap(BackupBlockJob 
*job, int ret)
 }
 }
 
+static void backup_commit(BlockJob *job)
+{
+BackupBlockJob *s = container_of(job, BackupBlockJob, common);
+if (s->sync_bitmap) {
+backup_cleanup_sync_bitmap(s, 0);
+}
+}
+
+static void backup_abort(BlockJob *job)
+{
+BackupBlockJob *s = container_of(job, BackupBlockJob, common);
+if (s->sync_bitmap) {
+backup_cleanup_sync_bitmap(s, -1);
+}
+}
+
 static const BlockJobDriver backup_job_driver = {
 .instance_size  = sizeof(BackupBlockJob),
 .job_type   = BLOCK_JOB_TYPE_BACKUP,
 .set_speed  = backup_set_speed,
 .iostatus_reset = backup_iostatus_reset,
+.commit = backup_commit,
+.abort  = backup_abort,
 };
 
 static BlockErrorAction backup_error_action(BackupBlockJob *job,
@@ -443,10 +461,6 @@ static void coroutine_fn backup_run(void *opaque)
 /* wait until pending backup_do_cow() calls have completed */
 qemu_co_rwlock_wrlock(>flush_rwlock);
 qemu_co_rwlock_unlock(>flush_rwlock);
-
-if (job->sync_bitmap) {
-backup_cleanup_sync_bitmap(job, ret);
-}
 hbitmap_free(job->bitmap);
 
 bdrv_iostatus_disable(target);
@@ -463,7 +477,7 @@ void backup_start(BlockDriverState *bs, BlockDriverState 
*target,
   BlockdevOnError on_source_error,
   BlockdevOnError on_target_error,
   BlockCompletionFunc *cb, void *opaque,
-  Error **errp)
+  BlockJobTxn *txn, Error **errp)
 {
 int64_t len;
 
@@ -545,6 +559,7 @@ void backup_start(BlockDriverState *bs, BlockDriverState 
*target,
sync_bitmap : NULL;
 job->common.len = len;
 job->common.co = qemu_coroutine_create(backup_run);
+block_job_txn_add_job(txn, >common);
 qemu_coroutine_enter(job->common.co, job);
 return;
 
diff --git a/blockdev.c b/blockdev.c
index 3d5e5cd..db1fdce 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1592,12 +1592,25 @@ typedef struct DriveBackupState {
 BlockJob *job;
 } DriveBackupState;
 
+static void do_drive_backup(const char *device, const char *target,
+bool has_format, const char *format,
+enum MirrorSyncMode sync,
+bool has_mode, enum NewImageMode mode,
+bool has_speed, int64_t speed,
+bool has_bitmap, const char *bitmap,
+bool has_on_source_error,
+BlockdevOnError on_source_error,
+bool has_on_target_error,
+BlockdevOnError on_target_error,
+BlockJobTxn *txn, Error **errp);
+
 static void drive_backup_prepare(BlkActionState *common, Error **errp)
 {
 DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
 BlockDriverState *bs;
 BlockBackend *blk;
 DriveBackup *backup;
+BlockJobTxn *txn = NULL;
 Error *local_err = NULL;
 
 assert(common->action->kind == TRANSACTION_ACTION_KIND_DRIVE_BACKUP);
@@ -1615,15 +1628,20 @@ static void drive_backup_prepare(BlkActionState 
*common, Error **errp)
 state->aio_context = bdrv_get_aio_context(bs);
 aio_context_acquire(state->aio_context);
 
-qmp_drive_backup(backup->device, backup->target,
- backup->has_format, backup->format,
- backup->sync,
- backup->has_mode, backup->mode,
- backup->has_speed, backup->speed,
- backup->has_bitmap, backup->bitmap,
- backup->has_on_source_error, backup->on_source_error,
- backup->has_on_target_error, backup->on_target_error,
- _err);
+if (backup->has_transactional_cancel &&
+backup->transactional_cancel) {
+txn = common->block_job_txn;
+}
+
+do_drive_backup(backup->device, backup->target,
+backup->has_format, backup->format,
+