From: Stefan Hajnoczi <[email protected]> Live migration of persistent reservations is designed to work without coordination between the source and destination. The destination uses the PREEMPT command to atomically take over the reservation or unregister keys belonging to the source. This approach avoids race conditions where other nodes in the cluster might acquire the reservation and it minimizes communication during the downtime window.
While this works at the SCSI level, this approach is incomplete when using multipathd. Multipathd tracks registered keys and automatically registers them again when paths come up. It is unaware that the destination has preempted the reservation and therefore it will still attempt to register the key on the source host. Use the new PR_HELPER_CLEANUP command to communicate with multipathd on the source after migration completes successfully. The effect is similar to REGISTER AND IGNORE EXISTING KEY, except it also discards multipathd's state so that the source will no longer attempt to register the key. PR_HELPER_CLEANUP is a QEMU-internal command that is rejected by scsi_req_new(), so note that this command is not exposed to guests although direct blk_aio_ioctl(SG_IO) calls can make use of it. Buglink: https://redhat.atlassian.net/browse/RHEL-274966 Signed-off-by: Stefan Hajnoczi <[email protected]> Reviewed-by: Eric Blake <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Paolo Bonzini <[email protected]> --- include/hw/scsi/scsi.h | 1 + hw/scsi/scsi-disk.c | 21 ++++++++++++++++++++- hw/scsi/scsi-generic.c | 29 +++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) diff --git a/include/hw/scsi/scsi.h b/include/hw/scsi/scsi.h index 3eab339b0d2..a2f6f0be7fc 100644 --- a/include/hw/scsi/scsi.h +++ b/include/hw/scsi/scsi.h @@ -256,6 +256,7 @@ SCSIDevice *scsi_device_get(SCSIBus *bus, int channel, int target, int lun); /* scsi-generic.c. */ extern const SCSIReqOps scsi_generic_req_ops; bool scsi_generic_pr_state_preempt(SCSIDevice *s, Error **errp); +bool scsi_generic_pr_state_cleanup(SCSIDevice *s, Error **errp); /* scsi-disk.c */ #define SCSI_DISK_QUIRK_MODE_PAGE_APPLE_VENDOR 0 diff --git a/hw/scsi/scsi-disk.c b/hw/scsi/scsi-disk.c index 3d88e57eded..6341089ddf2 100644 --- a/hw/scsi/scsi-disk.c +++ b/hw/scsi/scsi-disk.c @@ -2773,13 +2773,32 @@ static void scsi_block_migration_failed(SCSIDiskState *s) } } +/* + * Unregister keys on the source after migration since the destination has + * preempted and taken over. This cleanup is not necessary at the SCSI level + * but multipathd tracks registered keys and must be informed that this host no + * longer wishes to be registered. + */ +static void scsi_block_migration_done(SCSIDiskState *s) +{ + SCSIDevice *d = &s->qdev; + Error *local_err = NULL; + + if (!scsi_generic_pr_state_cleanup(d, &local_err)) { + error_prepend(&local_err, "scsi-block migration done: "); + warn_report_err(local_err); + } +} + static int scsi_block_migration_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp) { SCSIDiskState *s = container_of(notifier, SCSIDiskState, migration_notifier); - if (e->type == MIG_EVENT_FAILED) { + if (e->type == MIG_EVENT_DONE) { + scsi_block_migration_done(s); + } else if (e->type == MIG_EVENT_FAILED) { scsi_block_migration_failed(s); } return 0; diff --git a/hw/scsi/scsi-generic.c b/hw/scsi/scsi-generic.c index 153a5275567..e41e973aa92 100644 --- a/hw/scsi/scsi-generic.c +++ b/hw/scsi/scsi-generic.c @@ -652,6 +652,35 @@ bool scsi_generic_pr_state_preempt(SCSIDevice *s, Error **errp) return true; } +/* Clean up keys on source after migration */ +bool scsi_generic_pr_state_cleanup(SCSIDevice *s, Error **errp) +{ + SCSIPRState *pr_state = &s->pr_state; + uint8_t cmd[10] = { + PR_HELPER_CLEANUP, /* OPCODE */ + }; + int ret; + + WITH_QEMU_LOCK_GUARD(&pr_state->mutex) { + if (pr_state->key == 0) { + return true; /* nothing to do */ + } + } + + ret = scsi_SG_IO(s->conf.blk, SG_DXFER_TO_DEV, cmd, sizeof(cmd), NULL, 0, + s->io_timeout, errp); + if (ret < 0) { + error_prepend(errp, "pr-helper CLEANUP failed: "); + return false; + } + + WITH_QEMU_LOCK_GUARD(&pr_state->mutex) { + pr_state->key = 0; + pr_state->resv_type = 0; + } + return true; +} + static void scsi_read_complete(void * opaque, int ret) { SCSIGenericReq *r = (SCSIGenericReq *)opaque; -- 2.56.0
