Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On 2016/2/3 17:55, Wen Congyang wrote: On 02/03/2016 05:32 PM, Stefan Hajnoczi wrote: On Wed, Feb 03, 2016 at 09:29:15AM +0800, Wen Congyang wrote: On 02/02/2016 10:34 PM, Stefan Hajnoczi wrote: On Mon, Feb 01, 2016 at 09:13:36AM +0800, Wen Congyang wrote: On 01/29/2016 11:46 PM, Stefan Hajnoczi wrote: On Fri, Jan 29, 2016 at 11:13:42AM +0800, Changlong Xie wrote: On 01/28/2016 11:15 PM, Stefan Hajnoczi wrote: On Thu, Jan 28, 2016 at 09:13:24AM +0800, Wen Congyang wrote: On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote: On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote: I'm concerned that the bdrv_drain_all() in vm_stop() can take a long time if the disk is slow/failing. bdrv_drain_all() blocks until all in-flight I/O requests have completed. What does the Primary do if the Secondary becomes unresponsive? Actually, we knew this problem. But currently, there seems no better way to resolve it. If you have any ideas? Is it possible to hold the checkpoint information and acknowledge the checkpoint right away, without waiting for bdrv_drain_all() or any Secondory guest activity to complete? There is no way to know that secondary becomes unreponsive. I meant whether it is necessary for the Secondary to vm_stop() and apply the checkpoint before acknowledging the checkpoint to the Primary? I don't understand this. Here is the COLO checkpoint flow: PrimarySecondary new checkpoint notice ---> vm_stop() vm_stop() vm state(device state, memory, cpu) ---> load state <--- done vm_start() vm_start() If the Secondary's vm_stop() call blocks then the Primary is stuck too. I was wondering whether the Secondary can do: <--- done vm_stop() load state It simply receives the checkpoint data into a buffer and immediately replies with "done". vm_stop() and load state is only performed after sending "done". Secondary vm is running, so we should also get the pages that are dirtied by secondary vm, but not dirtied by primary vm. We have two ways to do it: 1. Cache all original memory in the secondary qemu 2. Send the dirty pfn list to primary qemu, and get it. If we ack the checkpoint and the call vm_stop(), we only can select 1. It means that secondary qemu costs more memory. In COLO mode, we will compare the output socket, and will do checkpoint if the application level data is different. If we ack the checkpoint and the call vm_stop(), the client can not get any more data until secondary vm is running again. So we still 'wait' the secondary vm. The advantage is that the Primary will not be delayed by the Secondary. It's an approach that doesn't block. But perhaps it's a problem if the Secondary is slower than the Primary since the Secondary still needs to complete vm_stop() and load state before it can resume execution? I think this really means falling back to microcheckpointing until the Secondary guest can checkpoint. Instead of a blocking vm_stop() we would prevent vcpus from running and when the last pending I/O finishes the Secondary could apply the last checkpoint. This approach does not block QEMU (the monitor, etc). If secondary host becomes unresponsive, it means that we cannot do mocrocheckpointing. We should do failover in this case. This is dangerous because it means that a delay/failure in the Secondary would cause the Primary to fail over to the broken Secondary. All the more reason not to perform blocking operations on the Secondary in the checkpoint code path. If the secondary is broken, primary qemu will take over. Does the Primary use a timeout between "new checkpoint notice" and Secondary's "done" so it can move on if the Secondary is unresponsive? To hailiang: IIRC, we don't use a timeout but I think we can do it. In our design, there is Yes, we may need a timeout to help detecting the unresponsive case which can not be caught by the external heartbeat module. I will investigate it. Thanks, Hailiang an exteranl heartbeat to check primary and secondary status, and decide when to do checkpoint. Thanks Wen Congyang Stefan .
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On 02/03/2016 05:32 PM, Stefan Hajnoczi wrote: > On Wed, Feb 03, 2016 at 09:29:15AM +0800, Wen Congyang wrote: >> On 02/02/2016 10:34 PM, Stefan Hajnoczi wrote: >>> On Mon, Feb 01, 2016 at 09:13:36AM +0800, Wen Congyang wrote: On 01/29/2016 11:46 PM, Stefan Hajnoczi wrote: > On Fri, Jan 29, 2016 at 11:13:42AM +0800, Changlong Xie wrote: >> On 01/28/2016 11:15 PM, Stefan Hajnoczi wrote: >>> On Thu, Jan 28, 2016 at 09:13:24AM +0800, Wen Congyang wrote: On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote: > On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote: >>> I'm concerned that the bdrv_drain_all() in vm_stop() can take a long >>> time if the disk is slow/failing. bdrv_drain_all() blocks until all >>> in-flight I/O requests have completed. What does the Primary do if the >>> Secondary becomes unresponsive? >> >> Actually, we knew this problem. But currently, there seems no better way >> to >> resolve it. If you have any ideas? > > Is it possible to hold the checkpoint information and acknowledge the > checkpoint right away, without waiting for bdrv_drain_all() or any > Secondory guest activity to complete? There is no way to know that secondary becomes unreponsive. >>> >>> I meant whether it is necessary for the Secondary to vm_stop() and apply >>> the checkpoint before acknowledging the checkpoint to the Primary? >> >> I don't understand this. >> Here is the COLO checkpoint flow: >> >> PrimarySecondary >> new checkpoint notice ---> >> vm_stop() vm_stop() >> vm state(device state, memory, cpu) ---> >>load state >> <--- done >> vm_start() vm_start() > > If the Secondary's vm_stop() call blocks then the Primary is stuck too. > > I was wondering whether the Secondary can do: > > <--- done > vm_stop() > load state > > It simply receives the checkpoint data into a buffer and immediately > replies with "done". vm_stop() and load state is only performed after > sending "done". Secondary vm is running, so we should also get the pages that are dirtied by secondary vm, but not dirtied by primary vm. We have two ways to do it: 1. Cache all original memory in the secondary qemu 2. Send the dirty pfn list to primary qemu, and get it. If we ack the checkpoint and the call vm_stop(), we only can select 1. It means that secondary qemu costs more memory. In COLO mode, we will compare the output socket, and will do checkpoint if the application level data is different. If we ack the checkpoint and the call vm_stop(), the client can not get any more data until secondary vm is running again. So we still 'wait' the secondary vm. > > The advantage is that the Primary will not be delayed by the Secondary. > It's an approach that doesn't block. > > But perhaps it's a problem if the Secondary is slower than the Primary > since the Secondary still needs to complete vm_stop() and load state > before it can resume execution? > > I think this really means falling back to microcheckpointing until the > Secondary guest can checkpoint. Instead of a blocking vm_stop() we > would prevent vcpus from running and when the last pending I/O finishes > the Secondary could apply the last checkpoint. This approach does not > block QEMU (the monitor, etc). > If secondary host becomes unresponsive, it means that we cannot do mocrocheckpointing. We should do failover in this case. >>> >>> This is dangerous because it means that a delay/failure in the Secondary >>> would cause the Primary to fail over to the broken Secondary. All the >>> more reason not to perform blocking operations on the Secondary in the >>> checkpoint code path. >> >> If the secondary is broken, primary qemu will take over. > > Does the Primary use a timeout between "new checkpoint notice" and > Secondary's "done" so it can move on if the Secondary is unresponsive? To hailiang: IIRC, we don't use a timeout but I think we can do it. In our design, there is an exteranl heartbeat to check primary and secondary status, and decide when to do checkpoint. Thanks Wen Congyang > > Stefan >
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On Wed, Feb 03, 2016 at 09:29:15AM +0800, Wen Congyang wrote: > On 02/02/2016 10:34 PM, Stefan Hajnoczi wrote: > > On Mon, Feb 01, 2016 at 09:13:36AM +0800, Wen Congyang wrote: > >> On 01/29/2016 11:46 PM, Stefan Hajnoczi wrote: > >>> On Fri, Jan 29, 2016 at 11:13:42AM +0800, Changlong Xie wrote: > On 01/28/2016 11:15 PM, Stefan Hajnoczi wrote: > > On Thu, Jan 28, 2016 at 09:13:24AM +0800, Wen Congyang wrote: > >> On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote: > >>> On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote: > > I'm concerned that the bdrv_drain_all() in vm_stop() can take a long > > time if the disk is slow/failing. bdrv_drain_all() blocks until all > > in-flight I/O requests have completed. What does the Primary do if the > > Secondary becomes unresponsive? > > Actually, we knew this problem. But currently, there seems no better way > to > resolve it. If you have any ideas? > >>> > >>> Is it possible to hold the checkpoint information and acknowledge the > >>> checkpoint right away, without waiting for bdrv_drain_all() or any > >>> Secondory guest activity to complete? > >> > >> There is no way to know that secondary becomes unreponsive. > > > > I meant whether it is necessary for the Secondary to vm_stop() and apply > > the checkpoint before acknowledging the checkpoint to the Primary? > > I don't understand this. > Here is the COLO checkpoint flow: > > PrimarySecondary > new checkpoint notice ---> > vm_stop() vm_stop() > vm state(device state, memory, cpu) ---> >load state > <--- done > vm_start() vm_start() If the Secondary's vm_stop() call blocks then the Primary is stuck too. I was wondering whether the Secondary can do: <--- done vm_stop() load state It simply receives the checkpoint data into a buffer and immediately replies with "done". vm_stop() and load state is only performed after sending "done". The advantage is that the Primary will not be delayed by the Secondary. It's an approach that doesn't block. But perhaps it's a problem if the Secondary is slower than the Primary since the Secondary still needs to complete vm_stop() and load state before it can resume execution? > >>> I think this really means falling back to microcheckpointing until the > >>> Secondary guest can checkpoint. Instead of a blocking vm_stop() we > >>> would prevent vcpus from running and when the last pending I/O finishes > >>> the Secondary could apply the last checkpoint. This approach does not > >>> block QEMU (the monitor, etc). > >>> > >> > >> If secondary host becomes unresponsive, it means that we cannot do > >> mocrocheckpointing. > >> We should do failover in this case. > > > > This is dangerous because it means that a delay/failure in the Secondary > > would cause the Primary to fail over to the broken Secondary. All the > > more reason not to perform blocking operations on the Secondary in the > > checkpoint code path. > > If the secondary is broken, primary qemu will take over. Does the Primary use a timeout between "new checkpoint notice" and Secondary's "done" so it can move on if the Secondary is unresponsive? Stefan signature.asc Description: PGP signature
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On 02/02/2016 10:34 PM, Stefan Hajnoczi wrote: > On Mon, Feb 01, 2016 at 09:13:36AM +0800, Wen Congyang wrote: >> On 01/29/2016 11:46 PM, Stefan Hajnoczi wrote: >>> On Fri, Jan 29, 2016 at 11:13:42AM +0800, Changlong Xie wrote: On 01/28/2016 11:15 PM, Stefan Hajnoczi wrote: > On Thu, Jan 28, 2016 at 09:13:24AM +0800, Wen Congyang wrote: >> On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote: >>> On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote: > I'm concerned that the bdrv_drain_all() in vm_stop() can take a long > time if the disk is slow/failing. bdrv_drain_all() blocks until all > in-flight I/O requests have completed. What does the Primary do if the > Secondary becomes unresponsive? Actually, we knew this problem. But currently, there seems no better way to resolve it. If you have any ideas? >>> >>> Is it possible to hold the checkpoint information and acknowledge the >>> checkpoint right away, without waiting for bdrv_drain_all() or any >>> Secondory guest activity to complete? >> >> There is no way to know that secondary becomes unreponsive. > > I meant whether it is necessary for the Secondary to vm_stop() and apply > the checkpoint before acknowledging the checkpoint to the Primary? I don't understand this. Here is the COLO checkpoint flow: PrimarySecondary new checkpoint notice ---> vm_stop() vm_stop() vm state(device state, memory, cpu) ---> load state <--- done vm_start() vm_start() > >>> I think this really means falling back to microcheckpointing until the >>> Secondary guest can checkpoint. Instead of a blocking vm_stop() we >>> would prevent vcpus from running and when the last pending I/O finishes >>> the Secondary could apply the last checkpoint. This approach does not >>> block QEMU (the monitor, etc). >>> >> >> If secondary host becomes unresponsive, it means that we cannot do >> mocrocheckpointing. >> We should do failover in this case. > > This is dangerous because it means that a delay/failure in the Secondary > would cause the Primary to fail over to the broken Secondary. All the > more reason not to perform blocking operations on the Secondary in the > checkpoint code path. If the secondary is broken, primary qemu will take over. Thanks Wen Congyang > > Stefan >
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On Mon, Feb 01, 2016 at 09:13:36AM +0800, Wen Congyang wrote: > On 01/29/2016 11:46 PM, Stefan Hajnoczi wrote: > > On Fri, Jan 29, 2016 at 11:13:42AM +0800, Changlong Xie wrote: > >> On 01/28/2016 11:15 PM, Stefan Hajnoczi wrote: > >>> On Thu, Jan 28, 2016 at 09:13:24AM +0800, Wen Congyang wrote: > On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote: > > On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote: > >>> I'm concerned that the bdrv_drain_all() in vm_stop() can take a long > >>> time if the disk is slow/failing. bdrv_drain_all() blocks until all > >>> in-flight I/O requests have completed. What does the Primary do if the > >>> Secondary becomes unresponsive? > >> > >> Actually, we knew this problem. But currently, there seems no better way to > >> resolve it. If you have any ideas? > > > > Is it possible to hold the checkpoint information and acknowledge the > > checkpoint right away, without waiting for bdrv_drain_all() or any > > Secondory guest activity to complete? > > There is no way to know that secondary becomes unreponsive. I meant whether it is necessary for the Secondary to vm_stop() and apply the checkpoint before acknowledging the checkpoint to the Primary? > > I think this really means falling back to microcheckpointing until the > > Secondary guest can checkpoint. Instead of a blocking vm_stop() we > > would prevent vcpus from running and when the last pending I/O finishes > > the Secondary could apply the last checkpoint. This approach does not > > block QEMU (the monitor, etc). > > > > If secondary host becomes unresponsive, it means that we cannot do > mocrocheckpointing. > We should do failover in this case. This is dangerous because it means that a delay/failure in the Secondary would cause the Primary to fail over to the broken Secondary. All the more reason not to perform blocking operations on the Secondary in the checkpoint code path. Stefan signature.asc Description: PGP signature
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On 01/29/2016 11:46 PM, Stefan Hajnoczi wrote: > On Fri, Jan 29, 2016 at 11:13:42AM +0800, Changlong Xie wrote: >> On 01/28/2016 11:15 PM, Stefan Hajnoczi wrote: >>> On Thu, Jan 28, 2016 at 09:13:24AM +0800, Wen Congyang wrote: On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote: > On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote: >>> I'm concerned that the bdrv_drain_all() in vm_stop() can take a long >>> time if the disk is slow/failing. bdrv_drain_all() blocks until all >>> in-flight I/O requests have completed. What does the Primary do if the >>> Secondary becomes unresponsive? >> >> Actually, we knew this problem. But currently, there seems no better way to >> resolve it. If you have any ideas? > > Is it possible to hold the checkpoint information and acknowledge the > checkpoint right away, without waiting for bdrv_drain_all() or any > Secondory guest activity to complete? There is no way to know that secondary becomes unreponsive. > > I think this really means falling back to microcheckpointing until the > Secondary guest can checkpoint. Instead of a blocking vm_stop() we > would prevent vcpus from running and when the last pending I/O finishes > the Secondary could apply the last checkpoint. This approach does not > block QEMU (the monitor, etc). > If secondary host becomes unresponsive, it means that we cannot do mocrocheckpointing. We should do failover in this case. Thanks Wen Congyang
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On Fri, Jan 29, 2016 at 11:13:42AM +0800, Changlong Xie wrote: > On 01/28/2016 11:15 PM, Stefan Hajnoczi wrote: > >On Thu, Jan 28, 2016 at 09:13:24AM +0800, Wen Congyang wrote: > >>On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote: > >>>On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote: > >I'm concerned that the bdrv_drain_all() in vm_stop() can take a long > >time if the disk is slow/failing. bdrv_drain_all() blocks until all > >in-flight I/O requests have completed. What does the Primary do if the > >Secondary becomes unresponsive? > > Actually, we knew this problem. But currently, there seems no better way to > resolve it. If you have any ideas? Is it possible to hold the checkpoint information and acknowledge the checkpoint right away, without waiting for bdrv_drain_all() or any Secondory guest activity to complete? I think this really means falling back to microcheckpointing until the Secondary guest can checkpoint. Instead of a blocking vm_stop() we would prevent vcpus from running and when the last pending I/O finishes the Secondary could apply the last checkpoint. This approach does not block QEMU (the monitor, etc). signature.asc Description: PGP signature
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On 01/28/2016 11:15 PM, Stefan Hajnoczi wrote:
On Thu, Jan 28, 2016 at 09:13:24AM +0800, Wen Congyang wrote:
On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote:
+static void secondary_do_checkpoint(BDRVReplicationState *s, Error **errp)
+{
+Error *local_err = NULL;
+int ret;
+
+if (!s->secondary_disk->job) {
+error_setg(errp, "Backup job is cancelled unexpectedly");
+return;
+}
+
+block_job_do_checkpoint(s->secondary_disk->job, &local_err);
+if (local_err) {
+error_propagate(errp, local_err);
+return;
+}
+
+ret = s->active_disk->drv->bdrv_make_empty(s->active_disk);
What happens to in-flight requests to the active and hidden disks?
we MUST call do_checkpoint() when the vm is stopped.
Please document the environment under which the block replication
callback functions run.
OK
I'm concerned that the bdrv_drain_all() in vm_stop() can take a long
time if the disk is slow/failing. bdrv_drain_all() blocks until all
in-flight I/O requests have completed. What does the Primary do if the
Secondary becomes unresponsive?
Actually, we knew this problem. But currently, there seems no better way
to resolve it. If you have any ideas?
+switch (s->mode) {
+case REPLICATION_MODE_PRIMARY:
+break;
+case REPLICATION_MODE_SECONDARY:
+s->active_disk = bs->file->bs;
+if (!bs->file->bs->backing) {
+error_setg(errp, "Active disk doesn't have backing file");
+return;
+}
+
+s->hidden_disk = s->active_disk->backing->bs;
+if (!s->hidden_disk->backing) {
+error_setg(errp, "Hidden disk doesn't have backing file");
+return;
+}
+
+s->secondary_disk = s->hidden_disk->backing->bs;
+if (!s->secondary_disk->blk) {
+error_setg(errp, "The secondary disk doesn't have block backend");
+return;
+}
Kevin: Is code allowed to stash away BlockDriverState pointers for
convenience or should it keep the BdrvChild pointers instead? In order
for replication to work as expected, the graph shouldn't change but for
consistency maybe BdrvChild is best.
I asked Kevin about this on IRC and he agreed that BdrvChild should be
used instead of holding on to BlockDriverState * pointers. Although
these pointers will not change during replication (if the op blockers
are set up correctly), it's more consistent and certainly safer to go
through BdrvChild.
Ok
+/* start backup job now */
+error_setg(&s->blocker,
+ "block device is in use by internal backup job");
+bdrv_op_block_all(s->top_bs, s->blocker);
+bdrv_op_unblock(s->top_bs, BLOCK_OP_TYPE_DATAPLANE, s->blocker);
+bdrv_ref(s->hidden_disk);
Why is the explicit reference to hidden_disk (but not secondary_disk or
active_disk) is necessary?
IIRC, we should reference the backup target before calling backup_start(),
and we will reference the backup source in backup_start().
I'm not sure why this is necessary since they are part of the backing
chain.
Just as Wen said, we should reference the backup target before calling
backup_start() to protect it from destroying, if backup job is stopped
unexpectedly.
If it is necessary, please add a comment so it's clear why the reference
is being taken.
Ok
Stefan
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On Thu, Jan 28, 2016 at 09:13:24AM +0800, Wen Congyang wrote:
> On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote:
> > On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote:
> >> +static void secondary_do_checkpoint(BDRVReplicationState *s, Error **errp)
> >> +{
> >> +Error *local_err = NULL;
> >> +int ret;
> >> +
> >> +if (!s->secondary_disk->job) {
> >> +error_setg(errp, "Backup job is cancelled unexpectedly");
> >> +return;
> >> +}
> >> +
> >> +block_job_do_checkpoint(s->secondary_disk->job, &local_err);
> >> +if (local_err) {
> >> +error_propagate(errp, local_err);
> >> +return;
> >> +}
> >> +
> >> +ret = s->active_disk->drv->bdrv_make_empty(s->active_disk);
> >
> > What happens to in-flight requests to the active and hidden disks?
>
> we MUST call do_checkpoint() when the vm is stopped.
Please document the environment under which the block replication
callback functions run.
I'm concerned that the bdrv_drain_all() in vm_stop() can take a long
time if the disk is slow/failing. bdrv_drain_all() blocks until all
in-flight I/O requests have completed. What does the Primary do if the
Secondary becomes unresponsive?
> >> +switch (s->mode) {
> >> +case REPLICATION_MODE_PRIMARY:
> >> +break;
> >> +case REPLICATION_MODE_SECONDARY:
> >> +s->active_disk = bs->file->bs;
> >> +if (!bs->file->bs->backing) {
> >> +error_setg(errp, "Active disk doesn't have backing file");
> >> +return;
> >> +}
> >> +
> >> +s->hidden_disk = s->active_disk->backing->bs;
> >> +if (!s->hidden_disk->backing) {
> >> +error_setg(errp, "Hidden disk doesn't have backing file");
> >> +return;
> >> +}
> >> +
> >> +s->secondary_disk = s->hidden_disk->backing->bs;
> >> +if (!s->secondary_disk->blk) {
> >> +error_setg(errp, "The secondary disk doesn't have block
> >> backend");
> >> +return;
> >> +}
> >
> > Kevin: Is code allowed to stash away BlockDriverState pointers for
> > convenience or should it keep the BdrvChild pointers instead? In order
> > for replication to work as expected, the graph shouldn't change but for
> > consistency maybe BdrvChild is best.
I asked Kevin about this on IRC and he agreed that BdrvChild should be
used instead of holding on to BlockDriverState * pointers. Although
these pointers will not change during replication (if the op blockers
are set up correctly), it's more consistent and certainly safer to go
through BdrvChild.
> >> +/* start backup job now */
> >> +error_setg(&s->blocker,
> >> + "block device is in use by internal backup job");
> >> +bdrv_op_block_all(s->top_bs, s->blocker);
> >> +bdrv_op_unblock(s->top_bs, BLOCK_OP_TYPE_DATAPLANE, s->blocker);
> >> +bdrv_ref(s->hidden_disk);
> >
> > Why is the explicit reference to hidden_disk (but not secondary_disk or
> > active_disk) is necessary?
>
> IIRC, we should reference the backup target before calling backup_start(),
> and we will reference the backup source in backup_start().
I'm not sure why this is necessary since they are part of the backing
chain.
If it is necessary, please add a comment so it's clear why the reference
is being taken.
Stefan
signature.asc
Description: PGP signature
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On 01/27/2016 10:46 PM, Stefan Hajnoczi wrote:
> On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote:
>> From: Wen Congyang
>>
>> Signed-off-by: Wen Congyang
>> Signed-off-by: zhanghailiang
>> Signed-off-by: Gonglei
>> Signed-off-by: Changlong Xie
>> ---
>> block/Makefile.objs | 1 +
>> block/replication-comm.c | 66 +
>> block/replication.c | 590
>> +++
>> include/block/replication-comm.h | 50
>> qapi/block-core.json | 13 +
>> 5 files changed, 720 insertions(+)
>> create mode 100644 block/replication-comm.c
>> create mode 100644 block/replication.c
>> create mode 100644 include/block/replication-comm.h
>>
>> diff --git a/block/Makefile.objs b/block/Makefile.objs
>> index fa05f37..7037662 100644
>> --- a/block/Makefile.objs
>> +++ b/block/Makefile.objs
>> @@ -23,6 +23,7 @@ block-obj-$(CONFIG_LIBSSH2) += ssh.o
>> block-obj-y += accounting.o
>> block-obj-y += write-threshold.o
>> block-obj-y += backup.o
>> +block-obj-y += replication-comm.o replication.o
>>
>> common-obj-y += stream.o
>> common-obj-y += commit.o
>> diff --git a/block/replication-comm.c b/block/replication-comm.c
>> new file mode 100644
>> index 000..8af748b
>> --- /dev/null
>> +++ b/block/replication-comm.c
>> @@ -0,0 +1,66 @@
>> +/*
>> + * Replication Block filter
>
> Is the start/stop/checkpoint callback interface only useful for block
> replication?
>
> This seems like a generic interface for registering with COLO. Other
> components (networking, etc) might also need start/stop/checkpoint
> callbacks. If that's the case then this code should be outside block/
> and the brs->bs field should either be void *opaque or removed (the
> caller needs to use container_of()).
Yes, we will do it in the next version.
>
>> + *
>> + * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
>> + * Copyright (c) 2015 Intel Corporation
>> + * Copyright (c) 2015 FUJITSU LIMITED
>> + *
>> + * Author:
>> + * Wen Congyang
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + */
>> +
>> +#include "block/replication-comm.h"
>> +
>> +static QLIST_HEAD(, BlockReplicationState) block_replication_states;
>> +
>> +BlockReplicationState *block_replication_new(BlockDriverState *bs,
>> + BlockReplicationOps *ops)
>> +{
>> +BlockReplicationState *brs;
>> +
>> +brs = g_new0(BlockReplicationState, 1);
>> +brs->bs = bs;
>> +brs->ops = ops;
>> +QLIST_INSERT_HEAD(&block_replication_states, brs, node);
>> +
>> +return brs;
>> +}
>> +
>> +void block_replication_remove(BlockReplicationState *brs)
>> +{
>> +QLIST_REMOVE(brs, node);
>> +g_free(brs);
>> +}
>> +
>> +void block_replication_start_all(ReplicationMode mode, Error **errp)
>> +{
>> +BlockReplicationState *brs, *next;
>> +QLIST_FOREACH_SAFE(brs, &block_replication_states, node, next) {
>> +if (brs->ops && brs->ops->start) {
>> +brs->ops->start(brs, mode, errp);
>> +}
>> +}
>> +}
>> +
>> +void block_replication_do_checkpoint_all(Error **errp)
>> +{
>> +BlockReplicationState *brs, *next;
>> +QLIST_FOREACH_SAFE(brs, &block_replication_states, node, next) {
>> +if (brs->ops && brs->ops->checkpoint) {
>> +brs->ops->checkpoint(brs, errp);
>> +}
>> +}
>> +}
>> +
>> +void block_replication_stop_all(bool failover, Error **errp)
>> +{
>> +BlockReplicationState *brs, *next;
>> +QLIST_FOREACH_SAFE(brs, &block_replication_states, node, next) {
>> +if (brs->ops && brs->ops->stop) {
>> +brs->ops->stop(brs, failover, errp);
>> +}
>> +}
>> +}
>> diff --git a/block/replication.c b/block/replication.c
>> new file mode 100644
>> index 000..29c677a
>> --- /dev/null
>> +++ b/block/replication.c
>> @@ -0,0 +1,590 @@
>> +/*
>> + * Replication Block filter
>> + *
>> + * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
>> + * Copyright (c) 2015 Intel Corporation
>> + * Copyright (c) 2015 FUJITSU LIMITED
>> + *
>> + * Author:
>> + * Wen Congyang
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + */
>> +
>> +#include "qemu-common.h"
>> +#include "block/blockjob.h"
>> +#include "block/nbd.h"
>> +#include "block/replication-comm.h"
>> +
>> +typedef struct BDRVReplicationState {
>> +ReplicationMode mode;
>> +int replication_state;
>> +BlockDriverState *active_disk;
>> +BlockDriverState *hidden_disk;
>> +BlockDriverState *secondary_disk;
>> +BlockDriverState *top_bs;
>> +BlockReplicationState *brs;
>> +Error *blocker;
>> +int orig_hidden_flags;
>> +int orig_secondary_flags;
>> +int error;
>> +} BDRVReplicationState;
>> +
>> +enum {
>> +BLOCK_REPLICATION_NONE, /* block replica
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On Wed, Jan 13, 2016 at 05:18:31PM +0800, Changlong Xie wrote:
> From: Wen Congyang
>
> Signed-off-by: Wen Congyang
> Signed-off-by: zhanghailiang
> Signed-off-by: Gonglei
> Signed-off-by: Changlong Xie
> ---
> block/Makefile.objs | 1 +
> block/replication-comm.c | 66 +
> block/replication.c | 590
> +++
> include/block/replication-comm.h | 50
> qapi/block-core.json | 13 +
> 5 files changed, 720 insertions(+)
> create mode 100644 block/replication-comm.c
> create mode 100644 block/replication.c
> create mode 100644 include/block/replication-comm.h
>
> diff --git a/block/Makefile.objs b/block/Makefile.objs
> index fa05f37..7037662 100644
> --- a/block/Makefile.objs
> +++ b/block/Makefile.objs
> @@ -23,6 +23,7 @@ block-obj-$(CONFIG_LIBSSH2) += ssh.o
> block-obj-y += accounting.o
> block-obj-y += write-threshold.o
> block-obj-y += backup.o
> +block-obj-y += replication-comm.o replication.o
>
> common-obj-y += stream.o
> common-obj-y += commit.o
> diff --git a/block/replication-comm.c b/block/replication-comm.c
> new file mode 100644
> index 000..8af748b
> --- /dev/null
> +++ b/block/replication-comm.c
> @@ -0,0 +1,66 @@
> +/*
> + * Replication Block filter
Is the start/stop/checkpoint callback interface only useful for block
replication?
This seems like a generic interface for registering with COLO. Other
components (networking, etc) might also need start/stop/checkpoint
callbacks. If that's the case then this code should be outside block/
and the brs->bs field should either be void *opaque or removed (the
caller needs to use container_of()).
> + *
> + * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
> + * Copyright (c) 2015 Intel Corporation
> + * Copyright (c) 2015 FUJITSU LIMITED
> + *
> + * Author:
> + * Wen Congyang
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#include "block/replication-comm.h"
> +
> +static QLIST_HEAD(, BlockReplicationState) block_replication_states;
> +
> +BlockReplicationState *block_replication_new(BlockDriverState *bs,
> + BlockReplicationOps *ops)
> +{
> +BlockReplicationState *brs;
> +
> +brs = g_new0(BlockReplicationState, 1);
> +brs->bs = bs;
> +brs->ops = ops;
> +QLIST_INSERT_HEAD(&block_replication_states, brs, node);
> +
> +return brs;
> +}
> +
> +void block_replication_remove(BlockReplicationState *brs)
> +{
> +QLIST_REMOVE(brs, node);
> +g_free(brs);
> +}
> +
> +void block_replication_start_all(ReplicationMode mode, Error **errp)
> +{
> +BlockReplicationState *brs, *next;
> +QLIST_FOREACH_SAFE(brs, &block_replication_states, node, next) {
> +if (brs->ops && brs->ops->start) {
> +brs->ops->start(brs, mode, errp);
> +}
> +}
> +}
> +
> +void block_replication_do_checkpoint_all(Error **errp)
> +{
> +BlockReplicationState *brs, *next;
> +QLIST_FOREACH_SAFE(brs, &block_replication_states, node, next) {
> +if (brs->ops && brs->ops->checkpoint) {
> +brs->ops->checkpoint(brs, errp);
> +}
> +}
> +}
> +
> +void block_replication_stop_all(bool failover, Error **errp)
> +{
> +BlockReplicationState *brs, *next;
> +QLIST_FOREACH_SAFE(brs, &block_replication_states, node, next) {
> +if (brs->ops && brs->ops->stop) {
> +brs->ops->stop(brs, failover, errp);
> +}
> +}
> +}
> diff --git a/block/replication.c b/block/replication.c
> new file mode 100644
> index 000..29c677a
> --- /dev/null
> +++ b/block/replication.c
> @@ -0,0 +1,590 @@
> +/*
> + * Replication Block filter
> + *
> + * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
> + * Copyright (c) 2015 Intel Corporation
> + * Copyright (c) 2015 FUJITSU LIMITED
> + *
> + * Author:
> + * Wen Congyang
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu-common.h"
> +#include "block/blockjob.h"
> +#include "block/nbd.h"
> +#include "block/replication-comm.h"
> +
> +typedef struct BDRVReplicationState {
> +ReplicationMode mode;
> +int replication_state;
> +BlockDriverState *active_disk;
> +BlockDriverState *hidden_disk;
> +BlockDriverState *secondary_disk;
> +BlockDriverState *top_bs;
> +BlockReplicationState *brs;
> +Error *blocker;
> +int orig_hidden_flags;
> +int orig_secondary_flags;
> +int error;
> +} BDRVReplicationState;
> +
> +enum {
> +BLOCK_REPLICATION_NONE, /* block replication is not started
> */
> +BLOCK_REPLICATION_RUNNING, /* block replication is running */
> +BLOCK_REPLICATION_FAILOVER, /* failover is running in background
> */
> +BLOCK_REPLICATION_FAILOVER_FAILED, /* failover failed*
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On 01/20/2016 08:04 AM, Eric Blake wrote:
On 01/13/2016 02:18 AM, Changlong Xie wrote:
From: Wen Congyang
Signed-off-by: Wen Congyang
Signed-off-by: zhanghailiang
Signed-off-by: Gonglei
Signed-off-by: Changlong Xie
---
block/Makefile.objs | 1 +
block/replication-comm.c | 66 +
block/replication.c | 590 +++
include/block/replication-comm.h | 50
qapi/block-core.json | 13 +
5 files changed, 720 insertions(+)
create mode 100644 block/replication-comm.c
create mode 100644 block/replication.c
create mode 100644 include/block/replication-comm.h
Just a high-level overview, mainly on the user-visible interface and
things that caught my eye.
Hi eric, thanks for your patience.
+++ b/block/replication-comm.c
@@ -0,0 +1,66 @@
+/*
+ * Replication Block filter
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 Intel Corporation
+ * Copyright (c) 2015 FUJITSU LIMITED
Do you want to claim 2016 in any of this?
Will correct all Copyright issues in next version.
+
+enum {
+BLOCK_REPLICATION_NONE, /* block replication is not started */
+BLOCK_REPLICATION_RUNNING, /* block replication is running */
+BLOCK_REPLICATION_FAILOVER, /* failover is running in background */
+BLOCK_REPLICATION_FAILOVER_FAILED, /* failover failed*/
Space before */
Will fix this space issue
+BLOCK_REPLICATION_DONE, /* block replication is done(after
failover) */
+};
Should this be an enum type in a .json file?
Since this is just an internal enum that only used in
block/replication.c, i think we don't need put it in *.json file.
+
+static int replication_open(BlockDriverState *bs, QDict *options,
+int flags, Error **errp)
+{
+
+fail:
+qemu_opts_del(opts);
+/* propagate error */
+if (local_err) {
+error_propagate(errp, local_err);
+}
It's safe to call error_propagate() unconditionally (you could drop the
'if (local_err)').
You're right, i'll fix all relevant issue in my patchset.
+static coroutine_fn int replication_co_discard(BlockDriverState *bs,
+ int64_t sector_num,
+ int nb_sectors)
+{
+BDRVReplicationState *s = bs->opaque;
+int ret;
+
+ret = replication_get_io_status(s);
+if (ret < 0) {
+return ret;
+}
+
+if (ret == 1) {
+/* It is secondary qemu and failover are running*/
Space before */
Will fix
+static void secondary_do_checkpoint(BDRVReplicationState *s, Error **errp)
+{
+Error *local_err = NULL;
+int ret;
+
+if (!s->secondary_disk->job) {
+error_setg(errp, "Backup job is cancelled unexpectedly");
Maybe s/is/was/
will fix
+static void replication_start(BlockReplicationState *brs, ReplicationMode mode,
+ Error **errp)
+{
+BlockDriverState *bs = brs->bs;
+BDRVReplicationState *s = brs->bs->opaque;
+int64_t active_length, hidden_length, disk_length;
+AioContext *aio_context;
+Error *local_err = NULL;
+
+if (s->replication_state != BLOCK_REPLICATION_NONE) {
+error_setg(errp, "Block replication is running or done");
+return;
+}
+
+if (s->mode != mode) {
+error_setg(errp, "The parameter mode's value is invalid, needs %d,"
+ " but receives %d", s->mode, mode);
s/receives/got/
will fix
+static void replication_do_checkpoint(BlockReplicationState *brs, Error **errp)
+{
+BDRVReplicationState *s = brs->bs->opaque;
+
+if (s->replication_state != BLOCK_REPLICATION_RUNNING) {
+error_setg(errp, "Block replication is not running");
+return;
+}
+
+if (s->error) {
+error_setg(errp, "I/O error occurs");
s/occurs/occurred/
Will fix
+++ b/qapi/block-core.json
@@ -1975,6 +1975,19 @@
'*read-pattern': 'QuorumReadPattern' } }
##
+# @ReplicationMode
+#
+# An enumeration of replication modes.
+#
+# @primary: Primary mode, the vm's state will be sent to secondary QEMU.
+#
+# @secondary: Secondary mode, receive the vm's state from primary QEMU.
+#
+# Since: 2.6
+##
+{ 'enum' : 'ReplicationMode', 'data' : [ 'primary', 'secondary' ] }
+
Interface looks okay.
Thanks
-Xie
Re: [Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
On 01/13/2016 02:18 AM, Changlong Xie wrote:
> From: Wen Congyang
>
> Signed-off-by: Wen Congyang
> Signed-off-by: zhanghailiang
> Signed-off-by: Gonglei
> Signed-off-by: Changlong Xie
> ---
> block/Makefile.objs | 1 +
> block/replication-comm.c | 66 +
> block/replication.c | 590
> +++
> include/block/replication-comm.h | 50
> qapi/block-core.json | 13 +
> 5 files changed, 720 insertions(+)
> create mode 100644 block/replication-comm.c
> create mode 100644 block/replication.c
> create mode 100644 include/block/replication-comm.h
>
Just a high-level overview, mainly on the user-visible interface and
things that caught my eye.
> +++ b/block/replication-comm.c
> @@ -0,0 +1,66 @@
> +/*
> + * Replication Block filter
> + *
> + * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
> + * Copyright (c) 2015 Intel Corporation
> + * Copyright (c) 2015 FUJITSU LIMITED
Do you want to claim 2016 in any of this?
> +
> +enum {
> +BLOCK_REPLICATION_NONE, /* block replication is not started
> */
> +BLOCK_REPLICATION_RUNNING, /* block replication is running */
> +BLOCK_REPLICATION_FAILOVER, /* failover is running in background
> */
> +BLOCK_REPLICATION_FAILOVER_FAILED, /* failover failed*/
Space before */
> +BLOCK_REPLICATION_DONE, /* block replication is done(after
> failover) */
> +};
Should this be an enum type in a .json file?
> +
> +static int replication_open(BlockDriverState *bs, QDict *options,
> +int flags, Error **errp)
> +{
> +
> +fail:
> +qemu_opts_del(opts);
> +/* propagate error */
> +if (local_err) {
> +error_propagate(errp, local_err);
> +}
It's safe to call error_propagate() unconditionally (you could drop the
'if (local_err)').
> +static coroutine_fn int replication_co_discard(BlockDriverState *bs,
> + int64_t sector_num,
> + int nb_sectors)
> +{
> +BDRVReplicationState *s = bs->opaque;
> +int ret;
> +
> +ret = replication_get_io_status(s);
> +if (ret < 0) {
> +return ret;
> +}
> +
> +if (ret == 1) {
> +/* It is secondary qemu and failover are running*/
Space before */
> +static void secondary_do_checkpoint(BDRVReplicationState *s, Error **errp)
> +{
> +Error *local_err = NULL;
> +int ret;
> +
> +if (!s->secondary_disk->job) {
> +error_setg(errp, "Backup job is cancelled unexpectedly");
Maybe s/is/was/
> +static void replication_start(BlockReplicationState *brs, ReplicationMode
> mode,
> + Error **errp)
> +{
> +BlockDriverState *bs = brs->bs;
> +BDRVReplicationState *s = brs->bs->opaque;
> +int64_t active_length, hidden_length, disk_length;
> +AioContext *aio_context;
> +Error *local_err = NULL;
> +
> +if (s->replication_state != BLOCK_REPLICATION_NONE) {
> +error_setg(errp, "Block replication is running or done");
> +return;
> +}
> +
> +if (s->mode != mode) {
> +error_setg(errp, "The parameter mode's value is invalid, needs %d,"
> + " but receives %d", s->mode, mode);
s/receives/got/
> +static void replication_do_checkpoint(BlockReplicationState *brs, Error
> **errp)
> +{
> +BDRVReplicationState *s = brs->bs->opaque;
> +
> +if (s->replication_state != BLOCK_REPLICATION_RUNNING) {
> +error_setg(errp, "Block replication is not running");
> +return;
> +}
> +
> +if (s->error) {
> +error_setg(errp, "I/O error occurs");
s/occurs/occurred/
> +++ b/qapi/block-core.json
> @@ -1975,6 +1975,19 @@
> '*read-pattern': 'QuorumReadPattern' } }
>
> ##
> +# @ReplicationMode
> +#
> +# An enumeration of replication modes.
> +#
> +# @primary: Primary mode, the vm's state will be sent to secondary QEMU.
> +#
> +# @secondary: Secondary mode, receive the vm's state from primary QEMU.
> +#
> +# Since: 2.6
> +##
> +{ 'enum' : 'ReplicationMode', 'data' : [ 'primary', 'secondary' ] }
> +
Interface looks okay.
--
Eric Blake eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org
signature.asc
Description: OpenPGP digital signature
[Qemu-devel] [PATCH v14 7/8] Implement new driver for block replication
From: Wen Congyang
Signed-off-by: Wen Congyang
Signed-off-by: zhanghailiang
Signed-off-by: Gonglei
Signed-off-by: Changlong Xie
---
block/Makefile.objs | 1 +
block/replication-comm.c | 66 +
block/replication.c | 590 +++
include/block/replication-comm.h | 50
qapi/block-core.json | 13 +
5 files changed, 720 insertions(+)
create mode 100644 block/replication-comm.c
create mode 100644 block/replication.c
create mode 100644 include/block/replication-comm.h
diff --git a/block/Makefile.objs b/block/Makefile.objs
index fa05f37..7037662 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -23,6 +23,7 @@ block-obj-$(CONFIG_LIBSSH2) += ssh.o
block-obj-y += accounting.o
block-obj-y += write-threshold.o
block-obj-y += backup.o
+block-obj-y += replication-comm.o replication.o
common-obj-y += stream.o
common-obj-y += commit.o
diff --git a/block/replication-comm.c b/block/replication-comm.c
new file mode 100644
index 000..8af748b
--- /dev/null
+++ b/block/replication-comm.c
@@ -0,0 +1,66 @@
+/*
+ * Replication Block filter
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 Intel Corporation
+ * Copyright (c) 2015 FUJITSU LIMITED
+ *
+ * Author:
+ * Wen Congyang
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "block/replication-comm.h"
+
+static QLIST_HEAD(, BlockReplicationState) block_replication_states;
+
+BlockReplicationState *block_replication_new(BlockDriverState *bs,
+ BlockReplicationOps *ops)
+{
+BlockReplicationState *brs;
+
+brs = g_new0(BlockReplicationState, 1);
+brs->bs = bs;
+brs->ops = ops;
+QLIST_INSERT_HEAD(&block_replication_states, brs, node);
+
+return brs;
+}
+
+void block_replication_remove(BlockReplicationState *brs)
+{
+QLIST_REMOVE(brs, node);
+g_free(brs);
+}
+
+void block_replication_start_all(ReplicationMode mode, Error **errp)
+{
+BlockReplicationState *brs, *next;
+QLIST_FOREACH_SAFE(brs, &block_replication_states, node, next) {
+if (brs->ops && brs->ops->start) {
+brs->ops->start(brs, mode, errp);
+}
+}
+}
+
+void block_replication_do_checkpoint_all(Error **errp)
+{
+BlockReplicationState *brs, *next;
+QLIST_FOREACH_SAFE(brs, &block_replication_states, node, next) {
+if (brs->ops && brs->ops->checkpoint) {
+brs->ops->checkpoint(brs, errp);
+}
+}
+}
+
+void block_replication_stop_all(bool failover, Error **errp)
+{
+BlockReplicationState *brs, *next;
+QLIST_FOREACH_SAFE(brs, &block_replication_states, node, next) {
+if (brs->ops && brs->ops->stop) {
+brs->ops->stop(brs, failover, errp);
+}
+}
+}
diff --git a/block/replication.c b/block/replication.c
new file mode 100644
index 000..29c677a
--- /dev/null
+++ b/block/replication.c
@@ -0,0 +1,590 @@
+/*
+ * Replication Block filter
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 Intel Corporation
+ * Copyright (c) 2015 FUJITSU LIMITED
+ *
+ * Author:
+ * Wen Congyang
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu-common.h"
+#include "block/blockjob.h"
+#include "block/nbd.h"
+#include "block/replication-comm.h"
+
+typedef struct BDRVReplicationState {
+ReplicationMode mode;
+int replication_state;
+BlockDriverState *active_disk;
+BlockDriverState *hidden_disk;
+BlockDriverState *secondary_disk;
+BlockDriverState *top_bs;
+BlockReplicationState *brs;
+Error *blocker;
+int orig_hidden_flags;
+int orig_secondary_flags;
+int error;
+} BDRVReplicationState;
+
+enum {
+BLOCK_REPLICATION_NONE, /* block replication is not started */
+BLOCK_REPLICATION_RUNNING, /* block replication is running */
+BLOCK_REPLICATION_FAILOVER, /* failover is running in background */
+BLOCK_REPLICATION_FAILOVER_FAILED, /* failover failed*/
+BLOCK_REPLICATION_DONE, /* block replication is done(after
failover) */
+};
+
+static void replication_start(BlockReplicationState *brs, ReplicationMode mode,
+ Error **errp);
+static void replication_do_checkpoint(BlockReplicationState *brs, Error
**errp);
+static void replication_stop(BlockReplicationState *brs, bool failover,
+ Error **errp);
+
+#define REPLICATION_MODE"mode"
+static QemuOptsList replication_runtime_opts = {
+.name = "replication",
+.head = QTAILQ_HEAD_INITIALIZER(replication_runtime_opts.head),
+.desc = {
+{
+.name = REPLICATION_MODE,
+.type = QEMU_OPT_STRING,
+},
+{ /* end of list */ }
