[RFC PATCH 01/17] configure: add CONFIG_COLO to switch COLO support

2014-07-23 Thread Yang Hongyang 
./configure --enable-colo/--disable-colo to switch COLO
support on/off.
COLO support is off by default.

Signed-off-by: Yang Hongyang 
---
 configure | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/configure b/configure
index f7685b5..4071943 100755
--- a/configure
+++ b/configure
@@ -258,6 +258,7 @@ xfs=""
 vhost_net="no"
 vhost_scsi="no"
 kvm="no"
+colo="no"
 rdma=""
 gprof="no"
 debug_tcg="no"
@@ -921,6 +922,10 @@ for opt do
   ;;
   --enable-kvm) kvm="yes"
   ;;
+  --disable-colo) colo="no"
+  ;;
+  --enable-colo) colo="yes"
+  ;;
   --disable-tcg-interpreter) tcg_interpreter="no"
   ;;
   --enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1314,6 +1319,10 @@ Advanced options (experts only):
   --disable-slirp  disable SLIRP userspace network connectivity
   --disable-kvmdisable KVM acceleration support
   --enable-kvm enable KVM acceleration support
+  --disable-colo   disable COarse-grain LOck-stepping Virtual
+   Machines for Non-stop Service(default)
+  --enable-coloenable COarse-grain LOck-stepping Virtual
+   Machines for Non-stop Service
   --disable-rdma   disable RDMA-based migration support
   --enable-rdmaenable RDMA-based migration support
   --enable-tcg-interpreter enable TCG with bytecode interpreter (TCI)
@@ -4215,6 +4224,7 @@ echo "Linux AIO support $linux_aio"
 echo "ATTR/XATTR support $attr"
 echo "Install blobs $blobs"
 echo "KVM support   $kvm"
+echo "COLO support  $colo"
 echo "RDMA support  $rdma"
 echo "TCG interpreter   $tcg_interpreter"
 echo "fdt support   $fdt"
@@ -4751,6 +4761,10 @@ if have_backend "ftrace"; then
 fi
 echo "CONFIG_TRACE_FILE=$trace_file" >> $config_host_mak
 
+if test "$colo" = "yes"; then
+  echo "CONFIG_COLO=y" >> $config_host_mak
+fi
+
 if test "$rdma" = "yes" ; then
   echo "CONFIG_RDMA=y" >> $config_host_mak
 fi
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 00/17] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service

2014-07-23 Thread Yang Hongyang 
Virtual machine (VM) replication is a well known technique for
providing application-agnostic software-implemented hardware fault
tolerance "non-stop service". COLO is a high availability solution.
Both primary VM (PVM) and secondary VM (SVM) run in parallel. They
receive the same request from client, and generate response in parallel
too. If the response packets from PVM and SVM are identical, they are
released immediately. Otherwise, a VM checkpoint (on demand) is
conducted. The idea is presented in Xen summit 2012, and 2013,
and academia paper in SOCC 2013. It's also presented in KVM forum
2013:
http://www.linux-kvm.org/wiki/images/1/1d/Kvm-forum-2013-COLO.pdf
Please refer to above document for detailed information. 
Please also refer to previous posted RFC proposal:
http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg05567.html

The patchset is also hosted on github:
https://github.com/macrosheep/qemu/tree/colo_v0.1

This patchset is RFC, implements the frame of colo, without
failover and nic/disk replication. But it is ready for demo
the COLO idea above QEMU-Kvm.
Steps using this patchset to get an overview of COLO:
1. configure the source with --enable-colo option
2. compile
3. just like QEMU's normal migration, run 2 QEMU VM:
   - Primary VM 
   - Secondary VM with -incoming tcp:[IP]:[PORT] option
4. on Primary VM's QEMU monitor, run following command:
   migrate_set_capability colo on
   migrate tcp:[IP]:[PORT]
5. done
you will see two runing VMs, whenever you make changes to PVM, SVM
will be synced to PVM's state.

TODO list:
1. failover
2. nic replication
3. disk replication[COLO Disk manager]

Any comments/feedbacks are warmly welcomed.

Thanks,
Yang

Yang Hongyang (17):
  configure: add CONFIG_COLO to switch COLO support
  COLO: introduce an api colo_supported() to indicate COLO support
  COLO migration: add a migration capability 'colo'
  COLO info: use colo info to tell migration target colo is enabled
  COLO save: integrate COLO checkpointed save into qemu migration
  COLO restore: integrate COLO checkpointed restore into qemu restore
  COLO buffer: implement colo buffer as well as QEMUFileOps based on it
  COLO: disable qdev hotplug
  COLO ctl: implement API's that communicate with colo agent
  COLO ctl: introduce is_slave() and is_master()
  COLO ctl: implement colo checkpoint protocol
  COLO ctl: add a RunState RUN_STATE_COLO
  COLO ctl: implement colo save
  COLO ctl: implement colo restore
  COLO save: reuse migration bitmap under colo checkpoint
  COLO ram cache: implement colo ram cache on slaver
  HACK: trigger checkpoint every 500ms

 Makefile.objs  |   2 +
 arch_init.c| 174 +-
 configure  |  14 +
 include/exec/cpu-all.h |   1 +
 include/migration/migration-colo.h |  36 +++
 include/migration/migration.h  |  13 +
 include/qapi/qmp/qerror.h  |   3 +
 migration-colo-comm.c  |  78 +
 migration-colo.c   | 643 +
 migration.c|  45 ++-
 qapi-schema.json   |   9 +-
 stubs/Makefile.objs|   1 +
 stubs/migration-colo.c |  34 ++
 vl.c   |  12 +
 14 files changed, 1044 insertions(+), 21 deletions(-)
 create mode 100644 include/migration/migration-colo.h
 create mode 100644 migration-colo-comm.c
 create mode 100644 migration-colo.c
 create mode 100644 stubs/migration-colo.c

-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 05/17] COLO save: integrate COLO checkpointed save into qemu migration

2014-07-23 Thread Yang Hongyang 
  Integrate COLO checkpointed save flow into qemu migration.
  Add a migrate state: MIG_STATE_COLO, enter this migrate state
after the first live migration successfully finished.
  Create a colo thread to do the checkpointed save.

Signed-off-by: Yang Hongyang 
---
 include/migration/migration-colo.h |  4 
 include/migration/migration.h  | 13 +++
 migration-colo-comm.c  |  2 +-
 migration-colo.c   | 48 ++
 migration.c| 36 
 stubs/migration-colo.c |  4 
 6 files changed, 91 insertions(+), 16 deletions(-)

diff --git a/include/migration/migration-colo.h 
b/include/migration/migration-colo.h
index e3735d8..24589c0 100644
--- a/include/migration/migration-colo.h
+++ b/include/migration/migration-colo.h
@@ -18,4 +18,8 @@ void colo_info_mig_init(void);
 
 bool colo_supported(void);
 
+/* save */
+bool migrate_use_colo(void);
+void colo_init_checkpointer(MigrationState *s);
+
 #endif
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 3cb5ba8..3e81a27 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -64,6 +64,19 @@ struct MigrationState
 int64_t dirty_sync_count;
 };
 
+enum {
+MIG_STATE_ERROR = -1,
+MIG_STATE_NONE,
+MIG_STATE_SETUP,
+MIG_STATE_CANCELLING,
+MIG_STATE_CANCELLED,
+MIG_STATE_ACTIVE,
+MIG_STATE_COLO,
+MIG_STATE_COMPLETED,
+};
+
+void migrate_set_state(MigrationState *s, int old_state, int new_state);
+
 void process_incoming_migration(QEMUFile *f);
 
 void qemu_start_incoming_migration(const char *uri, Error **errp);
diff --git a/migration-colo-comm.c b/migration-colo-comm.c
index ccbc246..4504ceb 100644
--- a/migration-colo-comm.c
+++ b/migration-colo-comm.c
@@ -25,7 +25,7 @@ static bool colo_requested;
 
 /* save */
 
-static bool migrate_use_colo(void)
+bool migrate_use_colo(void)
 {
 MigrationState *s = migrate_get_current();
 return s->enabled_capabilities[MIGRATION_CAPABILITY_COLO];
diff --git a/migration-colo.c b/migration-colo.c
index 1d3bef8..0cef8bd 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -8,9 +8,57 @@
  * the COPYING file in the top-level directory.
  */
 
+#include "qemu/main-loop.h"
+#include "qemu/thread.h"
 #include "migration/migration-colo.h"
 
+static QEMUBH *colo_bh;
+
 bool colo_supported(void)
 {
 return true;
 }
+
+/* save */
+
+static void *colo_thread(void *opaque)
+{
+MigrationState *s = opaque;
+
+/*TODO: COLO checkpointed save loop*/
+
+if (s->state != MIG_STATE_ERROR) {
+migrate_set_state(s, MIG_STATE_COLO, MIG_STATE_COMPLETED);
+}
+
+qemu_mutex_lock_iothread();
+qemu_bh_schedule(s->cleanup_bh);
+qemu_mutex_unlock_iothread();
+
+return NULL;
+}
+
+static void colo_start_checkpointer(void *opaque)
+{
+MigrationState *s = opaque;
+
+if (colo_bh) {
+qemu_bh_delete(colo_bh);
+colo_bh = NULL;
+}
+
+qemu_mutex_unlock_iothread();
+qemu_thread_join(&s->thread);
+qemu_mutex_lock_iothread();
+
+migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_COLO);
+
+qemu_thread_create(&s->thread, "colo", colo_thread, s,
+   QEMU_THREAD_JOINABLE);
+}
+
+void colo_init_checkpointer(MigrationState *s)
+{
+colo_bh = qemu_bh_new(colo_start_checkpointer, s);
+qemu_bh_schedule(colo_bh);
+}
diff --git a/migration.c b/migration.c
index ca83310..b7f8e7e 100644
--- a/migration.c
+++ b/migration.c
@@ -27,16 +27,6 @@
 #include "trace.h"
 #include "migration/migration-colo.h"
 
-enum {
-MIG_STATE_ERROR = -1,
-MIG_STATE_NONE,
-MIG_STATE_SETUP,
-MIG_STATE_CANCELLING,
-MIG_STATE_CANCELLED,
-MIG_STATE_ACTIVE,
-MIG_STATE_COMPLETED,
-};
-
 #define MAX_THROTTLE  (32 << 20)  /* Migration speed throttling */
 
 /* Amount of time to allocate to each "chunk" of bandwidth-throttled
@@ -229,6 +219,11 @@ MigrationInfo *qmp_query_migrate(Error **errp)
 
 get_xbzrle_cache_stats(info);
 break;
+case MIG_STATE_COLO:
+info->has_status = true;
+info->status = g_strdup("colo");
+/* TODO: display COLO specific informations(checkpoint info etc.),*/
+break;
 case MIG_STATE_COMPLETED:
 get_xbzrle_cache_stats(info);
 
@@ -272,7 +267,8 @@ void 
qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
 MigrationState *s = migrate_get_current();
 MigrationCapabilityStatusList *cap;
 
-if (s->state == MIG_STATE_ACTIVE || s->state == MIG_STATE_SETUP) {
+if (s->state == MIG_STATE_ACTIVE || s->state == MIG_STATE_SETUP ||
+s->state == MIG_STATE_COLO) {
 error_set(errp, QERR_MIGRATION_ACTIVE);
 return;
 }
@@ -289,7 +285,7 @@ void 
qmp_migrate_set_capabilities(MigrationCapabilityStat

[RFC PATCH 09/17] COLO ctl: implement API's that communicate with colo agent

2014-07-23 Thread Yang Hongyang 
We use COLO agent to compare the packets returned by
Primary VM and Secondary VM, and decide whether to start a
checkpoint according to some rules. It is a linux kernel
module for host.
COLO controller communicate with the agent through ioctl().

Signed-off-by: Yang Hongyang 
---
 migration-colo.c | 115 +--
 1 file changed, 112 insertions(+), 3 deletions(-)

diff --git a/migration-colo.c b/migration-colo.c
index f295e56..802f8b0 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -13,7 +13,16 @@
 #include "block/coroutine.h"
 #include "qemu/error-report.h"
 #include "hw/qdev-core.h"
+#include "qemu/timer.h"
 #include "migration/migration-colo.h"
+#include 
+
+/*
+ * checkpoint timer: unit ms
+ * this is large because COLO checkpoint will mostly depend on
+ * COLO compare module.
+ */
+#define CHKPOINT_TIMER 1
 
 static QEMUBH *colo_bh;
 
@@ -22,6 +31,56 @@ bool colo_supported(void)
 return true;
 }
 
+/* colo compare */
+#define COMP_IOC_MAGIC 'k'
+#define COMP_IOCTWAIT   _IO(COMP_IOC_MAGIC, 0)
+#define COMP_IOCTFLUSH  _IO(COMP_IOC_MAGIC, 1)
+#define COMP_IOCTRESUME _IO(COMP_IOC_MAGIC, 2)
+
+#define COMPARE_DEV "/dev/HA_compare"
+/* COLO compare module FD */
+static int comp_fd = -1;
+
+static int colo_compare_init(void)
+{
+comp_fd = open(COMPARE_DEV, O_RDONLY);
+if (comp_fd < 0) {
+return -1;
+}
+
+return 0;
+}
+
+static void colo_compare_destroy(void)
+{
+if (comp_fd >= 0) {
+close(comp_fd);
+comp_fd = -1;
+}
+}
+
+/*
+ * Communicate with COLO Agent through ioctl.
+ * return:
+ * 0: start a checkpoint
+ * other: errno == ETIME or ERESTART, try again
+ *errno == other, error, quit colo save
+ */
+static int colo_compare(void)
+{
+return ioctl(comp_fd, COMP_IOCTWAIT, 250);
+}
+
+static __attribute__((unused)) int colo_compare_flush(void)
+{
+return ioctl(comp_fd, COMP_IOCTFLUSH, 1);
+}
+
+static __attribute__((unused)) int colo_compare_resume(void)
+{
+return ioctl(comp_fd, COMP_IOCTRESUME, 1);
+}
+
 /* colo buffer */
 
 #define COLO_BUFFER_BASE_SIZE (1000*1000*4ULL)
@@ -131,15 +190,48 @@ static const QEMUFileOps colo_read_ops = {
 static void *colo_thread(void *opaque)
 {
 MigrationState *s = opaque;
-int dev_hotplug = qdev_hotplug;
+int dev_hotplug = qdev_hotplug, wait_cp = 0;
+int64_t start_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+int64_t current_time;
+
+if (colo_compare_init() < 0) {
+error_report("Init colo compare error\n");
+goto out;
+}
 
 qdev_hotplug = 0;
 
 colo_buffer_init();
 
-/*TODO: COLO checkpointed save loop*/
+while (s->state == MIG_STATE_COLO) {
+/* wait for a colo checkpoint */
+wait_cp = colo_compare();
+if (wait_cp) {
+if (errno != ETIME && errno != ERESTART) {
+error_report("compare module failed(%s)", strerror(errno));
+goto out;
+}
+/*
+ * no checkpoint is needed, wait for 1ms and then
+ * check if we need checkpoint
+ */
+current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+if (current_time - start_time < CHKPOINT_TIMER) {
+usleep(1000);
+continue;
+}
+}
+
+/* start a colo checkpoint */
+
+/*TODO: COLO save */
 
+start_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+}
+
+out:
 colo_buffer_destroy();
+colo_compare_destroy();
 
 if (s->state != MIG_STATE_ERROR) {
 migrate_set_state(s, MIG_STATE_COLO, MIG_STATE_COMPLETED);
@@ -183,6 +275,17 @@ void colo_init_checkpointer(MigrationState *s)
 
 static Coroutine *colo;
 
+/*
+ * return:
+ * 0: start a checkpoint
+ * 1: some error happend, exit colo restore
+ */
+static int slave_wait_new_checkpoint(QEMUFile *f)
+{
+/* TODO: wait checkpoint start command from master */
+return 1;
+}
+
 void colo_process_incoming_checkpoints(QEMUFile *f)
 {
 int dev_hotplug = qdev_hotplug;
@@ -198,7 +301,13 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
 
 colo_buffer_init();
 
-/* TODO: COLO checkpointed restore loop */
+while (true) {
+if (slave_wait_new_checkpoint(f)) {
+break;
+}
+
+/* TODO: COLO restore */
+}
 
 colo_buffer_destroy();
 colo = NULL;
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 02/17] COLO: introduce an api colo_supported() to indicate COLO support

2014-07-23 Thread Yang Hongyang 
introduce an api colo_supported() to indicate COLO support, returns
true if colo supported(configured with --enable-colo).

Signed-off-by: Yang Hongyang 
---
 Makefile.objs  |  1 +
 include/migration/migration-colo.h | 18 ++
 migration-colo.c   | 16 
 stubs/Makefile.objs|  1 +
 stubs/migration-colo.c | 16 
 5 files changed, 52 insertions(+)
 create mode 100644 include/migration/migration-colo.h
 create mode 100644 migration-colo.c
 create mode 100644 stubs/migration-colo.c

diff --git a/Makefile.objs b/Makefile.objs
index 1f76cea..cab5824 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -50,6 +50,7 @@ common-obj-$(CONFIG_POSIX) += os-posix.o
 common-obj-$(CONFIG_LINUX) += fsdev/
 
 common-obj-y += migration.o migration-tcp.o
+common-obj-$(CONFIG_COLO) += migration-colo.o
 common-obj-y += vmstate.o
 common-obj-y += qemu-file.o
 common-obj-$(CONFIG_RDMA) += migration-rdma.o
diff --git a/include/migration/migration-colo.h 
b/include/migration/migration-colo.h
new file mode 100644
index 000..35b384c
--- /dev/null
+++ b/include/migration/migration-colo.h
@@ -0,0 +1,18 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_MIGRATION_COLO_H
+#define QEMU_MIGRATION_COLO_H
+
+#include "qemu-common.h"
+
+bool colo_supported(void);
+
+#endif
diff --git a/migration-colo.c b/migration-colo.c
new file mode 100644
index 000..1d3bef8
--- /dev/null
+++ b/migration-colo.c
@@ -0,0 +1,16 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ */
+
+#include "migration/migration-colo.h"
+
+bool colo_supported(void)
+{
+return true;
+}
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 528e161..6810c89 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -39,3 +39,4 @@ stub-obj-$(CONFIG_WIN32) += fd-register.o
 stub-obj-y += cpus.o
 stub-obj-y += kvm.o
 stub-obj-y += qmp_pc_dimm_device_list.o
+stub-obj-y += migration-colo.o
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
new file mode 100644
index 000..b9ee6a0
--- /dev/null
+++ b/stubs/migration-colo.c
@@ -0,0 +1,16 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ */
+
+#include "migration/migration-colo.h"
+
+bool colo_supported(void)
+{
+return false;
+}
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 07/17] COLO buffer: implement colo buffer as well as QEMUFileOps based on it

2014-07-23 Thread Yang Hongyang 
We need a buffer to store migration data.

On save side:
  all saved data was write into colo buffer first, so that we can know
the total size of the migration data. this can also separate the data
transmission from colo control data, we use colo control data over
socket fd to synchronous both side's stat.

On restore side:
  all migration data was read into colo buffer first, then load data
from the buffer: If network error happens while data transmission,
the slaver can still functinal because the migration data are not yet
loaded.

Signed-off-by: Yang Hongyang 
---
 migration-colo.c | 112 +++
 1 file changed, 112 insertions(+)

diff --git a/migration-colo.c b/migration-colo.c
index d566b9d..b90d9b6 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -11,6 +11,7 @@
 #include "qemu/main-loop.h"
 #include "qemu/thread.h"
 #include "block/coroutine.h"
+#include "qemu/error-report.h"
 #include "migration/migration-colo.h"
 
 static QEMUBH *colo_bh;
@@ -20,14 +21,122 @@ bool colo_supported(void)
 return true;
 }
 
+/* colo buffer */
+
+#define COLO_BUFFER_BASE_SIZE (1000*1000*4ULL)
+#define COLO_BUFFER_MAX_SIZE (1000*1000*1000*10ULL)
+
+typedef struct colo_buffer {
+uint8_t *data;
+uint64_t used;
+uint64_t freed;
+uint64_t size;
+} colo_buffer_t;
+
+static colo_buffer_t colo_buffer;
+
+static void colo_buffer_init(void)
+{
+if (colo_buffer.size == 0) {
+colo_buffer.data = g_malloc(COLO_BUFFER_BASE_SIZE);
+colo_buffer.size = COLO_BUFFER_BASE_SIZE;
+}
+colo_buffer.used = 0;
+colo_buffer.freed = 0;
+}
+
+static void colo_buffer_destroy(void)
+{
+if (colo_buffer.data) {
+g_free(colo_buffer.data);
+colo_buffer.data = NULL;
+}
+colo_buffer.used = 0;
+colo_buffer.freed = 0;
+colo_buffer.size = 0;
+}
+
+static void colo_buffer_extend(uint64_t len)
+{
+if (len > colo_buffer.size - colo_buffer.used) {
+len = len + colo_buffer.used - colo_buffer.size;
+len = ROUND_UP(len, COLO_BUFFER_BASE_SIZE) + COLO_BUFFER_BASE_SIZE;
+
+colo_buffer.size += len;
+if (colo_buffer.size > COLO_BUFFER_MAX_SIZE) {
+error_report("colo_buffer overflow!\n");
+exit(EXIT_FAILURE);
+}
+colo_buffer.data = g_realloc(colo_buffer.data, colo_buffer.size);
+}
+}
+
+static int colo_put_buffer(void *opaque, const uint8_t *buf,
+   int64_t pos, int size)
+{
+colo_buffer_extend(size);
+memcpy(colo_buffer.data + colo_buffer.used, buf, size);
+colo_buffer.used += size;
+
+return size;
+}
+
+static int colo_get_buffer_internal(uint8_t *buf, int size)
+{
+if ((size + colo_buffer.freed) > colo_buffer.used) {
+size = colo_buffer.used - colo_buffer.freed;
+}
+memcpy(buf, colo_buffer.data + colo_buffer.freed, size);
+colo_buffer.freed += size;
+
+return size;
+}
+
+static int colo_get_buffer(void *opaque, uint8_t *buf, int64_t pos, int size)
+{
+return colo_get_buffer_internal(buf, size);
+}
+
+static int colo_close(void *opaque)
+{
+colo_buffer_t *cb = opaque ;
+
+cb->used = 0;
+cb->freed = 0;
+
+return 0;
+}
+
+static int colo_get_fd(void *opaque)
+{
+/* colo buffer, no fd */
+return -1;
+}
+
+static const QEMUFileOps colo_write_ops = {
+.put_buffer = colo_put_buffer,
+.get_fd = colo_get_fd,
+.close = colo_close,
+};
+
+static const QEMUFileOps colo_read_ops = {
+.get_buffer = colo_get_buffer,
+.get_fd = colo_get_fd,
+.close = colo_close,
+};
+
 /* save */
 
 static void *colo_thread(void *opaque)
 {
 MigrationState *s = opaque;
 
+colo_buffer_init();
+
 /*TODO: COLO checkpointed save loop*/
 
+colo_buffer_destroy();
+
 if (s->state != MIG_STATE_ERROR) {
 migrate_set_state(s, MIG_STATE_COLO, MIG_STATE_COMPLETED);
 }
@@ -77,8 +186,11 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
 colo = qemu_coroutine_self();
 assert(colo != NULL);
 
+colo_buffer_init();
+
 /* TODO: COLO checkpointed restore loop */
 
+colo_buffer_destroy();
 colo = NULL;
 restore_exit_colo();
 
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 10/17] COLO ctl: introduce is_slave() and is_master()

2014-07-23 Thread Yang Hongyang 
is_slaver is to determine whether the QEMU instance is a
slaver(migration target) at runtime.
is_master is to determine whether the QEMU instance is a
master(migration starter) at runtime.
This 2 APIs will be used later.

Signed-off-by: Yang Hongyang 
---
 migration-colo.c | 11 +++
 1 file changed, 11 insertions(+)

diff --git a/migration-colo.c b/migration-colo.c
index 802f8b0..2699e77 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -187,6 +187,12 @@ static const QEMUFileOps colo_read_ops = {
 
 /* save */
 
+static __attribute__((unused)) bool is_master(void)
+{
+MigrationState *s = migrate_get_current();
+return (s->state == MIG_STATE_COLO);
+}
+
 static void *colo_thread(void *opaque)
 {
 MigrationState *s = opaque;
@@ -275,6 +281,11 @@ void colo_init_checkpointer(MigrationState *s)
 
 static Coroutine *colo;
 
+static __attribute__((unused)) bool is_slave(void)
+{
+return colo != NULL;
+}
+
 /*
  * return:
  * 0: start a checkpoint
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 12/17] COLO ctl: add a RunState RUN_STATE_COLO

2014-07-23 Thread Yang Hongyang 
Guest will enter this state when paused to save/resore VM state
under colo checkpoint.

Signed-off-by: Yang Hongyang 
---
 qapi-schema.json | 4 +++-
 vl.c | 8 
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/qapi-schema.json b/qapi-schema.json
index 807f5a2..b42171c 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -145,12 +145,14 @@
 # @watchdog: the watchdog action is configured to pause and has been triggered
 #
 # @guest-panicked: guest has been panicked as a result of guest OS panic
+#
+# @colo: guest is paused to save/restore VM state under colo checkpoint
 ##
 { 'enum': 'RunState',
   'data': [ 'debug', 'inmigrate', 'internal-error', 'io-error', 'paused',
 'postmigrate', 'prelaunch', 'finish-migrate', 'restore-vm',
 'running', 'save-vm', 'shutdown', 'suspended', 'watchdog',
-'guest-panicked' ] }
+'guest-panicked', 'colo' ] }
 
 ##
 # @StatusInfo:
diff --git a/vl.c b/vl.c
index 1a282d8..545155d 100644
--- a/vl.c
+++ b/vl.c
@@ -597,6 +597,7 @@ static const RunStateTransition runstate_transitions_def[] 
= {
 
 { RUN_STATE_INMIGRATE, RUN_STATE_RUNNING },
 { RUN_STATE_INMIGRATE, RUN_STATE_PAUSED },
+{ RUN_STATE_INMIGRATE, RUN_STATE_COLO },
 
 { RUN_STATE_INTERNAL_ERROR, RUN_STATE_PAUSED },
 { RUN_STATE_INTERNAL_ERROR, RUN_STATE_FINISH_MIGRATE },
@@ -606,6 +607,7 @@ static const RunStateTransition runstate_transitions_def[] 
= {
 
 { RUN_STATE_PAUSED, RUN_STATE_RUNNING },
 { RUN_STATE_PAUSED, RUN_STATE_FINISH_MIGRATE },
+{ RUN_STATE_PAUSED, RUN_STATE_COLO},
 
 { RUN_STATE_POSTMIGRATE, RUN_STATE_RUNNING },
 { RUN_STATE_POSTMIGRATE, RUN_STATE_FINISH_MIGRATE },
@@ -616,9 +618,12 @@ static const RunStateTransition runstate_transitions_def[] 
= {
 
 { RUN_STATE_FINISH_MIGRATE, RUN_STATE_RUNNING },
 { RUN_STATE_FINISH_MIGRATE, RUN_STATE_POSTMIGRATE },
+{ RUN_STATE_FINISH_MIGRATE, RUN_STATE_COLO},
 
 { RUN_STATE_RESTORE_VM, RUN_STATE_RUNNING },
 
+{ RUN_STATE_COLO, RUN_STATE_RUNNING },
+
 { RUN_STATE_RUNNING, RUN_STATE_DEBUG },
 { RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR },
 { RUN_STATE_RUNNING, RUN_STATE_IO_ERROR },
@@ -629,6 +634,7 @@ static const RunStateTransition runstate_transitions_def[] 
= {
 { RUN_STATE_RUNNING, RUN_STATE_SHUTDOWN },
 { RUN_STATE_RUNNING, RUN_STATE_WATCHDOG },
 { RUN_STATE_RUNNING, RUN_STATE_GUEST_PANICKED },
+{ RUN_STATE_RUNNING, RUN_STATE_COLO},
 
 { RUN_STATE_SAVE_VM, RUN_STATE_RUNNING },
 
@@ -639,9 +645,11 @@ static const RunStateTransition runstate_transitions_def[] 
= {
 { RUN_STATE_RUNNING, RUN_STATE_SUSPENDED },
 { RUN_STATE_SUSPENDED, RUN_STATE_RUNNING },
 { RUN_STATE_SUSPENDED, RUN_STATE_FINISH_MIGRATE },
+{ RUN_STATE_SUSPENDED, RUN_STATE_COLO},
 
 { RUN_STATE_WATCHDOG, RUN_STATE_RUNNING },
 { RUN_STATE_WATCHDOG, RUN_STATE_FINISH_MIGRATE },
+{ RUN_STATE_WATCHDOG, RUN_STATE_COLO},
 
 { RUN_STATE_GUEST_PANICKED, RUN_STATE_RUNNING },
 { RUN_STATE_GUEST_PANICKED, RUN_STATE_FINISH_MIGRATE },
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 16/17] COLO ram cache: implement colo ram cache on slaver

2014-07-23 Thread Yang Hongyang 
The ram cache was initially the same as PVM's memory. At
checkpoint, we cache the dirty memory of PVM into ram cache
(so that ram cache always the same as PVM's memory at every
checkpoint), flush cached memory to SVM after we received
all PVM dirty memory(only needed to flush memory that was
both dirty on PVM and SVM since last checkpoint).

Signed-off-by: Yang Hongyang 
---
 arch_init.c| 154 -
 include/exec/cpu-all.h |   1 +
 include/migration/migration-colo.h |   3 +
 migration-colo.c   |   4 +
 4 files changed, 159 insertions(+), 3 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index c84e6c8..009bcb5 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -1013,6 +1013,7 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void 
*host)
 return 0;
 }
 
+static void *memory_region_get_ram_cache_ptr(MemoryRegion *mr, RAMBlock 
*block);
 static inline void *host_from_stream_offset(QEMUFile *f,
 ram_addr_t offset,
 int flags)
@@ -1027,7 +1028,12 @@ static inline void *host_from_stream_offset(QEMUFile *f,
 return NULL;
 }
 
-return memory_region_get_ram_ptr(block->mr) + offset;
+if (is_slave()) {
+migration_bitmap_set_dirty(block->mr->ram_addr + offset);
+return memory_region_get_ram_cache_ptr(block->mr, block) + offset;
+} else {
+return memory_region_get_ram_ptr(block->mr) + offset;
+}
 }
 
 len = qemu_get_byte(f);
@@ -1035,8 +1041,15 @@ static inline void *host_from_stream_offset(QEMUFile *f,
 id[len] = 0;
 
 QTAILQ_FOREACH(block, &ram_list.blocks, next) {
-if (!strncmp(id, block->idstr, sizeof(id)))
-return memory_region_get_ram_ptr(block->mr) + offset;
+if (!strncmp(id, block->idstr, sizeof(id))) {
+if (is_slave()) {
+migration_bitmap_set_dirty(block->mr->ram_addr + offset);
+return memory_region_get_ram_cache_ptr(block->mr, block)
+   + offset;
+} else {
+return memory_region_get_ram_ptr(block->mr) + offset;
+}
+}
 }
 
 error_report("Can't find block %s!", id);
@@ -1054,11 +1067,13 @@ void ram_handle_compressed(void *host, uint8_t ch, 
uint64_t size)
 }
 }
 
+static void ram_flush_cache(void);
 static int ram_load(QEMUFile *f, void *opaque, int version_id)
 {
 ram_addr_t addr;
 int flags, ret = 0;
 static uint64_t seq_iter;
+bool need_flush = false;
 
 seq_iter++;
 
@@ -1121,6 +1136,7 @@ static int ram_load(QEMUFile *f, void *opaque, int 
version_id)
 break;
 }
 
+need_flush = true;
 ch = qemu_get_byte(f);
 ram_handle_compressed(host, ch, TARGET_PAGE_SIZE);
 } else if (flags & RAM_SAVE_FLAG_PAGE) {
@@ -1133,6 +1149,7 @@ static int ram_load(QEMUFile *f, void *opaque, int 
version_id)
 break;
 }
 
+need_flush = true;
 qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
 } else if (flags & RAM_SAVE_FLAG_XBZRLE) {
 void *host = host_from_stream_offset(f, addr, flags);
@@ -1148,6 +1165,7 @@ static int ram_load(QEMUFile *f, void *opaque, int 
version_id)
 ret = -EINVAL;
 break;
 }
+need_flush = true;
 } else if (flags & RAM_SAVE_FLAG_HOOK) {
 ram_control_load_hook(f, flags);
 } else if (flags & RAM_SAVE_FLAG_EOS) {
@@ -1161,11 +1179,141 @@ static int ram_load(QEMUFile *f, void *opaque, int 
version_id)
 ret = qemu_file_get_error(f);
 }
 
+if (!ret && is_slave() && need_flush) {
+ram_flush_cache();
+}
+
 DPRINTF("Completed load of VM with exit code %d seq iteration "
 "%" PRIu64 "\n", ret, seq_iter);
 return ret;
 }
 
+/*
+ * colo cache: this is for secondary VM, we cache the whole
+ * memory of the secondary VM.
+ */
+void create_and_init_ram_cache(void)
+{
+/*
+ * called after first migration
+ */
+RAMBlock *block;
+int64_t ram_cache_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+QTAILQ_FOREACH(block, &ram_list.blocks, next) {
+block->host_cache = g_malloc(block->length);
+memcpy(block->host_cache, block->host, block->length);
+}
+
+migration_bitmap = bitmap_new(ram_cache_pages);
+migration_dirty_pages = 0;
+memory_global_dirty_log_start();
+}
+
+void release_ram_cache(void)
+{
+RAMBlock *block;
+
+if (migration_bitmap) {
+memory_global_dirty_log_stop();
+g_free(migration_bitmap);
+migration_bitmap = NULL;
+}
+
+QTAILQ_FOREACH(block, &ram_list.bl

[RFC PATCH 15/17] COLO save: reuse migration bitmap under colo checkpoint

2014-07-23 Thread Yang Hongyang 
reuse migration bitmap under colo checkpoint, only send dirty pages
per-checkpoint.

Signed-off-by: Yang Hongyang 
---
 arch_init.c| 20 +++-
 include/migration/migration-colo.h |  2 ++
 migration-colo.c   |  6 ++
 stubs/migration-colo.c | 10 ++
 4 files changed, 33 insertions(+), 5 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 8ddaf35..c84e6c8 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -52,6 +52,7 @@
 #include "exec/ram_addr.h"
 #include "hw/acpi/acpi.h"
 #include "qemu/host-utils.h"
+#include "migration/migration-colo.h"
 
 #ifdef DEBUG_ARCH_INIT
 #define DPRINTF(fmt, ...) \
@@ -769,6 +770,15 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
 RAMBlock *block;
 int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */
 
+/*
+ * migration has already setup the bitmap, reuse it.
+ */
+if (is_master()) {
+qemu_mutex_lock_ramlist();
+reset_ram_globals();
+goto out_setup;
+}
+
 mig_throttle_on = false;
 dirty_rate_high_cnt = 0;
 bitmap_sync_count = 0;
@@ -828,6 +838,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
 migration_bitmap_sync();
 qemu_mutex_unlock_iothread();
 
+out_setup:
 qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
 
 QTAILQ_FOREACH(block, &ram_list.blocks, next) {
@@ -937,7 +948,14 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
 }
 
 ram_control_after_iterate(f, RAM_CONTROL_FINISH);
-migration_end();
+
+/*
+ * Since we need to reuse dirty bitmap in colo,
+ * don't cleanup the bitmap.
+ */
+if (!migrate_use_colo() || migration_has_failed(migrate_get_current())) {
+migration_end();
+}
 
 qemu_mutex_unlock_ramlist();
 qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
diff --git a/include/migration/migration-colo.h 
b/include/migration/migration-colo.h
index 861fa27..c286a60 100644
--- a/include/migration/migration-colo.h
+++ b/include/migration/migration-colo.h
@@ -21,10 +21,12 @@ bool colo_supported(void);
 /* save */
 bool migrate_use_colo(void);
 void colo_init_checkpointer(MigrationState *s);
+bool is_master(void);
 
 /* restore */
 bool restore_use_colo(void);
 void restore_exit_colo(void);
+bool is_slave(void);
 
 void colo_process_incoming_checkpoints(QEMUFile *f);
 
diff --git a/migration-colo.c b/migration-colo.c
index 8596845..13a6a57 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -222,8 +222,6 @@ static const QEMUFileOps colo_read_ops = {
 };
 
 /* colo checkpoint control helper */
-static bool is_master(void);
-static bool is_slave(void);
 
 static void ctl_error_handler(void *opaque, int err)
 {
@@ -295,7 +293,7 @@ static int colo_ctl_get(QEMUFile *f, uint64_t require)
 
 /* save */
 
-static bool is_master(void)
+bool is_master(void)
 {
 MigrationState *s = migrate_get_current();
 return (s->state == MIG_STATE_COLO);
@@ -499,7 +497,7 @@ void colo_init_checkpointer(MigrationState *s)
 
 static Coroutine *colo;
 
-static bool is_slave(void)
+bool is_slave(void)
 {
 return colo != NULL;
 }
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index 55f0d37..ef65be6 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -22,3 +22,13 @@ void colo_init_checkpointer(MigrationState *s)
 void colo_process_incoming_checkpoints(QEMUFile *f)
 {
 }
+
+bool is_master(void)
+{
+return false;
+}
+
+bool is_slave(void)
+{
+return false;
+}
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 03/17] COLO migration: add a migration capability 'colo'

2014-07-23 Thread Yang Hongyang 
Add a migration capability 'colo'. If this capability is on,
The migration will never end, and the VM will be continuously
checkpointed.

Signed-off-by: Yang Hongyang 
---
 include/qapi/qmp/qerror.h | 3 +++
 migration.c   | 6 ++
 qapi-schema.json  | 5 -
 3 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/include/qapi/qmp/qerror.h b/include/qapi/qmp/qerror.h
index 902d1a7..226b805 100644
--- a/include/qapi/qmp/qerror.h
+++ b/include/qapi/qmp/qerror.h
@@ -166,4 +166,7 @@ void qerror_report_err(Error *err);
 #define QERR_SOCKET_CREATE_FAILED \
 ERROR_CLASS_GENERIC_ERROR, "Failed to create socket"
 
+#define QERR_COLO_UNSUPPORTED \
+ERROR_CLASS_GENERIC_ERROR, "COLO is not currently supported, please rerun 
configure with --enable-colo option in order to support COLO feature"
+
 #endif /* QERROR_H */
diff --git a/migration.c b/migration.c
index 8d675b3..ca83310 100644
--- a/migration.c
+++ b/migration.c
@@ -25,6 +25,7 @@
 #include "qemu/thread.h"
 #include "qmp-commands.h"
 #include "trace.h"
+#include "migration/migration-colo.h"
 
 enum {
 MIG_STATE_ERROR = -1,
@@ -277,6 +278,11 @@ void 
qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
 }
 
 for (cap = params; cap; cap = cap->next) {
+if (cap->value->capability == MIGRATION_CAPABILITY_COLO &&
+cap->value->state && !colo_supported()) {
+error_set(errp, QERR_COLO_UNSUPPORTED);
+continue;
+}
 s->enabled_capabilities[cap->value->capability] = cap->value->state;
 }
 }
diff --git a/qapi-schema.json b/qapi-schema.json
index b11aad2..807f5a2 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -491,10 +491,13 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #  to speed up convergence of RAM migration. (since 1.6)
 #
+# @colo: The migration will never end, and the VM will instead be continuously
+#checkpointed. The feature is disabled by default. (since 2.1)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+  'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', 'colo'] }
 
 ##
 # @MigrationCapabilityStatus
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 14/17] COLO ctl: implement colo restore

2014-07-23 Thread Yang Hongyang 
implement colo restore

Signed-off-by: Yang Hongyang 
---
 migration-colo.c | 43 +++
 1 file changed, 35 insertions(+), 8 deletions(-)

diff --git a/migration-colo.c b/migration-colo.c
index 03ac157..8596845 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -535,8 +535,9 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
 {
 int fd = qemu_get_fd(f);
 int dev_hotplug = qdev_hotplug;
-QEMUFile *ctl = NULL;
+QEMUFile *ctl = NULL, *fb = NULL;
 int ret;
+uint64_t total_size;
 
 if (!restore_use_colo()) {
 return;
@@ -560,7 +561,8 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
 goto out;
 }
 
-/* TODO: in COLO mode, slave is runing, so start the vm */
+/* in COLO mode, slave is runing, so start the vm */
+vm_start();
 
 while (true) {
 if (slave_wait_new_checkpoint(f)) {
@@ -569,43 +571,68 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
 
 /* start colo checkpoint */
 
-/* TODO: suspend guest */
+/* suspend guest */
+vm_stop_force_state(RUN_STATE_COLO);
 
 ret = colo_ctl_put(ctl, COLO_CHECKPOINT_SUSPENDED);
 if (ret) {
 goto out;
 }
 
-/* TODO: open colo buffer for read */
+/* open colo buffer for read */
+fb = qemu_fopen_ops(&colo_buffer, &colo_read_ops);
+if (!fb) {
+error_report("can't open colo buffer\n");
+goto out;
+}
 
 ret = colo_ctl_get(f, COLO_CHECKPOINT_SEND);
 if (ret) {
 goto out;
 }
 
-/* TODO: read migration data into colo buffer */
+/* read migration data into colo buffer */
+
+/* read the vmstate total size first */
+ret = colo_ctl_get_value(f, &total_size);
+if (ret) {
+goto out;
+}
+colo_buffer_extend(total_size);
+qemu_get_buffer(f, colo_buffer.data, total_size);
+colo_buffer.used = total_size;
 
 ret = colo_ctl_put(ctl, COLO_CHECKPOINT_RECEIVED);
 if (ret) {
 goto out;
 }
 
-/* TODO: load vm state */
+/* load vm state */
+if (qemu_loadvm_state(fb) < 0) {
+error_report("COLO: loadvm failed\n");
+goto out;
+}
 
 ret = colo_ctl_put(ctl, COLO_CHECKPOINT_LOADED);
 if (ret) {
 goto out;
 }
 
-/* TODO: resume guest */
+/* resume guest */
+vm_start();
 
-/* TODO: close colo buffer */
+qemu_fclose(fb);
+fb = NULL;
 }
 
 out:
 colo_buffer_destroy();
 colo = NULL;
 
+if (fb) {
+qemu_fclose(fb);
+}
+
 if (ctl) {
 qemu_fclose(ctl);
 }
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 08/17] COLO: disable qdev hotplug

2014-07-23 Thread Yang Hongyang 
COLO do not support qdev hotplug migration, disable it.

Signed-off-by: Yang Hongyang 
---
 migration-colo.c | 12 
 1 file changed, 12 insertions(+)

diff --git a/migration-colo.c b/migration-colo.c
index b90d9b6..f295e56 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -12,6 +12,7 @@
 #include "qemu/thread.h"
 #include "block/coroutine.h"
 #include "qemu/error-report.h"
+#include "hw/qdev-core.h"
 #include "migration/migration-colo.h"
 
 static QEMUBH *colo_bh;
@@ -130,6 +131,9 @@ static const QEMUFileOps colo_read_ops = {
 static void *colo_thread(void *opaque)
 {
 MigrationState *s = opaque;
+int dev_hotplug = qdev_hotplug;
+
+qdev_hotplug = 0;
 
 colo_buffer_init();
 
@@ -145,6 +149,8 @@ static void *colo_thread(void *opaque)
 qemu_bh_schedule(s->cleanup_bh);
 qemu_mutex_unlock_iothread();
 
+qdev_hotplug = dev_hotplug;
+
 return NULL;
 }
 
@@ -179,10 +185,14 @@ static Coroutine *colo;
 
 void colo_process_incoming_checkpoints(QEMUFile *f)
 {
+int dev_hotplug = qdev_hotplug;
+
 if (!restore_use_colo()) {
 return;
 }
 
+qdev_hotplug = 0;
+
 colo = qemu_coroutine_self();
 assert(colo != NULL);
 
@@ -194,5 +204,7 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
 colo = NULL;
 restore_exit_colo();
 
+qdev_hotplug = dev_hotplug;
+
 return;
 }
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 13/17] COLO ctl: implement colo save

2014-07-23 Thread Yang Hongyang 
implement colo save

Signed-off-by: Yang Hongyang 
---
 migration-colo.c | 44 ++--
 1 file changed, 38 insertions(+), 6 deletions(-)

diff --git a/migration-colo.c b/migration-colo.c
index a708872..03ac157 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -14,6 +14,7 @@
 #include "qemu/error-report.h"
 #include "hw/qdev-core.h"
 #include "qemu/timer.h"
+#include "sysemu/sysemu.h"
 #include "migration/migration-colo.h"
 #include 
 
@@ -106,12 +107,12 @@ static int colo_compare(void)
 return ioctl(comp_fd, COMP_IOCTWAIT, 250);
 }
 
-static __attribute__((unused)) int colo_compare_flush(void)
+static int colo_compare_flush(void)
 {
 return ioctl(comp_fd, COMP_IOCTFLUSH, 1);
 }
 
-static __attribute__((unused)) int colo_compare_resume(void)
+static int colo_compare_resume(void)
 {
 return ioctl(comp_fd, COMP_IOCTRESUME, 1);
 }
@@ -315,30 +316,61 @@ static int do_colo_transaction(MigrationState *s, 
QEMUFile *control,
 goto out;
 }
 
-/* TODO: suspend and save vm state to colo buffer */
+/* suspend and save vm state to colo buffer */
+
+qemu_mutex_lock_iothread();
+vm_stop_force_state(RUN_STATE_COLO);
+qemu_mutex_unlock_iothread();
+/* Disable block migration */
+s->params.blk = 0;
+s->params.shared = 0;
+qemu_savevm_state_begin(trans, &s->params);
+qemu_savevm_state_complete(trans);
+
+qemu_fflush(trans);
 
 ret = colo_ctl_put(s->file, COLO_CHECKPOINT_SEND);
 if (ret) {
 goto out;
 }
 
-/* TODO: send vmstate to slave */
+/* send vmstate to slave */
+
+/* we send the total size of the vmstate first */
+ret = colo_ctl_put(s->file, colo_buffer.used);
+if (ret) {
+goto out;
+}
+
+qemu_put_buffer_async(s->file, colo_buffer.data, colo_buffer.used);
+ret = qemu_file_get_error(s->file);
+if (ret < 0) {
+goto out;
+}
+qemu_fflush(s->file);
 
 ret = colo_ctl_get(control, COLO_CHECKPOINT_RECEIVED);
 if (ret) {
 goto out;
 }
 
-/* TODO: Flush network etc. */
+/* Flush network etc. */
+colo_compare_flush();
 
 ret = colo_ctl_get(control, COLO_CHECKPOINT_LOADED);
 if (ret) {
 goto out;
 }
 
-/* TODO: resume master */
+colo_compare_resume();
+ret = 0;
 
 out:
+/* resume master */
+qemu_mutex_lock_iothread();
+vm_start();
+qemu_mutex_unlock_iothread();
+
 return ret;
 }
 
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 17/17] HACK: trigger checkpoint every 500ms

2014-07-23 Thread Yang Hongyang 
Because COLO Agent is under development. We add this hack for
test purpose. Trigger checkpoint every 500ms so that we can
test the process of COLO save/restore.
NOTE:
  This is only a hack, and will be removed at last.

Signed-off-by: Yang Hongyang 
---
 migration-colo.c | 14 +-
 1 file changed, 5 insertions(+), 9 deletions(-)

diff --git a/migration-colo.c b/migration-colo.c
index 52156e7..4be037e 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -23,7 +23,7 @@
  * this is large because COLO checkpoint will mostly depend on
  * COLO compare module.
  */
-#define CHKPOINT_TIMER 1
+#define CHKPOINT_TIMER 500
 
 enum {
 COLO_READY = 0x46,
@@ -79,11 +79,6 @@ static int comp_fd = -1;
 
 static int colo_compare_init(void)
 {
-comp_fd = open(COMPARE_DEV, O_RDONLY);
-if (comp_fd < 0) {
-return -1;
-}
-
 return 0;
 }
 
@@ -104,17 +99,18 @@ static void colo_compare_destroy(void)
  */
 static int colo_compare(void)
 {
-return ioctl(comp_fd, COMP_IOCTWAIT, 250);
+errno = ERESTART;
+return 1;
 }
 
 static int colo_compare_flush(void)
 {
-return ioctl(comp_fd, COMP_IOCTFLUSH, 1);
+return 0;
 }
 
 static int colo_compare_resume(void)
 {
-return ioctl(comp_fd, COMP_IOCTRESUME, 1);
+return 0;
 }
 
 /* colo buffer */
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 11/17] COLO ctl: implement colo checkpoint protocol

2014-07-23 Thread Yang Hongyang 
implement colo checkpoint protocol.

Checkpoint synchronzing points.

  Primary Secondary
  NEW @
  Suspend
  SUSPENDED   @
  Suspend&Save state
  SEND@
  Send state  Receive state
  RECEIVED@
  Flush network   Load state
  LOADED  @
  Resume  Resume

  Start Comparing
NOTE:
 1) '@' who sends the message
 2) Every sync-point is synchronized by two sides with only
one handshake(single direction) for low-latency.
If more strict synchronization is required, a opposite direction
sync-point should be added.
 3) Since sync-points are single direction, the remote side may
go forward a lot when this side just receives the sync-point.

Signed-off-by: Yang Hongyang 
---
 migration-colo.c | 268 +--
 1 file changed, 262 insertions(+), 6 deletions(-)

diff --git a/migration-colo.c b/migration-colo.c
index 2699e77..a708872 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -24,6 +24,41 @@
  */
 #define CHKPOINT_TIMER 1
 
+enum {
+COLO_READY = 0x46,
+
+/*
+ * Checkpoint synchronzing points.
+ *
+ *  Primary Secondary
+ *  NEW @
+ *  Suspend
+ *  SUSPENDED   @
+ *  Suspend&Save state
+ *  SEND@
+ *  Send state  Receive state
+ *  RECEIVED@
+ *  Flush network   Load state
+ *  LOADED  @
+ *  Resume  Resume
+ *
+ *  Start Comparing
+ * NOTE:
+ * 1) '@' who sends the message
+ * 2) Every sync-point is synchronized by two sides with only
+ *one handshake(single direction) for low-latency.
+ *If more strict synchronization is required, a opposite direction
+ *sync-point should be added.
+ * 3) Since sync-points are single direction, the remote side may
+ *go forward a lot when this side just receives the sync-point.
+ */
+COLO_CHECKPOINT_NEW,
+COLO_CHECKPOINT_SUSPENDED,
+COLO_CHECKPOINT_SEND,
+COLO_CHECKPOINT_RECEIVED,
+COLO_CHECKPOINT_LOADED,
+};
+
 static QEMUBH *colo_bh;
 
 bool colo_supported(void)
@@ -185,30 +220,161 @@ static const QEMUFileOps colo_read_ops = {
 .close = colo_close,
 };
 
+/* colo checkpoint control helper */
+static bool is_master(void);
+static bool is_slave(void);
+
+static void ctl_error_handler(void *opaque, int err)
+{
+if (is_slave()) {
+/* TODO: determine whether we need to failover */
+/* FIXME: we will not failover currently, just kill slave */
+error_report("error: colo transmission failed!\n");
+exit(1);
+} else if (is_master()) {
+/* Master still alive, do not failover */
+error_report("error: colo transmission failed!\n");
+return;
+} else {
+error_report("COLO: Unexpected error happend!\n");
+exit(EXIT_FAILURE);
+}
+}
+
+static int colo_ctl_put(QEMUFile *f, uint64_t request)
+{
+int ret = 0;
+
+qemu_put_be64(f, request);
+qemu_fflush(f);
+
+ret = qemu_file_get_error(f);
+if (ret < 0) {
+ctl_error_handler(f, ret);
+return 1;
+}
+
+return ret;
+}
+
+static int colo_ctl_get_value(QEMUFile *f, uint64_t *value)
+{
+int ret = 0;
+uint64_t temp;
+
+temp = qemu_get_be64(f);
+
+ret = qemu_file_get_error(f);
+if (ret < 0) {
+ctl_error_handler(f, ret);
+return 1;
+}
+
+*value = temp;
+return 0;
+}
+
+static int colo_ctl_get(QEMUFile *f, uint64_t require)
+{
+int ret;
+uint64_t value;
+
+ret = colo_ctl_get_value(f, &value);
+if (ret) {
+return ret;
+}
+
+if (value != require) {
+error_report("unexpected state received!\n");
+exit(1);
+}
+
+return ret;
+}
+
 /* save */
 
-static __attribute__((unused)) bool is_master(void)
+static bool is_master(void)
 {
 MigrationState *s = migrate_get_current();
 return (s->state == MIG_STATE_COLO);
 }
 
+static int do_colo_transaction(MigrationState *s, QEMUFile *control,
+   QEMUFile *trans)
+{
+int ret;
+
+ret = colo_ctl_put(s->file, COLO_CHECKPOINT_NEW);
+if (ret) {
+goto out;
+}
+
+ret = colo_ctl_get(control, COLO_CHECKPOINT_SUSPENDED);
+if (ret) {
+goto out;
+}
+
+/* TODO: suspend and save vm state to colo buffer */
+
+ret = colo_ctl_put(s-&g

[RFC PATCH 06/17] COLO restore: integrate COLO checkpointed restore into qemu restore

2014-07-23 Thread Yang Hongyang 
enter colo checkpointed restore loop after live migration.

Signed-off-by: Yang Hongyang 
---
 include/migration/migration-colo.h |  6 ++
 migration-colo-comm.c  | 10 ++
 migration-colo.c   | 22 ++
 migration.c|  3 +++
 stubs/migration-colo.c |  4 
 5 files changed, 45 insertions(+)

diff --git a/include/migration/migration-colo.h 
b/include/migration/migration-colo.h
index 24589c0..861fa27 100644
--- a/include/migration/migration-colo.h
+++ b/include/migration/migration-colo.h
@@ -22,4 +22,10 @@ bool colo_supported(void);
 bool migrate_use_colo(void);
 void colo_init_checkpointer(MigrationState *s);
 
+/* restore */
+bool restore_use_colo(void);
+void restore_exit_colo(void);
+
+void colo_process_incoming_checkpoints(QEMUFile *f);
+
 #endif
diff --git a/migration-colo-comm.c b/migration-colo-comm.c
index 4504ceb..b12a57a 100644
--- a/migration-colo-comm.c
+++ b/migration-colo-comm.c
@@ -38,6 +38,16 @@ static void colo_info_save(QEMUFile *f, void *opaque)
 
 /* restore */
 
+bool restore_use_colo(void)
+{
+return colo_requested;
+}
+
+void restore_exit_colo(void)
+{
+colo_requested = false;
+}
+
 static int colo_info_load(QEMUFile *f, void *opaque, int version_id)
 {
 int value = qemu_get_byte(f);
diff --git a/migration-colo.c b/migration-colo.c
index 0cef8bd..d566b9d 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -10,6 +10,7 @@
 
 #include "qemu/main-loop.h"
 #include "qemu/thread.h"
+#include "block/coroutine.h"
 #include "migration/migration-colo.h"
 
 static QEMUBH *colo_bh;
@@ -62,3 +63,24 @@ void colo_init_checkpointer(MigrationState *s)
 colo_bh = qemu_bh_new(colo_start_checkpointer, s);
 qemu_bh_schedule(colo_bh);
 }
+
+/* restore */
+
+static Coroutine *colo;
+
+void colo_process_incoming_checkpoints(QEMUFile *f)
+{
+if (!restore_use_colo()) {
+return;
+}
+
+colo = qemu_coroutine_self();
+assert(colo != NULL);
+
+/* TODO: COLO checkpointed restore loop */
+
+colo = NULL;
+restore_exit_colo();
+
+return;
+}
diff --git a/migration.c b/migration.c
index b7f8e7e..190571d 100644
--- a/migration.c
+++ b/migration.c
@@ -86,6 +86,9 @@ static void process_incoming_migration_co(void *opaque)
 int ret;
 
 ret = qemu_loadvm_state(f);
+if (!ret) {
+colo_process_incoming_checkpoints(f);
+}
 qemu_fclose(f);
 free_xbzrle_decoded_buf();
 if (ret < 0) {
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index 9013c40..55f0d37 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -18,3 +18,7 @@ bool colo_supported(void)
 void colo_init_checkpointer(MigrationState *s)
 {
 }
+
+void colo_process_incoming_checkpoints(QEMUFile *f)
+{
+}
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 04/17] COLO info: use colo info to tell migration target colo is enabled

2014-07-23 Thread Yang Hongyang 
migrate colo info to migration target to tell the target colo is
enabled.

Signed-off-by: Yang Hongyang 
---
 Makefile.objs  |  1 +
 include/migration/migration-colo.h |  3 ++
 migration-colo-comm.c  | 68 ++
 vl.c   |  4 +++
 4 files changed, 76 insertions(+)
 create mode 100644 migration-colo-comm.c

diff --git a/Makefile.objs b/Makefile.objs
index cab5824..1836a68 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -50,6 +50,7 @@ common-obj-$(CONFIG_POSIX) += os-posix.o
 common-obj-$(CONFIG_LINUX) += fsdev/
 
 common-obj-y += migration.o migration-tcp.o
+common-obj-y += migration-colo-comm.o
 common-obj-$(CONFIG_COLO) += migration-colo.o
 common-obj-y += vmstate.o
 common-obj-y += qemu-file.o
diff --git a/include/migration/migration-colo.h 
b/include/migration/migration-colo.h
index 35b384c..e3735d8 100644
--- a/include/migration/migration-colo.h
+++ b/include/migration/migration-colo.h
@@ -12,6 +12,9 @@
 #define QEMU_MIGRATION_COLO_H
 
 #include "qemu-common.h"
+#include "migration/migration.h"
+
+void colo_info_mig_init(void);
 
 bool colo_supported(void);
 
diff --git a/migration-colo-comm.c b/migration-colo-comm.c
new file mode 100644
index 000..ccbc246
--- /dev/null
+++ b/migration-colo-comm.c
@@ -0,0 +1,68 @@
+/*
+ *  COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ *  (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ *  Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ *
+ */
+
+#include 
+
+#define DEBUG_COLO
+
+#ifdef DEBUG_COLO
+#define DPRINTF(fmt, ...) \
+do { fprintf(stdout, "COLO: " fmt, ## __VA_ARGS__); } while (0)
+#else
+#define DPRINTF(fmt, ...) \
+do { } while (0)
+#endif
+
+static bool colo_requested;
+
+/* save */
+
+static bool migrate_use_colo(void)
+{
+MigrationState *s = migrate_get_current();
+return s->enabled_capabilities[MIGRATION_CAPABILITY_COLO];
+}
+
+static void colo_info_save(QEMUFile *f, void *opaque)
+{
+qemu_put_byte(f, migrate_use_colo());
+}
+
+/* restore */
+
+static int colo_info_load(QEMUFile *f, void *opaque, int version_id)
+{
+int value = qemu_get_byte(f);
+
+if (value && !colo_supported()) {
+fprintf(stderr, "COLO is not supported\n");
+return -EINVAL;
+}
+
+if (value && !colo_requested) {
+DPRINTF("COLO requested!\n");
+}
+
+colo_requested = value;
+
+return 0;
+}
+
+static SaveVMHandlers savevm_colo_info_handlers = {
+.save_state = colo_info_save,
+.load_state = colo_info_load,
+};
+
+void colo_info_mig_init(void)
+{
+register_savevm_live(NULL, "colo info", -1, 1,
+ &savevm_colo_info_handlers, NULL);
+}
diff --git a/vl.c b/vl.c
index fe451aa..1a282d8 100644
--- a/vl.c
+++ b/vl.c
@@ -89,6 +89,7 @@ int main(int argc, char **argv)
 #include "sysemu/dma.h"
 #include "audio/audio.h"
 #include "migration/migration.h"
+#include "migration/migration-colo.h"
 #include "sysemu/kvm.h"
 #include "qapi/qmp/qjson.h"
 #include "qemu/option.h"
@@ -4339,6 +4340,9 @@ int main(int argc, char **argv, char **envp)
 
 blk_mig_init();
 ram_mig_init();
+if (colo_supported()) {
+colo_info_mig_init();
+}
 
 /* open the virtual block devices */
 if (snapshot)
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html