This is the 3th version of COLO, it is only COLO frame part, include: VM checkpoint, failover, proxy API, block replication API, not include block replication. The block part has been sent by wencongyang: '[RFC PATCH 00/14] Block replication for continuous checkpoints'
You can get the integrated qemu colo patches from github: https://github.com/coloft/qemu/commits/colo-v1.0 Compared with the previous version, we have realized all parts of COLO frame, and it is works now. The main change since last version is, we use colo proxy mode instead of colo agent, they are all used for network packets compare, but proxy is more efficient, it is based on netfilter. Another modification is we implement new block replication scheme, you can get more info from wencongyang's block patch series If you don't know about COLO, please refer to below link for detailed information. The idea is presented in Xen summit 2012, and 2013, and academia paper in SOCC 2013. It's also presented in KVM forum in 2013: http://www.linux-kvm.org/wiki/images/1/1d/Kvm-forum-2013-COLO.pdf Previous posted RFC proposal: http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg05567.html http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg04459.html The below is the detail about test COLO, you can also get this info from http://wiki.qemu.org/Features/COLO. * Hardware requirements There is at least one directly connected nic to forward the network requests from client to secondary VM. The directly connected nic must not be used by any other purpose. * Network link topology =================================normal ====================================== +--------+ |client | master +----+---+ slave -------------------------+ | + -------------------------+ PVM | + | | +-------+ +----[eth0]-----[switch]-----[eth0]---------+ | |guest | +---+-+ | | +---+-+ | | [tap0]--+ br0 | | | | br0 | | | | +-----+ [eth1]-----[forward]----[eth1]--+ +-----+ SVM | +-------+ | | | +-------+| | | | +-----+ | guest || [eth2]---[checkpoint]---[eth2] +--+br1 |-[tap0] || | | +-----+ | || | | +-------+| -------------------------+ +--------------------------+ e.g. master: br0: 192.168.0.33 eth1: 192.168.1.33 eth2: 192.168.2.33 slave: br0: 192.168.0.88 br1: no ip address eth1: 192.168.1.88 eth2: 192.168.2.88 (Actually, you can also use eth0 as checkpoint channel) Note: in normal, SVM will always be linked to br1 like above until failover. * Test environment prepare: 1. Set Up the Bridge and network environment You must setup you network environment like above picture, In master, setup a bridge br0, using command brctl, like: # ifconfig eth0 down # ifconfig eth0 0.0.0.0 # brctl addbr br0 # brctl addif br0 eth0 # ifconfig br0 192.168.0.33 netmask 255.255.255.0 # ifconfig eth0 up In slave, setup two bridge br0, br1, commands are same with above, please note that br1 is linked to eth1(the forward nic). 2.Qemu-ifup We need a script to bring up the TAP interface. You can find this info from http://en.wikibooks.org/wiki/QEMU/Networking. Master: root@master# cat /etc/qemu-ifup #!/bin/sh switch=br0 if [ -n "$1" ]; then ip link set $1 up brctl addif ${switch} $1 fi Slave: root@slave # cat /etc/qemu-ifup #!/bin/sh switch=br1 #in primary, switch is br0. in secondary switch is br1 if [ -n "$1" ]; then ip link set $1 up brctl addif ${switch} $1 fi 3. Prepare host kernel colo-proxy kernel module need cooperate with linux kernel. You should put a kernel patch 'colo-patch-for-kernel.patch' (It's based on linux kernel-3.19) which you can get from https://github.com/gao-feng/colo-proxy.git and then compile kernel and intall the new kernel. 4. Proxy module proxy module is used for network packets compare, you can also get the lastest version from: https://github.com/gao-feng/colo-proxy.git. You can compile and install it by using command 'make' && 'make install'. 5. Modified iptables We have add a new rule to iptables command, so please get the patch from https://github.com/gao-feng/colo-proxy/blob/master/COLO-library_for_iptables-1.4.21.patch It is based on version 1.4.21. 6. Qemu colo Checkout the latest colo branch from https://github.com/coloft/qemu/commits/colo-v1.0 configure and make: # ./configure --target-list=x86_64-softmmu --enable-colo --enable-quorum # make * Test steps: 1. load module # modprobe nf_conntrack_colo (Other colo module will be automatically loaded by script colo-proxy-script.sh) # modprobe xt_mark # modprobe kvm-intel 2. startup qemu master: # qemu-system-x86_64 -enable-kvm -netdev tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1 -device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive driver=quorum,read-pattern=first,children.0.file.filename=suse11_3.img,children.0.driver=raw,children.1.file.driver=nbd+colo,children.1.file.host=192.168.2.88,children.1.file.port=8889,children.1.file.export=colo1,children.1.driver=raw,if=virtio -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -S slave: # qemu-system-x86_64 -enable-kvm -netdev tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1 -device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive driver=blkcolo,export=colo1,backing.file.filename=suse11_3.img,backing.driver=raw,if=virtio -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -incoming tcp:0:8888 3. On Secondary VM's QEMU monitor, run (qemu) nbd_server_start 192.168.2.88:8889 4.on Primary VM's QEMU monitor, run following command: (qemu) migrate_set_capability colo on (qemu) migrate tcp:192.168.2.88:8888 5. done You will see two runing VMs, whenever you make changes to PVM, SVM will be synced to PVM's state. 6. failover test: You can kill SVM (PVM) and run 'colo_lost_heartbeat' in PVM's (SVM's) monitor at the same time, then PVM (SVM) will failover and client will not feel this change. It is still a framework, far away from commercial use, so any comments/feedbacks are warmly welcomed ;) PS: We (huawei) have cooperated with fujitsu on COLO work, and we work mainly on COLO frame and fujitsu will focus on COLO block. TODO list: 1) Optimize the process of checkpoint, shorten the time-consuming 2) Add more debug/stat info 3) Strengthen failover 4) The capability of continuous FT v3: - use proxy instead of colo agent to compare network packets - add block replication - Optimize failover disposal - handle shutdown v2: - use QEMUSizedBuffer/QEMUFile as COLO buffer - colo support is enabled by default - add nic replication support - addressed comments from Eric Blake and Dr. David Alan Gilbert v1: - implement the frame of colo zhanghailiang (27): configure: Add parameter for configure to enable/disable COLO support migration: Introduce capability 'colo' to migration COLO: migrate colo related info to slave migration: Integrate COLO checkpoint process into migration migration: Integrate COLO checkpoint process into loadvm migration: Don't send vm description in COLO mode COLO: Implement colo checkpoint protocol COLO: Add a new RunState RUN_STATE_COLO QEMUSizedBuffer: Introduce two help functions for qsb COLO: Save VM state to slave when do checkpoint COLO RAM: Load PVM's dirty page into SVM's RAM cache temporarily COLO VMstate: Load VM state into qsb before restore it COLO RAM: Flush cached RAM into SVM's memory COLO failover: Introduce a new command to trigger a failover COLO failover: Implement COLO master/slave failover work COLO failover: Don't do failover during loading VM's state COLO: Add new command parameter 'colo_nicname' 'colo_script' for net COLO NIC: Init/remove colo nic devices when add/cleanup tap devices COLO NIC: Implement colo nic device interface configure() COLO NIC : Implement colo nic init/destroy function COLO NIC: Some init work related with proxy module COLO: Do checkpoint according to the result of net packets comparing COLO: Improve checkpoint efficiency by do additional periodic checkpoint COLO NIC: Implement NIC checkpoint and failover COLO: Disable qdev hotplug when VM is in COLO mode COLO: Implement shutdown checkpoint COLO: Add block replication into colo process arch_init.c | 196 ++++++++- configure | 14 + hmp-commands.hx | 15 + hmp.c | 7 + hmp.h | 1 + include/exec/cpu-all.h | 1 + include/migration/migration-colo.h | 57 +++ include/migration/migration-failover.h | 22 + include/migration/migration.h | 14 + include/migration/qemu-file.h | 3 +- include/net/colo-nic.h | 25 ++ include/net/net.h | 4 + include/sysemu/sysemu.h | 3 + migration/Makefile.objs | 2 + migration/colo-comm.c | 81 ++++ migration/colo-failover.c | 48 +++ migration/colo.c | 743 +++++++++++++++++++++++++++++++++ migration/migration.c | 74 +++- migration/qemu-file-buf.c | 57 +++ net/Makefile.objs | 1 + net/colo-nic.c | 438 +++++++++++++++++++ net/tap.c | 45 +- qapi-schema.json | 27 +- qemu-options.hx | 10 +- qmp-commands.hx | 19 + savevm.c | 10 +- scripts/colo-proxy-script.sh | 88 ++++ stubs/Makefile.objs | 1 + stubs/migration-colo.c | 49 +++ vl.c | 36 +- 30 files changed, 2047 insertions(+), 44 deletions(-) create mode 100644 include/migration/migration-colo.h create mode 100644 include/migration/migration-failover.h create mode 100644 include/net/colo-nic.h create mode 100644 migration/colo-comm.c create mode 100644 migration/colo-failover.c create mode 100644 migration/colo.c create mode 100644 net/colo-nic.c create mode 100755 scripts/colo-proxy-script.sh create mode 100644 stubs/migration-colo.c -- 1.7.12.4