Re: [Qemu-devel] [Bug 1754542] [NEW] colo: secondary vm crash when execute x-colo-lost-heartbeat

2018-03-17 Thread Zhang Chen
Hi Suiheng, Sorry for slow reply, the document 'COLO-FT.txt' in qemu is out of date, I will update it lately. Please follow this step to run COLO(the command has been changed). https://wiki.qemu.org/Features/COLO Thanks Zhang Chen On Fri, Mar 9, 2018 at 10:54 AM,

[Qemu-devel] [PATCH V6 17/17] COLO: quick failover process by kick COLO thread

2018-03-11 Thread Zhang Chen
From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang --- migration/colo.c | 8 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migra

Re: [Qemu-devel] [PATCH V5 11/17] qapi: Add new command to query colo status

2018-03-11 Thread Zhang Chen
On Sun, Mar 11, 2018 at 5:33 AM, Eric Blake wrote: > On 03/10/2018 03:22 PM, Zhang Chen wrote: > >> Libvirt or other high level sofware can use this command query colo >> status. >> > > s/sofware/software/ > > You can test this command like that: >> {&

[Qemu-devel] [PATCH V6 16/17] COLO: notify net filters about checkpoint/failover event

2018-03-11 Thread Zhang Chen
From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang --- migration/colo.c | 12 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index f3d588e..1a59baa 100644 --- a/migration/colo.c +++ b

[Qemu-devel] [PATCH V6 14/17] filter: Add handle_event method for NetFilterClass

2018-03-10 Thread Zhang Chen
Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 + net/filter.c | 17 + net/net.c| 28 3 files changed, 50 insertions(+)

[Qemu-devel] [PATCH V6 15/17] filter-rewriter: handle checkpoint and failover event

2018-03-10 Thread Zhang Chen
: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 + net/filter-rewriter.c | 40 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index f514492..f3d588e 100644 --- a/migration/colo.c +++ b

[Qemu-devel] [PATCH V6 12/17] savevm: split the process of different stages for loadvm/savevm

2018-03-10 Thread Zhang Chen
_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c

[Qemu-devel] [PATCH V6 13/17] COLO: flush host dirty ram from cache

2018-03-10 Thread Zhang Chen
From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- migration/ram.c | 12 1 file changed, 12 insertions(+) diff --git a

[Qemu-devel] [PATCH V6 11/17] qapi: Add new command to query colo status

2018-03-10 Thread Zhang Chen
Libvirt or other high level software can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen --- migration/colo.c| 31 +++ qapi/migration.json | 33

[Qemu-devel] [PATCH V6 04/17] COLO: integrate colo compare with colo frame

2018-03-10 Thread Zhang Chen
-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 -- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index

[Qemu-devel] [PATCH V6 10/17] qmp event: Add COLO_EXIT event to notify users while exited COLO

2018-03-10 Thread Zhang Chen
x27;s failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c| 20 qapi/migration.json | 35 +

[Qemu-devel] [PATCH V6 09/17] COLO: Flush memory data from ram cache

2018-03-10 Thread Zhang Chen
From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all

[Qemu-devel] [PATCH V6 05/17] COLO: Add block replication into colo process

2018-03-10 Thread Zhang Chen
Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c

[Qemu-devel] [PATCH V6 07/17] COLO: Load dirty pages into SVM's RAM cache firstly

2018-03-10 Thread Zhang Chen
eive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 99 +++-- migration/ram.h | 4 ++ mig

[Qemu-devel] [PATCH V6 02/17] colo-compare: implement the process of checkpoint

2018-03-10 Thread Zhang Chen
While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include

[Qemu-devel] [PATCH V6 08/17] ram/COLO: Record the dirty pages that SVM received

2018-03-10 Thread Zhang Chen
We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the

[Qemu-devel] [PATCH V6 01/17] filter-rewriter: fix memory leak for connection in connection_track_table

2018-03-10 Thread Zhang Chen
After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by:

[Qemu-devel] [PATCH V6 06/17] COLO: Remove colo_state migration struct

2018-03-10 Thread Zhang Chen
: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 5 ++-- migration/Makefile.objs | 2 +- migration/colo-comm.c| 76 migration/colo.c | 13 - migration/migration.c| 23 ++- migration/savevm.c

[Qemu-devel] [PATCH V6 00/17] COLO: integrate colo frame with block replication and COLO proxy

2018-03-10 Thread Zhang Chen
k part to indepentent patch set. - Optimize codes. Zhang Chen (10): filter-rewriter: fix memory leak for connection in connection_track_table colo-compare: implement the process of checkpoint colo-compare: use notifier to notify packets comparing result COLO: integrate colo compare with

[Qemu-devel] [PATCH V6 03/17] colo-compare: use notifier to notify packets comparing result

2018-03-10 Thread Zhang Chen
It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +--- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-)

[Qemu-devel] [PATCH V6 00/17] COLO: integrate colo frame with block replication and COLO proxy

2018-03-10 Thread Zhang Chen
am codes. - Fix mingw compile error in patch 02/16. - Fix some comments. - Fix conflect with the patch "migration: remove "enable_colo" var" in patch 06/16. V3: - Address community comments from V2. - Rebase on upstream codes. - Fix several bugs. - Splite shared disk part to

[Qemu-devel] [PATCH V5 13/17] COLO: flush host dirty ram from cache

2018-03-10 Thread Zhang Chen
From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- migration/ram.c | 12 1 file changed, 12 insertions(+) diff --git a

[Qemu-devel] [PATCH V5 16/17] COLO: notify net filters about checkpoint/failover event

2018-03-10 Thread Zhang Chen
From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang --- migration/colo.c | 12 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 6834f3d..5fb4d8c 100644 --- a/migration/colo.c +++ b

[Qemu-devel] [PATCH V5 17/17] COLO: quick failover process by kick COLO thread

2018-03-10 Thread Zhang Chen
From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang --- migration/colo.c | 8 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migra

[Qemu-devel] [PATCH V5 11/17] qapi: Add new command to query colo status

2018-03-10 Thread Zhang Chen
Libvirt or other high level sofware can use this command query colo status. You can test this command like that: {'execute':'query-colo-status'} Signed-off-by: Zhang Chen --- migration/colo.c| 35 +++ qapi

[Qemu-devel] [PATCH V5 09/17] COLO: Flush memory data from ram cache

2018-03-10 Thread Zhang Chen
From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all

[Qemu-devel] [PATCH V5 10/17] qmp event: Add COLO_EXIT event to notify users while exited COLO

2018-03-10 Thread Zhang Chen
x27;s failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c| 20 qapi/migration.json | 35 +

[Qemu-devel] [PATCH V5 08/17] ram/COLO: Record the dirty pages that SVM received

2018-03-10 Thread Zhang Chen
We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the

[Qemu-devel] [PATCH V5 15/17] filter-rewriter: handle checkpoint and failover event

2018-03-10 Thread Zhang Chen
: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 + net/filter-rewriter.c | 40 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 1633aed..6834f3d 100644 --- a/migration/colo.c +++ b

[Qemu-devel] [PATCH V5 14/17] filter: Add handle_event method for NetFilterClass

2018-03-10 Thread Zhang Chen
Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 + net/filter.c | 17 + net/net.c| 28 3 files changed, 50 insertions(+)

[Qemu-devel] [PATCH V5 07/17] COLO: Load dirty pages into SVM's RAM cache firstly

2018-03-10 Thread Zhang Chen
eive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 99 +++-- migration/ram.h | 4 ++ mig

[Qemu-devel] [PATCH V5 12/17] savevm: split the process of different stages for loadvm/savevm

2018-03-10 Thread Zhang Chen
_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c

[Qemu-devel] [PATCH V5 05/17] COLO: Add block replication into colo process

2018-03-10 Thread Zhang Chen
Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c

[Qemu-devel] [PATCH V5 04/17] COLO: integrate colo compare with colo frame

2018-03-10 Thread Zhang Chen
-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 -- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index

[Qemu-devel] [PATCH V5 02/17] colo-compare: implement the process of checkpoint

2018-03-10 Thread Zhang Chen
While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include

[Qemu-devel] [PATCH V5 01/17] filter-rewriter: fix memory leak for connection in connection_track_table

2018-03-10 Thread Zhang Chen
After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by:

[Qemu-devel] [PATCH V5 03/17] colo-compare: use notifier to notify packets comparing result

2018-03-10 Thread Zhang Chen
It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +--- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-)

[Qemu-devel] [PATCH V5 00/17] COLO: integrate colo frame with block replication and COLO proxy

2018-03-10 Thread Zhang Chen
ble_colo" var" in patch 06/16. V3: - Address community comments from V2. - Rebase on upstream codes. - Fix several bugs. - Splite shared disk part to indepentent patch set. - Optimize codes. Zhang Chen (10): filter-rewriter: fix memory leak for connection in connection_track_t

[Qemu-devel] [PATCH V5 06/17] COLO: Remove colo_state migration struct

2018-03-10 Thread Zhang Chen
: zhanghailiang Signed-off-by: Zhang Chen --- include/migration/colo.h | 5 ++-- migration/Makefile.objs | 2 +- migration/colo-comm.c| 76 migration/colo.c | 13 - migration/migration.c| 23 ++- migration/savevm.c

Re: [Qemu-devel] [PATCH V4 10/16] qmp event: Add COLO_EXIT event to notify users while exited COLO

2018-02-06 Thread Zhang Chen
On Tue, Feb 6, 2018 at 5:53 PM, Markus Armbruster wrote: > Zhang Chen writes: > > > On Tue, Feb 6, 2018 at 3:27 PM, Markus Armbruster > wrote: > > > >> Zhang Chen writes: > >> > >> > On Sat, Feb 3, 2018 at 3:49 PM, Markus Armbruster > w

Re: [Qemu-devel] [PATCH V4 10/16] qmp event: Add COLO_EXIT event to notify users while exited COLO

2018-02-06 Thread Zhang Chen
On Tue, Feb 6, 2018 at 3:27 PM, Markus Armbruster wrote: > Zhang Chen writes: > > > On Sat, Feb 3, 2018 at 3:49 PM, Markus Armbruster > wrote: > > > >> Zhang Chen writes: > >> > >> > From: zhanghailiang > >> > > >&

Re: [Qemu-devel] [PATCH V4 10/16] qmp event: Add COLO_EXIT event to notify users while exited COLO

2018-02-05 Thread Zhang Chen
On Sat, Feb 3, 2018 at 3:49 PM, Markus Armbruster wrote: > Zhang Chen writes: > > > From: zhanghailiang > > > > If some errors happen during VM's COLO FT stage, it's important to > > notify the users of this event. Together with 'x-colo-lost-

Re: [Qemu-devel] [PATCH V4 00/16] COLO: integrate colo frame with block replication and COLO proxy

2018-01-29 Thread Zhang Chen
Hi~ All~ No news for a long time, this is the last part to make COLO running in upstream. Any comments for this series? Thanks Zhang Chen On Fri, Jan 19, 2018 at 1:44 PM, Zhang Chen wrote: > From: Zhang Chen > > Hi~ > > COLO Frame, block replication and COLO proxy(colo-compar

[Qemu-devel] [PATCH V4 16/16] COLO: quick failover process by kick COLO thread

2018-01-19 Thread Zhang Chen
From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang --- migration/colo.c | 8 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migra

[Qemu-devel] [PATCH V4 15/16] COLO: notify net filters about checkpoint/failover event

2018-01-19 Thread Zhang Chen
From: zhanghailiang Notify all net filters about the checkpoint and failover event. Signed-off-by: zhanghailiang --- migration/colo.c | 12 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 9eab4a3..10bc80c 100644 --- a/migration/colo.c +++ b

[Qemu-devel] [PATCH V4 11/16] savevm: split the process of different stages for loadvm/savevm

2018-01-19 Thread Zhang Chen
_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c

[Qemu-devel] [PATCH V4 13/16] filter: Add handle_event method for NetFilterClass

2018-01-19 Thread Zhang Chen
Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Signed-off-by: zhanghailiang --- include/net/filter.h | 5 + net/filter.c | 17 + net/net.c| 28 3 files changed, 50 insertions(+)

[Qemu-devel] [PATCH V4 10/16] qmp event: Add COLO_EXIT event to notify users while exited COLO

2018-01-19 Thread Zhang Chen
x27;s failover verdict, it is still necessary to notify users that we exited COLO mode. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c| 19 +++ qapi/migration.json | 35

[Qemu-devel] [PATCH V4 12/16] COLO: flush host dirty ram from cache

2018-01-19 Thread Zhang Chen
From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- migration/ram.c | 12 1 file changed, 12 insertions(+) diff --git a

[Qemu-devel] [PATCH V4 09/16] COLO: Flush memory data from ram cache

2018-01-19 Thread Zhang Chen
From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all

[Qemu-devel] [PATCH V4 07/16] COLO: Load dirty pages into SVM's RAM cache firstly

2018-01-19 Thread Zhang Chen
eive all PVM's state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 97 +++-- migration/ram.h | 4 ++ mig

[Qemu-devel] [PATCH V4 14/16] filter-rewriter: handle checkpoint and failover event

2018-01-19 Thread Zhang Chen
: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 + net/filter-rewriter.c | 40 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index a931ff2..9eab4a3 100644 --- a/migration/colo.c +++ b

[Qemu-devel] [PATCH V4 08/16] ram/COLO: Record the dirty pages that SVM received

2018-01-19 Thread Zhang Chen
We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the

[Qemu-devel] [PATCH V4 04/16] COLO: integrate colo compare with colo frame

2018-01-19 Thread Zhang Chen
-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 -- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index

[Qemu-devel] [PATCH V4 05/16] COLO: Add block replication into colo process

2018-01-19 Thread Zhang Chen
Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- migration/colo.c

[Qemu-devel] [PATCH V4 06/16] COLO: Remove colo_state migration struct

2018-01-19 Thread Zhang Chen
From: zhanghailiang We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Sign

[Qemu-devel] [PATCH V4 03/16] colo-compare: use notifier to notify packets comparing result

2018-01-19 Thread Zhang Chen
It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +--- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(+), 7 deletions(-)

[Qemu-devel] [PATCH V4 01/16] filter-rewriter: fix memory leak for connection in connection_track_table

2018-01-19 Thread Zhang Chen
After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by:

[Qemu-devel] [PATCH V4 02/16] colo-compare: implement the process of checkpoint

2018-01-19 Thread Zhang Chen
While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- include

[Qemu-devel] [PATCH V4 00/16] COLO: integrate colo frame with block replication and COLO proxy

2018-01-19 Thread Zhang Chen
From: Zhang Chen Hi~ COLO Frame, block replication and COLO proxy(colo-compare,filter-mirror, filter-redirector,filter-rewriter) have been exist in qemu for long time, it's time to integrate these three parts to make COLO really works. In this series, we have some optimizations for COLO

Re: [Qemu-devel] [PATCH RESEND V3 10/16] qmp event: Add COLO_EXIT event to notify users while exited COLO

2018-01-07 Thread Zhang Chen
On Thu, Jan 4, 2018 at 5:10 PM, Eric Blake wrote: > On 01/04/2018 12:01 AM, Zhang Chen wrote: > > From: zhanghailiang > > > > If some errors happen during VM's COLO FT stage, it's important to > > notify the users of this event. Together with 'x_colo

[Qemu-devel] [PATCH RESEND V3 16/16] COLO: quick failover process by kick COLO thread

2018-01-03 Thread Zhang Chen
From: zhanghailiang COLO thread may sleep at qemu_sem_wait(&s->colo_checkpoint_sem), while failover works begin, It's better to wakeup it to quick the process. Signed-off-by: zhanghailiang --- migration/colo.c | 8 1 file changed, 8 insertions(+) diff --git a/migration/colo.c b/migra

[Qemu-devel] [PATCH RESEND V3 14/16] filter-rewriter: handle checkpoint and failover event

2018-01-03 Thread Zhang Chen
Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen --- migration/colo.c | 13 + net/filter-rewriter.c | 40 2 files changed, 53 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index a931ff2..9eab4a3 100644 --- a/migration

[Qemu-devel] [PATCH RESEND V3 12/16] COLO: flush host dirty ram from cache

2018-01-03 Thread Zhang Chen
From: zhanghailiang Don't need to flush all VM's ram from cache, only flush the dirty pages since last checkpoint Cc: Juan Quintela Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- migration/ram.c | 12 1 file changed, 12 insertion

[Qemu-devel] [PATCH RESEND V3 15/16] COLO: notify net filters about checkpoint/failover event

2018-01-03 Thread Zhang Chen
From: zhanghailiang Notify all net filters about the checkpoint and failover event. Cc: Jason Wang Signed-off-by: zhanghailiang --- migration/colo.c | 12 1 file changed, 12 insertions(+) diff --git a/migration/colo.c b/migration/colo.c index 9eab4a3..10bc80c 100644 --- a/migrat

[Qemu-devel] [PATCH RESEND V3 10/16] qmp event: Add COLO_EXIT event to notify users while exited COLO

2018-01-03 Thread Zhang Chen
x27;s failover verdict, it is still necessary to notify users that we exited COLO mode. Cc: Markus Armbruster Cc: Michael Roth Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Eric Blake --- migration/colo.c| 19 ++

[Qemu-devel] [PATCH RESEND V3 08/16] ram/COLO: Record the dirty pages that SVM received

2018-01-03 Thread Zhang Chen
We record the address of the dirty pages that received, it will help flushing pages that cached into SVM. Here, it is a trick, we record dirty pages by re-using migration dirty bitmap. In the later patch, we will start the dirty log for SVM, just like migration, in this way, we can record both the

[Qemu-devel] [PATCH RESEND V3 09/16] COLO: Flush memory data from ram cache

2018-01-03 Thread Zhang Chen
From: zhanghailiang During the time of VM's running, PVM may dirty some pages, we will transfer PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint time. So, the content of SVM's RAM cache will always be same with PVM's memory after checkpoint. Instead of flushing all

[Qemu-devel] [PATCH RESEND V3 13/16] filter: Add handle_event method for NetFilterClass

2018-01-03 Thread Zhang Chen
Filter needs to process the event of checkpoint/failover or other event passed by COLO frame. Cc: Jason Wang Signed-off-by: zhanghailiang --- include/net/filter.h | 5 + net/filter.c | 17 + net/net.c| 28 3 files changed, 50

[Qemu-devel] [PATCH RESEND V3 04/16] COLO: integrate colo compare with colo frame

2018-01-03 Thread Zhang Chen
Wang Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 42 -- migration/migration.c | 4 ++-- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/migration/colo.c b/migration

[Qemu-devel] [PATCH RESEND V3 06/16] COLO: Remove colo_state migration struct

2018-01-03 Thread Zhang Chen
From: zhanghailiang We need to know if migration is going into COLO state for incoming side before start normal migration. Instead by using the VMStateDescription to send colo_state from source side to destination side, we use MIG_CMD_ENABLE_COLO to indicate whether COLO is enabled or not. Sign

[Qemu-devel] [PATCH RESEND V3 11/16] savevm: split the process of different stages for loadvm/savevm

2018-01-03 Thread Zhang Chen
_main() and qemu_save_device_state() public, and simplify the codes of qemu_save_device_state() by calling the wrapper qemu_savevm_state_header(). Cc: Juan Quintela Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Reviewed-by: Dr. David Alan Gilbert --- migration/c

[Qemu-devel] [PATCH RESEND V3 07/16] COLO: Load dirty pages into SVM's RAM cache firstly

2018-01-03 Thread Zhang Chen
eive all PVM's state. Cc: Dr. David Alan Gilbert Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen --- include/exec/ram_addr.h | 1 + migration/migration.c | 2 + migration/ram.c | 97 +++-- mig

[Qemu-devel] [PATCH RESEND V3 02/16] colo-compare: implement the process of checkpoint

2018-01-03 Thread Zhang Chen
While do checkpoint, we need to flush all the unhandled packets, By using the filter notifier mechanism, we can easily to notify every compare object to do this process, which runs inside of compare threads as a coroutine. Cc: Jason Wang Signed-off-by: zhanghailiang Signed-off-by: Zhang Chen

[Qemu-devel] [PATCH RESEND V3 03/16] colo-compare: use notifier to notify packets comparing result

2018-01-03 Thread Zhang Chen
It's a good idea to use notifier to notify COLO frame of inconsistent packets comparing. Cc: Jason Wang Signed-off-by: Zhang Chen Signed-off-by: zhanghailiang --- net/colo-compare.c | 32 +--- net/colo-compare.h | 2 ++ 2 files changed, 27 insertions(

[Qemu-devel] [PATCH RESEND V3 05/16] COLO: Add block replication into colo process

2018-01-03 Thread Zhang Chen
Make sure master start block replication after slave's block replication started. Besides, we need to activate VM's blocks before goes into COLO state. Signed-off-by: zhanghailiang Signed-off-by: Li Zhijian Signed-off-by: Zhang Chen Cc: Stefan Hajnoczi Cc: Kevin Wolf Cc: Max Reit

[Qemu-devel] [PATCH RESEND V3 01/16] filter-rewriter: fix memory leak for connection in connection_track_table

2018-01-03 Thread Zhang Chen
After a net connection is closed, we didn't clear its releated resources in connection_track_table, which will lead to memory leak. Let't track the state of net connection, if it is closed, its related resources will be cleared up. Signed-off-by: zhanghailiang Signed-off-by:

[Qemu-devel] [PATCH RESEND V3 00/16] COLO: integrate colo frame with block replication and COLO proxy

2018-01-03 Thread Zhang Chen
plite shared disk part to indepentent patch set. - Optimize codes. Zhang Chen (8): filter-rewriter: fix memory leak for connection in connection_track_table colo-compare: implement the process of checkpoint colo-compare: use notifier to notify packets comparing result COLO: integrate

Re: [Qemu-devel] [PATCH v3 2/2] colo: compare the packet based on the tcp sequence number

2017-12-24 Thread Zhang Chen
|6|7|8|9|0| spkt1 > ---|\| >| \offset | > -v-v > | header |4|5|6|7|8|9|0| ppkt2 > > > In this way, the performance can reach native 20% in my multiple > tests.

Re: [Qemu-devel] [PATCH v3 1/2] colo: modified the payload compare function

2017-12-24 Thread Zhang Chen
On Fri, Dec 22, 2017 at 5:28 PM, Mao Zhongyi wrote: > Modified the function colo_packet_compare_common to prepare for the > tcp packet comparison in the next patch. > > Cc: Zhang Chen > Cc: Li Zhijian > Cc: Jason Wang > > Signed-off-by: Mao Zhongyi > Signed-off-b

Re: [Qemu-devel] [PATCH v2 3/4] colo: compare the packet based on the tcp sequence number

2017-12-13 Thread Zhang Chen
On Wed, Dec 13, 2017 at 6:25 AM, Mao Zhongyi wrote: > Hi, Chen > > On 12/12/2017 11:21 PM, Zhang Chen wrote: > >> >> >> On Wed, Dec 6, 2017 at 5:57 PM, Mao Zhongyi > <mailto:maozy.f...@cn.fujitsu.com>> wrote: >> >> The primary and secon

Re: [Qemu-devel] [PATCH v2 1/4] colo: remove the incorrect if conditions in colo_compare_packet_tcp()

2017-12-13 Thread Zhang Chen
On Wed, Dec 13, 2017 at 5:32 AM, Mao Zhongyi wrote: > Hi, Chen > > On 12/12/2017 11:19 PM, Zhang Chen wrote: > >> This patch should be moved behind the patch with payload comparition. >> > > Well, it is not impossible, but I think it is better to be here. Because

Re: [Qemu-devel] [PATCH v2 4/4] colo: add trace for the tcp packet comparison

2017-12-12 Thread Zhang Chen
On Wed, Dec 6, 2017 at 5:57 PM, Mao Zhongyi wrote: > Cc: Zhang Chen > Cc: Li Zhijian > Cc: Jason Wang > > Signed-off-by: Mao Zhongyi > --- > net/colo-compare.c | 16 > net/colo.c | 1 + > net/colo.h | 1 + > net/trace-events

Re: [Qemu-devel] [PATCH v2 3/4] colo: compare the packet based on the tcp sequence number

2017-12-12 Thread Zhang Chen
: > > | header |1|2|3| ppkt1 > -|-| > | | > -v-v-- > | header |1|2|3|4|5|6|7|8|9|0| spkt1 > ---|\| >| \offset | > -----v-v > | header |4|5|6|7|8|9|0| ppkt2 >

Re: [Qemu-devel] [PATCH v2 2/4] colo: modified the payload compare function

2017-12-12 Thread Zhang Chen
On Wed, Dec 6, 2017 at 5:57 PM, Mao Zhongyi wrote: > Modified the function colo_packet_compare_common to prepare for the > tcp packet comparison in the next patch. > > Cc: Zhang Chen > Cc: Li Zhijian > Cc: Jason Wang > > Signed-off-by: Mao Zhongyi > ---

Re: [Qemu-devel] [PATCH v2 1/4] colo: remove the incorrect if conditions in colo_compare_packet_tcp()

2017-12-12 Thread Zhang Chen
This patch should be moved behind the patch with payload comparition. People needs look your changes to understand why we need this patch. Thanks Zhang Chen On Wed, Dec 6, 2017 at 9:57 AM, Mao Zhongyi wrote: > The th_off field specifies the size of the TCP header, if th_off > 5, >

Re: [Qemu-devel] [PATCH 1/2] colo: compare the packet based on the tcp sequence number

2017-12-03 Thread Zhang Chen
On Mon, Dec 4, 2017 at 3:32 AM, Mao Zhongyi wrote: > > > On 12/04/2017 09:41 AM, Zhang Chen wrote: > >> >> >> On Tue, Nov 28, 2017 at 8:04 PM, Mao Zhongyi > <mailto:maozy.f...@cn.fujitsu.com>> wrote: >> >> The primary and secondary

Re: [Qemu-devel] [PATCH 2/2] colo: add trace for the tcp packet comparison

2017-12-03 Thread Zhang Chen
On Tue, Nov 28, 2017 at 8:04 PM, Mao Zhongyi wrote: > Cc: Zhang Chen > Cc: Li Zhijian > Cc: Jason Wang > > Signed-off-by: Mao Zhongyi > --- > net/colo-compare.c | 16 > net/colo.c | 1 + > net/colo.h | 1 + > net/trace-event

Re: [Qemu-devel] [PATCH 1/2] colo: compare the packet based on the tcp sequence number

2017-12-03 Thread Zhang Chen
| header |1|2|3| ppkt1 > -|-| > | | > -v-v-- > | header |1|2|3|4|5|6|7|8|9|0| spkt1 > ---|\| >| \offset | > -v-v > | header |4|5|6|7|8|9|0| ppkt2 > ---

Re: [Qemu-devel] [PATCHv5 03/03] colo-compare: Update the COLO document to add the IOThread configuration

2017-09-06 Thread Zhang Chen
On 08/29/2017 03:22 PM, Wang yong wrote: From: Wang Yong Update colo-proxy.txt,add IOThread configuration. Later we have to configure IOThread,if not COLO can not work. Signed-off-by: Wang Yong Signed-off-by: Wang Guang Reviewed-by: Zhang Chen --- docs/colo-proxy.txt | 3 ++- 1

Re: [Qemu-devel] [PATCHv5 02/03] colo-compare: Use IOThread to Check old packet regularly and Process pactkets of the primary

2017-09-06 Thread Zhang Chen
g_main_loop in the IOThread.then the packets from the primary and the secondary are processed in the IOThread. Finally remove the colo-compare thread using the IOThread instead. Signed-off-by: Wang Yong Signed-off-by: Wang Guang Reviewed-by: Zhang Chen Hi~ Jason. Have any comments to this series

[Qemu-devel] [PATCH V6 1/3] net/colo-compare.c: Optimize unpredictable tcp options comparison

2017-09-04 Thread Zhang Chen
When network is busy, some tcp options(like sack) will unpredictable occur in primary side or secondary side. it will make packet size not same, but the two packet's payload is identical. colo just care about packet payload, so we skip the option field. Signed-off-by: Zhang Chen --- net

[Qemu-devel] [PATCH V6 2/3] net/colo-compare.c: Adjust net queue pop order for performance

2017-09-04 Thread Zhang Chen
The packet_enqueue() use g_queue_push_tail() to enqueue net packet, so it is more efficent way use g_queue_pop_head() to get packet for compare. That will improve the success rate of comparison. In my test the performance of ftp put 1000M file will increase 10% Signed-off-by: Zhang Chen --- net

[Qemu-devel] [PATCH V6 3/3] net/colo-compare.c: Fix comments and scheme

2017-09-04 Thread Zhang Chen
Signed-off-by: Zhang Chen --- net/colo-compare.c | 59 -- 1 file changed, 31 insertions(+), 28 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index c31adc6..d5fcfea 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c

[Qemu-devel] [PATCH V6 0/3] Optimize COLO-compare performance

2017-09-04 Thread Zhang Chen
p2. - Move the "checkpoint_time_ms" to CompareState, in order to aviod multi colo-compare instance conflict. - Add "TODO comments" for reset s->checkpoint_time_ms. - Add a new patch fix comments and scheme. V2: - Rename p2's subject. Zhang Chen (3):

Re: [Qemu-devel] [PATCH V5 1/3] net/colo-compare.c: Optimize unpredictable tcp options comparison

2017-09-04 Thread Zhang Chen
On 09/05/2017 10:11 AM, Dou Liyang wrote: Hi Chen, At 09/04/2017 02:14 PM, Zhang Chen wrote: When network is busy, some tcp options(like sack) will unpredictable occur in primary side or secondary side. it will make packet size not same, but the two packet's payload is identical. colo

[Qemu-devel] [PATCH V5 1/3] net/colo-compare.c: Optimize unpredictable tcp options comparison

2017-09-03 Thread Zhang Chen
When network is busy, some tcp options(like sack) will unpredictable occur in primary side or secondary side. it will make packet size not same, but the two packet's payload is identical. colo just care about packet payload, so we skip the option field. Signed-off-by: Zhang Chen --- net

[Qemu-devel] [PATCH V5 3/3] net/colo-compare.c: Fix comments and scheme

2017-09-03 Thread Zhang Chen
Signed-off-by: Zhang Chen --- net/colo-compare.c | 59 -- 1 file changed, 31 insertions(+), 28 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 8dd76e4..c9a7174 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c

[Qemu-devel] [PATCH V5 0/3] Optimize COLO-compare performance

2017-09-03 Thread Zhang Chen
s" to CompareState, in order to aviod multi colo-compare instance conflict. - Add "TODO comments" for reset s->checkpoint_time_ms. - Add a new patch fix comments and scheme. V2: - Rename p2's subject. Zhang Chen (3): net/colo-compare.c: Optimize unpredictable tcp o

[Qemu-devel] [PATCH V5 2/3] net/colo-compare.c: Adjust net queue pop order for performance

2017-09-03 Thread Zhang Chen
The packet_enqueue() use g_queue_push_tail() to enqueue net packet, so it is more efficent way use g_queue_pop_head() to get packet for compare. That will improve the success rate of comparison. In my test the performance of ftp put 1000M file will increase 10% Signed-off-by: Zhang Chen --- net

<    5   6   7   8   9   10   11   12   13   14   >