Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Dr. David Alan Gilbert]

* Li Zhijian (lizhij...@cn.fujitsu.com) wrote:
> 
> 
> On 03/01/2016 04:04 AM, Dr. David Alan Gilbert wrote:
> >* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote:
> >>From: zhangchen 
> >>
> >>Hi,all
> >>
> >>This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
> >>based on qemu netfilter and it's a plugin for qemu netfilter. the function
> >>keep Secondary VM connect normal to Primary VM and compare packets
> >>sent by PVM to sent by SVM.if the packet difference,notify COLO do
> >>checkpoint and send all primary packet has queued.
> >
> >Hi Zhangchen,
> >   How are you dealing with the IP 'identification' field?
> >It's a very very random field, and not just the initial value in the 
> >connection.
> >I can't see how the kernel colo-proxy dealt with it either; but I think it's
> >comparison was after defragementation so probably ignored the identification
> >field
> You are right, most kernel colo-proxy code is working at mangle table (after 
> defrag).
> and colo proxy only compare the contents of L4(TCP/UDP) excluding IP 
> identification.
> 
> > - wouldn't that confuse a client at failover?
> Err..., instersting question.
> 
> for example, a COLO including primay(PVM) adn secondary(SVM)
> 1. primay is sending a compared P_IP packet(identification=0x12345, split to 
> IP_s1, IP_s2..IP_s100) to client
> 2. client is receiving the ip segment(but IP_s2, IP_s50, IP_s80..IP_s99 are 
> missing)
>and primary host is down.
> 3. secondary VM takeover, and send a S_IP packet(ip contents is always same 
> as at PVM)
> 
> in step 3,
> if the S_IP identification isn't 0x12345, client will drop the ip segment at 
> step 2 because of defrag timeout.

So that triggers a timeout (30 second?) - hmm OK, a bit slow but OK.

> if the S_IP identification isn 0x12345, client may mix the segment from PVM 
> or SVM(just like segment come
> from different router), but that's Okay, because we have ensured the ip 
> contents is identical.

Could the S_IP identification match a later/earlier fragment?

Dave

> so IMO, it will not confuse the client at failover.
> 
> >
> >Dave
> >
> >>You can also get the series from:
> >>
> >>https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
> >>
> >>Usage:
> >>
> >>primary:
> >>-netdev tap,id=bn0 -device e1000,netdev=bn0
> >>-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
> >>
> >>secondary:
> >>-netdev tap,id=bn0 -device e1000,netdev=bn0
> >>-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
> >>
> >>NOTE:
> >>queue must set "all". See enum NetFilterDirection for detail.
> >>colo-proxy need queue all packets
> >>colo-proxy V2 just can compare ip packet
> >>
> >>
> >>## Background
> >>
> >>COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop 
> >>Service)
> >>project is a high availability solution. Both Primary VM (PVM) and 
> >>Secondary VM
> >>(SVM) run in parallel. They receive the same request from client, and 
> >>generate
> >>responses in parallel too. If the response packets from PVM and SVM are
> >>identical, they are released immediately. Otherwise, a VM checkpoint (on
> >>demand)is conducted.
> >>
> >>Paper:
> >>http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
> >>
> >>COLO on Xen:
> >>http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> >>
> >>COLO on Qemu/KVM:
> >>http://wiki.qemu.org/Features/COLO
> >>
> >>By the needs of capturing response packets from PVM and SVM and finding out
> >>whether they are identical, we introduce a new module to qemu networking
> >>called colo-proxy.
> >>
> >>V2:
> >>   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
> >>   fix dave's comments
> >>   fix wency's comments
> >>   fix zhanghailiang's comments
> >>
> >>v1:
> >>   initial patch.
> >>
> >>
> >>
> >>zhangchen (10):
> >>   Init colo-proxy object based on netfilter
> >>   Jhash: add linux kernel jhashtable in qemu
> >>   Colo-proxy: add colo-proxy framework
> >>   Colo-proxy: add data structure and jhash func
> >>   net/colo-proxy: Add colo interface to use proxy
> >>   net/colo-proxy: add socket used by forward func
> >>   net/colo-proxy: Add packet enqueue & handle func
> >>   net/colo-proxy: Handle packet and connection
> >>   net/colo-proxy: Compare pri pkt to sec pkt
> >>   net/colo-proxy: Colo-proxy do checkpoint and clear
> >>
> >>  include/qemu/jhash.h |  61 
> >>  net/Makefile.objs|   1 +
> >>  net/colo-proxy.c | 939 
> >> +++
> >>  net/colo-proxy.h |  24 ++
> >>  qemu-options.hx  |   6 +
> >>  trace-events |   8 +
> >>  vl.c |   3 +-
> >>  7 files changed, 1041 insertions(+), 1 deletion(-)
> >>  create mode 100644 include/qemu/jhash.h
> >>  create mode 100644 net/colo-proxy.c
> >>  create mode 100644 net/colo-proxy.h
> >>
> >>--
> >>1.9.1
> >>
> >>
> >>
> >--
> >Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
> 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Li Zhijian]

On 03/01/2016 04:04 AM, Dr. David Alan Gilbert wrote:

* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.


Hi Zhangchen,
   How are you dealing with the IP 'identification' field?
It's a very very random field, and not just the initial value in the connection.
I can't see how the kernel colo-proxy dealt with it either; but I think it's
comparison was after defragementation so probably ignored the identification
field

You are right, most kernel colo-proxy code is working at mangle table (after 
defrag).
and colo proxy only compare the contents of L4(TCP/UDP) excluding IP 
identification.


 - wouldn't that confuse a client at failover?

Err..., instersting question.

for example, a COLO including primay(PVM) adn secondary(SVM)
1. primay is sending a compared P_IP packet(identification=0x12345, split to 
IP_s1, IP_s2..IP_s100) to client
2. client is receiving the ip segment(but IP_s2, IP_s50, IP_s80..IP_s99 are 
missing)
   and primary host is down.
3. secondary VM takeover, and send a S_IP packet(ip contents is always same as 
at PVM)

in step 3,
if the S_IP identification isn't 0x12345, client will drop the ip segment at 
step 2 because of defrag timeout.
if the S_IP identification isn 0x12345, client may mix the segment from PVM or 
SVM(just like segment come
from different router), but that's Okay, because we have ensured the ip 
contents is identical.

so IMO, it will not confuse the client at failover.



Dave


You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2

Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port

NOTE:
queue must set "all". See enum NetFilterDirection for detail.
colo-proxy need queue all packets
colo-proxy V2 just can compare ip packet


## Background

COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on
demand)is conducted.

Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0

COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking
called colo-proxy.

V2:
   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
   fix dave's comments
   fix wency's comments
   fix zhanghailiang's comments

v1:
   initial patch.



zhangchen (10):
   Init colo-proxy object based on netfilter
   Jhash: add linux kernel jhashtable in qemu
   Colo-proxy: add colo-proxy framework
   Colo-proxy: add data structure and jhash func
   net/colo-proxy: Add colo interface to use proxy
   net/colo-proxy: add socket used by forward func
   net/colo-proxy: Add packet enqueue & handle func
   net/colo-proxy: Handle packet and connection
   net/colo-proxy: Compare pri pkt to sec pkt
   net/colo-proxy: Colo-proxy do checkpoint and clear

  include/qemu/jhash.h |  61 
  net/Makefile.objs|   1 +
  net/colo-proxy.c | 939 +++
  net/colo-proxy.h |  24 ++
  qemu-options.hx  |   6 +
  trace-events |   8 +
  vl.c |   3 +-
  7 files changed, 1041 insertions(+), 1 deletion(-)
  create mode 100644 include/qemu/jhash.h
  create mode 100644 net/colo-proxy.c
  create mode 100644 net/colo-proxy.h

--
1.9.1




--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK


.



--
Best regards.
Li Zhijian (8555)

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Dr. David Alan Gilbert]

* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote:
> From: zhangchen 
> 
> Hi,all
> 
> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
> based on qemu netfilter and it's a plugin for qemu netfilter. the function
> keep Secondary VM connect normal to Primary VM and compare packets
> sent by PVM to sent by SVM.if the packet difference,notify COLO do
> checkpoint and send all primary packet has queued.

Hi Zhangchen,
  How are you dealing with the IP 'identification' field?
It's a very very random field, and not just the initial value in the connection.
I can't see how the kernel colo-proxy dealt with it either; but I think it's
comparison was after defragementation so probably ignored the identification
field - wouldn't that confuse a client at failover?

Dave

> You can also get the series from:
> 
> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
> 
> Usage:
> 
> primary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
> 
> secondary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port 
> 
> NOTE:
> queue must set "all". See enum NetFilterDirection for detail.
> colo-proxy need queue all packets
> colo-proxy V2 just can compare ip packet
> 
> 
> ## Background
> 
> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
> project is a high availability solution. Both Primary VM (PVM) and Secondary 
> VM
> (SVM) run in parallel. They receive the same request from client, and generate
> responses in parallel too. If the response packets from PVM and SVM are
> identical, they are released immediately. Otherwise, a VM checkpoint (on
> demand)is conducted.
> 
> Paper:
> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
> 
> COLO on Xen:
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> 
> COLO on Qemu/KVM:
> http://wiki.qemu.org/Features/COLO
> 
> By the needs of capturing response packets from PVM and SVM and finding out
> whether they are identical, we introduce a new module to qemu networking
> called colo-proxy.
> 
> V2:
>   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
>   fix dave's comments
>   fix wency's comments
>   fix zhanghailiang's comments
> 
> v1:
>   initial patch.
> 
> 
> 
> zhangchen (10):
>   Init colo-proxy object based on netfilter
>   Jhash: add linux kernel jhashtable in qemu
>   Colo-proxy: add colo-proxy framework
>   Colo-proxy: add data structure and jhash func
>   net/colo-proxy: Add colo interface to use proxy
>   net/colo-proxy: add socket used by forward func
>   net/colo-proxy: Add packet enqueue & handle func
>   net/colo-proxy: Handle packet and connection
>   net/colo-proxy: Compare pri pkt to sec pkt
>   net/colo-proxy: Colo-proxy do checkpoint and clear
> 
>  include/qemu/jhash.h |  61 
>  net/Makefile.objs|   1 +
>  net/colo-proxy.c | 939 
> +++
>  net/colo-proxy.h |  24 ++
>  qemu-options.hx  |   6 +
>  trace-events |   8 +
>  vl.c |   3 +-
>  7 files changed, 1041 insertions(+), 1 deletion(-)
>  create mode 100644 include/qemu/jhash.h
>  create mode 100644 net/colo-proxy.c
>  create mode 100644 net/colo-proxy.h
> 
> -- 
> 1.9.1
> 
> 
> 
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Eric Blake]

On 01/22/2016 12:46 AM, Wen Congyang wrote:
...

> On 01/20/2016 11:29 AM, Zhang Chen wrote:
> Sure.

Wow, that's a lot of wasted quoting.  Your mail weighed in at 24k, even
though...


>> Thanks. I think I get the point. So if there's a difference, primary
>> packet will only be sent after checkpoint and we could not assume the
>> checkpoint itself is reliable.
> 
> Yes.
> 
>>
>> Back to the filters design. We'd better still decouple packet comparing
>> out of netdev. Maybe a little bit more tweak on what you've suggested:
>>
>> -netdev tap,id=hn0
>> -object traffic-mirrorer,id=f0,netdev=hn0,queue=tx,outdev=mirrorer0
>> -object
>> traffic-redirector,id=f1,netdev=hn0,queue=rx,outdev=comparer0,indev=comparer2
>> -colo-comparer
>> primary_traffic=comparer0,secondary_traffic=comparer1,outdev=comparer2
>>
>> Just add one more socket for comparer for sending primary packet, and
>> let f1 redirector its output to netdev?
> 
> OK, I understand it now.
> Thanks for your suggestion.

...content-wise, you only added about 100 bytes.  It's okay to trim
replies down to relevant portions, to make it easier for readers to get
to the meat of your message.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/22/2016 11:15 AM, Jason Wang wrote:
> 
> 
> On 01/20/2016 06:30 PM, Wen Congyang wrote:
>> On 01/20/2016 06:19 PM, Jason Wang wrote:


 On 01/20/2016 06:01 PM, Wen Congyang wrote:
>> On 01/20/2016 02:54 PM, Jason Wang wrote:

 On 01/20/2016 11:29 AM, Zhang Chen wrote:
 Sure.

 Two main comments/suggestions:

 - TCP analysis is missed in current version, maybe you point a git 
 tree
 (or another version of RFC) to me for a better understanding of the
 design. (Just a skeleton for TCP should be sufficient to discuss).
 - I prefer to make the code as reusable as possible. So it's 
 better to
 split/decouple the reusable parts from the codes. So a vague idea 
 is:

 1) Decouple the packet comparing from the netfilter. You've 
 achieved
 this 99% since the work has been done in a thread. Just let the 
 thread
 poll sockets directly, then the comparing have the possibility to 
 be
 reused by other kinds of dataplane.
 2) Implement traffic mirror/redirector as filter.
 3) Implement TCP seq rewriting as a filter.

 Then, in primary node, you need just a traffic mirror, which did:
 - mirror ingress traffic to secondary node
 - mirror outgress traffic to packet comparing thread

 And in secondadry node, you need two filters:
 - A TCP seq rewriter which adjust tcp sequence number.
 - A traffic redirector which redirect packet from a socket as 
 ingress
 traffic, and redirect outgress traffic to the socket which could be
 polled by remote packet comparing thread.
   Thoughts?

 Thanks

>> Thanks
>> zhangchen
>>
>> Hi, Jason.
>> We consider your suggestion to split/decouple
>> the reusable parts from the codes.
>> Due to filter plugin are traversed one by one in order
>> we will split colo-proxy to three filters in each side.
>>
>> But in this plan,primary and secondary both have socket
>> server,startup is a problem.
 I believe this issue could be solved by reusing socket chardev.

>>
>>  Primary qemu  
>> Secondary qemu
>> +--+  
>> +---+
>> | +-+  |   | 
>> +--+ |
>> | | |  |   | 
>> |  | |
>> | |guest|  |   | 
>> |guest | |
>> | | |  |   | 
>> |  | |
>> | +---^--+--+  |   | 
>> +-++---+ |
>> | |  | |  
>> |^| |
>> | |  | |  
>> ||| |
>> | +-+ 
>> ||| |
>> |  netfilter  |  | ||  | 
>>  
>> netfilter|| |
>> | +-+  ||  | 
>> +--+ |
>> | |   |  | filter excute order  |  ||  | 
>> | ||  filter excute order  | |
>> | |   |  |+---> |  ||  | 
>> | || +---> | |
>> | |   |  |  |  ||  | 
>> | ||   TCP | |
>> | | +-+-+ +--v-+++ +-+  |  ||  | 
>> | +---+   +---++---v+rewriter+  ++ | |
>> | | |   | ||||  |  ||  | 
>> | |   |  

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/20/2016 06:30 PM, Wen Congyang wrote:
> On 01/20/2016 06:19 PM, Jason Wang wrote:
>> > 
>> > 
>> > On 01/20/2016 06:01 PM, Wen Congyang wrote:
>>> >> On 01/20/2016 02:54 PM, Jason Wang wrote:
 >>>
 >>> On 01/20/2016 11:29 AM, Zhang Chen wrote:
>> > Sure.
>> >
>> > Two main comments/suggestions:
>> >
>> > - TCP analysis is missed in current version, maybe you point a git 
>> > tree
>> > (or another version of RFC) to me for a better understanding of the
>> > design. (Just a skeleton for TCP should be sufficient to discuss).
>> > - I prefer to make the code as reusable as possible. So it's 
>> > better to
>> > split/decouple the reusable parts from the codes. So a vague idea 
>> > is:
>> >
>> > 1) Decouple the packet comparing from the netfilter. You've 
>> > achieved
>> > this 99% since the work has been done in a thread. Just let the 
>> > thread
>> > poll sockets directly, then the comparing have the possibility to 
>> > be
>> > reused by other kinds of dataplane.
>> > 2) Implement traffic mirror/redirector as filter.
>> > 3) Implement TCP seq rewriting as a filter.
>> >
>> > Then, in primary node, you need just a traffic mirror, which did:
>> > - mirror ingress traffic to secondary node
>> > - mirror outgress traffic to packet comparing thread
>> >
>> > And in secondadry node, you need two filters:
>> > - A TCP seq rewriter which adjust tcp sequence number.
>> > - A traffic redirector which redirect packet from a socket as 
>> > ingress
>> > traffic, and redirect outgress traffic to the socket which could be
>> > polled by remote packet comparing thread.
>> >   Thoughts?
>> >
>> > Thanks
>> >
>>> >> Thanks
>>> >> zhangchen
> 
>  Hi, Jason.
>  We consider your suggestion to split/decouple
>  the reusable parts from the codes.
>  Due to filter plugin are traversed one by one in order
>  we will split colo-proxy to three filters in each side.
> 
>  But in this plan,primary and secondary both have socket
>  server,startup is a problem.
 >>> I believe this issue could be solved by reusing socket chardev.
 >>>
> 
>   Primary qemu  
>  Secondary qemu
>  +--+  
>  +---+
>  | +-+  |   | 
>  +--+ |
>  | | |  |   | 
>  |  | |
>  | |guest|  |   | 
>  |guest | |
>  | | |  |   | 
>  |  | |
>  | +---^--+--+  |   | 
>  +-++---+ |
>  | |  | |  
>  |^| |
>  | |  | |  
>  ||| |
>  | +-+ 
>  ||| |
>  |  netfilter  |  | ||  | 
>   
>  netfilter|| |
>  | +-+  ||  | 
>  +--+ |
>  | |   |  | filter excute order  |  ||  | 
>  | ||  filter excute order  | |
>  | |   |  |+---> |  ||  | 
>  | || +---> | |
>  | |   |  |  |  ||  | 
>  | ||   TCP | |
>  | | +-+-+ +--v-+++ +-+  |  ||  | 
>  | +---+   +---++---v+rewriter+  ++ | |
>  | | |   | ||||  |  ||  | 
>  | |   |   || |  || | |

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/22/2016 01:56 PM, Wen Congyang wrote:
> On 01/22/2016 01:41 PM, Jason Wang wrote:
>> > 
>> > 
>> > On 01/22/2016 11:28 AM, Wen Congyang wrote:
>>> >> On 01/22/2016 11:15 AM, Jason Wang wrote:
 >>>
 >>> On 01/20/2016 06:30 PM, Wen Congyang wrote:
>  On 01/20/2016 06:19 PM, Jason Wang wrote:
>>> >>
>>> >> On 01/20/2016 06:01 PM, Wen Congyang wrote:
>  On 01/20/2016 02:54 PM, Jason Wang wrote:
>>> >> On 01/20/2016 11:29 AM, Zhang Chen wrote:
>>> >> Sure.
>>> >>
>>> >> Two main comments/suggestions:
>>> >>
>>> >> - TCP analysis is missed in current version, 
>>> >> maybe you point a git tree
>>> >> (or another version of RFC) to me for a better 
>>> >> understanding of the
>>> >> design. (Just a skeleton for TCP should be 
>>> >> sufficient to discuss).
>>> >> - I prefer to make the code as reusable as 
>>> >> possible. So it's better to
>>> >> split/decouple the reusable parts from the 
>>> >> codes. So a vague idea is:
>>> >>
>>> >> 1) Decouple the packet comparing from the 
>>> >> netfilter. You've achieved
>>> >> this 99% since the work has been done in a 
>>> >> thread. Just let the thread
>>> >> poll sockets directly, then the comparing have 
>>> >> the possibility to be
>>> >> reused by other kinds of dataplane.
>>> >> 2) Implement traffic mirror/redirector as filter.
>>> >> 3) Implement TCP seq rewriting as a filter.
>>> >>
>>> >> Then, in primary node, you need just a traffic 
>>> >> mirror, which did:
>>> >> - mirror ingress traffic to secondary node
>>> >> - mirror outgress traffic to packet comparing 
>>> >> thread
>>> >>
>>> >> And in secondadry node, you need two filters:
>>> >> - A TCP seq rewriter which adjust tcp sequence 
>>> >> number.
>>> >> - A traffic redirector which redirect packet 
>>> >> from a socket as ingress
>>> >> traffic, and redirect outgress traffic to the 
>>> >> socket which could be
>>> >> polled by remote packet comparing thread.
>>> >>   Thoughts?
>>> >>
>>> >> Thanks
>>> >>
>  Thanks
>  zhangchen
>  Hi, Jason.
>  We consider your suggestion to split/decouple
>  the reusable parts from the codes.
>  Due to filter plugin are traversed one by one in 
>  order
>  we will split colo-proxy to three filters in each 
>  side.
> 
>  But in this plan,primary and secondary both have 
>  socket
>  server,startup is a problem.
>>> >> I believe this issue could be solved by reusing socket 
>>> >> chardev.
>>> >>
>   Primary qemu
>    
>  Secondary qemu
>  +--+
>    
>  +---+
>  | 
>  +-+
>    |   | 
>  +--+
>   |
>  | |  
> |  |   | 
>  |
>    | |
>  | |guest 
> |  |   | 
>  |guest   
>    | |
> 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/22/2016 02:21 PM, Jason Wang wrote:
> 
> 
> On 01/22/2016 01:56 PM, Wen Congyang wrote:
>> On 01/22/2016 01:41 PM, Jason Wang wrote:


 On 01/22/2016 11:28 AM, Wen Congyang wrote:
>> On 01/22/2016 11:15 AM, Jason Wang wrote:

 On 01/20/2016 06:30 PM, Wen Congyang wrote:
>> On 01/20/2016 06:19 PM, Jason Wang wrote:
>>
>> On 01/20/2016 06:01 PM, Wen Congyang wrote:
>> On 01/20/2016 02:54 PM, Jason Wang wrote:
>> On 01/20/2016 11:29 AM, Zhang Chen wrote:
>> Sure.
>>
>> Two main comments/suggestions:
>>
>> - TCP analysis is missed in current version, 
>> maybe you point a git tree
>> (or another version of RFC) to me for a better 
>> understanding of the
>> design. (Just a skeleton for TCP should be 
>> sufficient to discuss).
>> - I prefer to make the code as reusable as 
>> possible. So it's better to
>> split/decouple the reusable parts from the 
>> codes. So a vague idea is:
>>
>> 1) Decouple the packet comparing from the 
>> netfilter. You've achieved
>> this 99% since the work has been done in a 
>> thread. Just let the thread
>> poll sockets directly, then the comparing have 
>> the possibility to be
>> reused by other kinds of dataplane.
>> 2) Implement traffic mirror/redirector as filter.
>> 3) Implement TCP seq rewriting as a filter.
>>
>> Then, in primary node, you need just a traffic 
>> mirror, which did:
>> - mirror ingress traffic to secondary node
>> - mirror outgress traffic to packet comparing 
>> thread
>>
>> And in secondadry node, you need two filters:
>> - A TCP seq rewriter which adjust tcp sequence 
>> number.
>> - A traffic redirector which redirect packet 
>> from a socket as ingress
>> traffic, and redirect outgress traffic to the 
>> socket which could be
>> polled by remote packet comparing thread.
>>   Thoughts?
>>
>> Thanks
>>
>> Thanks
>> zhangchen
>> Hi, Jason.
>> We consider your suggestion to split/decouple
>> the reusable parts from the codes.
>> Due to filter plugin are traversed one by one in 
>> order
>> we will split colo-proxy to three filters in each 
>> side.
>>
>> But in this plan,primary and secondary both have 
>> socket
>> server,startup is a problem.
>> I believe this issue could be solved by reusing socket 
>> chardev.
>>
>>  Primary qemu
>>   
>> Secondary qemu
>> +--+
>>   
>> +---+
>> | 
>> +-+
>>   |   | 
>> +--+
>>  |
>> | |  
>>|  |   | 
>> |
>>   | |
>> | |guest 
>>|  |   | 
>> |guest   

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/20/2016 06:34 PM, Wen Congyang wrote:
> On 01/20/2016 06:03 PM, Jason Wang wrote:
>>
>> On 01/20/2016 05:49 PM, Wen Congyang wrote:
>>> On 01/20/2016 05:20 PM, Jason Wang wrote:
 On 01/20/2016 03:44 PM, Wen Congyang wrote:
>>> ...
>>> -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait
>>> -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait
>>> -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait
>>> -netdev tap,id=hn0
>>> -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0
>>> -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1
>>> ...
>>>
>>> packet comparer compares the packets from two chardev: comparer0 and
>>> comparer1.
>>> traffic-mirrorer mirror tx to secondary node through chardev mirrorer0,
>>> and mirror rx to packet comparer through chardev comparer0.
>>>
>>> In secondary node:
>>>
>>> ...
>>> -chardev socket,id=redirector0,host=ip_primary,port=Y
>>> -chardev socket,id=redirector1,host=ip_primary,port=Z
>>> -netdev tap,id=hn0
>>> -traffic-redirector 
>>> netdev=hn0,id,r0,indev=redirector0,outdev=redirector1
>>> -colo-rewriter netdev=hn0,id=c0
>>> ...
>>>
>>> traffic-redirector redirect the rx traffic from primary node through
>>> redirector0 and redirect the tx traffic to promary node through 
>>> redirector1.
>>> colo-rewriter rewrite seq number as a normal netfilter.
> What are traffic-mirrorer and colo-comparer, traffic-redirector, 
> colo-rewriter?
> A netfilter driver?
 traffic-mirrorer/redirector is a type of netfilter that just
 mirror/redirect packets between netdev and chardev (just the mirror
 client/sever and redirect client/sever in the above graph)
 colo-rewriter is a type of netfilter that did ack/seq adjust (just the
 TCP rewriter in the above graph)
 colo-comparer is a thread object that did packet comparing (similar to
 "compare" in the above graph but not a netfiler)
>>> Thanks. I have another question:
>>> IIRC, both rx and tx packets walk through all netfilter objects in the same 
>>> order.
>>>
>>> tx packet(sent to the guest): we want that redirector hanldes it first
>>> rx packet(sent from the guest): we want that colo-rewriter handles it first
>>> Change the order or use two traffic-redirectors?
>>>
>>> Thanks
>>> Wen Congyang
>> Interesting question.
>>
>> Two redirectors sounds ok or maybe we can go through rx filters in a
>> reverse order?
> netdev <---> filter1 <> filter2 <>  <> emulated device <> 
> guest
> So I think we can go through rx filters in a reverse order. But it changes
> the behavior. So I am not sure if we can do it.

I think we can. Both dump and buffer filter does not require strict
order, so it's a good time and change to do this.

>
> Thanks
> Wen Congyang
>
>>
>> .
>>
>
>

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/22/2016 01:41 PM, Jason Wang wrote:
> 
> 
> On 01/22/2016 11:28 AM, Wen Congyang wrote:
>> On 01/22/2016 11:15 AM, Jason Wang wrote:
>>>
>>> On 01/20/2016 06:30 PM, Wen Congyang wrote:
 On 01/20/2016 06:19 PM, Jason Wang wrote:
>>
>> On 01/20/2016 06:01 PM, Wen Congyang wrote:
 On 01/20/2016 02:54 PM, Jason Wang wrote:
>> On 01/20/2016 11:29 AM, Zhang Chen wrote:
>> Sure.
>>
>> Two main comments/suggestions:
>>
>> - TCP analysis is missed in current version, maybe you point a 
>> git tree
>> (or another version of RFC) to me for a better understanding of 
>> the
>> design. (Just a skeleton for TCP should be sufficient to 
>> discuss).
>> - I prefer to make the code as reusable as possible. So it's 
>> better to
>> split/decouple the reusable parts from the codes. So a vague 
>> idea is:
>>
>> 1) Decouple the packet comparing from the netfilter. You've 
>> achieved
>> this 99% since the work has been done in a thread. Just let the 
>> thread
>> poll sockets directly, then the comparing have the possibility 
>> to be
>> reused by other kinds of dataplane.
>> 2) Implement traffic mirror/redirector as filter.
>> 3) Implement TCP seq rewriting as a filter.
>>
>> Then, in primary node, you need just a traffic mirror, which did:
>> - mirror ingress traffic to secondary node
>> - mirror outgress traffic to packet comparing thread
>>
>> And in secondadry node, you need two filters:
>> - A TCP seq rewriter which adjust tcp sequence number.
>> - A traffic redirector which redirect packet from a socket as 
>> ingress
>> traffic, and redirect outgress traffic to the socket which could 
>> be
>> polled by remote packet comparing thread.
>>   Thoughts?
>>
>> Thanks
>>
 Thanks
 zhangchen
 Hi, Jason.
 We consider your suggestion to split/decouple
 the reusable parts from the codes.
 Due to filter plugin are traversed one by one in order
 we will split colo-proxy to three filters in each side.

 But in this plan,primary and secondary both have socket
 server,startup is a problem.
>> I believe this issue could be solved by reusing socket chardev.
>>
  Primary qemu  
 Secondary qemu
 +--+  
 +---+
 | +-+  |   
 | 
 +--+ |
 | | |  |   
 | 
 |  | |
 | |guest|  |   
 | 
 |guest | |
 | | |  |   
 | 
 |  | |
 | +---^--+--+  |   
 | 
 +-++---+ |
 | |  | |  
 |^| |
 | |  | |  
 ||| |
 | +-+ 
 ||| |
 |  netfilter  |  | ||  
 |  
 netfilter|| |
 | +-+  ||  
 | 
 +--+ |
 | |   |  | filter excute order  |  ||  
 | 
 | ||  filter excute order  | |
 | |   |  |+---> |  ||  
 | 
 | || +---> | |
 | |

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/22/2016 01:33 PM, Jason Wang wrote:
> 
> 
> On 01/20/2016 06:34 PM, Wen Congyang wrote:
>> On 01/20/2016 06:03 PM, Jason Wang wrote:
>>>
>>> On 01/20/2016 05:49 PM, Wen Congyang wrote:
 On 01/20/2016 05:20 PM, Jason Wang wrote:
> On 01/20/2016 03:44 PM, Wen Congyang wrote:
 ...
 -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait
 -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait
 -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait
 -netdev tap,id=hn0
 -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0
 -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1
 ...

 packet comparer compares the packets from two chardev: comparer0 and
 comparer1.
 traffic-mirrorer mirror tx to secondary node through chardev mirrorer0,
 and mirror rx to packet comparer through chardev comparer0.

 In secondary node:

 ...
 -chardev socket,id=redirector0,host=ip_primary,port=Y
 -chardev socket,id=redirector1,host=ip_primary,port=Z
 -netdev tap,id=hn0
 -traffic-redirector 
 netdev=hn0,id,r0,indev=redirector0,outdev=redirector1
 -colo-rewriter netdev=hn0,id=c0
 ...

 traffic-redirector redirect the rx traffic from primary node through
 redirector0 and redirect the tx traffic to promary node through 
 redirector1.
 colo-rewriter rewrite seq number as a normal netfilter.
>> What are traffic-mirrorer and colo-comparer, traffic-redirector, 
>> colo-rewriter?
>> A netfilter driver?
> traffic-mirrorer/redirector is a type of netfilter that just
> mirror/redirect packets between netdev and chardev (just the mirror
> client/sever and redirect client/sever in the above graph)
> colo-rewriter is a type of netfilter that did ack/seq adjust (just the
> TCP rewriter in the above graph)
> colo-comparer is a thread object that did packet comparing (similar to
> "compare" in the above graph but not a netfiler)
 Thanks. I have another question:
 IIRC, both rx and tx packets walk through all netfilter objects in the 
 same order.

 tx packet(sent to the guest): we want that redirector hanldes it first
 rx packet(sent from the guest): we want that colo-rewriter handles it first
 Change the order or use two traffic-redirectors?

 Thanks
 Wen Congyang
>>> Interesting question.
>>>
>>> Two redirectors sounds ok or maybe we can go through rx filters in a
>>> reverse order?
>> netdev <---> filter1 <> filter2 <>  <> emulated device 
>> <> guest
>> So I think we can go through rx filters in a reverse order. But it changes
>> the behavior. So I am not sure if we can do it.
> 
> I think we can. Both dump and buffer filter does not require strict
> order, so it's a good time and change to do this.

OK, we will do it.

Thanks
Wen Congyang

> 
>>
>> Thanks
>> Wen Congyang
>>
>>>
>>> .
>>>
>>
>>
> 
> 
> 
> .
> 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/22/2016 02:47 PM, Wen Congyang wrote:
> On 01/22/2016 02:21 PM, Jason Wang wrote:
>>
>> On 01/22/2016 01:56 PM, Wen Congyang wrote:
>>> On 01/22/2016 01:41 PM, Jason Wang wrote:
>
> On 01/22/2016 11:28 AM, Wen Congyang wrote:
>>> On 01/22/2016 11:15 AM, Jason Wang wrote:
> On 01/20/2016 06:30 PM, Wen Congyang wrote:
>>> On 01/20/2016 06:19 PM, Jason Wang wrote:
>>> On 01/20/2016 06:01 PM, Wen Congyang wrote:
>>> On 01/20/2016 02:54 PM, Jason Wang wrote:
>>> On 01/20/2016 11:29 AM, Zhang Chen wrote:
>>> Sure.
>>>
>>> Two main comments/suggestions:
>>>
>>> - TCP analysis is missed in current version, 
>>> maybe you point a git tree
>>> (or another version of RFC) to me for a better 
>>> understanding of the
>>> design. (Just a skeleton for TCP should be 
>>> sufficient to discuss).
>>> - I prefer to make the code as reusable as 
>>> possible. So it's better to
>>> split/decouple the reusable parts from the 
>>> codes. So a vague idea is:
>>>
>>> 1) Decouple the packet comparing from the 
>>> netfilter. You've achieved
>>> this 99% since the work has been done in a 
>>> thread. Just let the thread
>>> poll sockets directly, then the comparing have 
>>> the possibility to be
>>> reused by other kinds of dataplane.
>>> 2) Implement traffic mirror/redirector as 
>>> filter.
>>> 3) Implement TCP seq rewriting as a filter.
>>>
>>> Then, in primary node, you need just a traffic 
>>> mirror, which did:
>>> - mirror ingress traffic to secondary node
>>> - mirror outgress traffic to packet comparing 
>>> thread
>>>
>>> And in secondadry node, you need two filters:
>>> - A TCP seq rewriter which adjust tcp sequence 
>>> number.
>>> - A traffic redirector which redirect packet 
>>> from a socket as ingress
>>> traffic, and redirect outgress traffic to the 
>>> socket which could be
>>> polled by remote packet comparing thread.
>>>   Thoughts?
>>>
>>> Thanks
>>>
>>> Thanks
>>> zhangchen
>>> Hi, Jason.
>>> We consider your suggestion to split/decouple
>>> the reusable parts from the codes.
>>> Due to filter plugin are traversed one by one in 
>>> order
>>> we will split colo-proxy to three filters in each 
>>> side.
>>>
>>> But in this plan,primary and secondary both have 
>>> socket
>>> server,startup is a problem.
>>> I believe this issue could be solved by reusing socket 
>>> chardev.
>>>
>>>  Primary qemu   
>>>
>>> Secondary qemu
>>> +--+
>>>   
>>> +---+
>>> | 
>>> +-+
>>>   |   | 
>>> +--+
>>>  |
>>> | | 
>>> |  |   | 
>>> |   
>>>| |
>>> | |guest

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/22/2016 11:28 AM, Wen Congyang wrote:
> On 01/22/2016 11:15 AM, Jason Wang wrote:
>>
>> On 01/20/2016 06:30 PM, Wen Congyang wrote:
>>> On 01/20/2016 06:19 PM, Jason Wang wrote:
>
> On 01/20/2016 06:01 PM, Wen Congyang wrote:
>>> On 01/20/2016 02:54 PM, Jason Wang wrote:
> On 01/20/2016 11:29 AM, Zhang Chen wrote:
> Sure.
>
> Two main comments/suggestions:
>
> - TCP analysis is missed in current version, maybe you point a 
> git tree
> (or another version of RFC) to me for a better understanding of 
> the
> design. (Just a skeleton for TCP should be sufficient to discuss).
> - I prefer to make the code as reusable as possible. So it's 
> better to
> split/decouple the reusable parts from the codes. So a vague idea 
> is:
>
> 1) Decouple the packet comparing from the netfilter. You've 
> achieved
> this 99% since the work has been done in a thread. Just let the 
> thread
> poll sockets directly, then the comparing have the possibility to 
> be
> reused by other kinds of dataplane.
> 2) Implement traffic mirror/redirector as filter.
> 3) Implement TCP seq rewriting as a filter.
>
> Then, in primary node, you need just a traffic mirror, which did:
> - mirror ingress traffic to secondary node
> - mirror outgress traffic to packet comparing thread
>
> And in secondadry node, you need two filters:
> - A TCP seq rewriter which adjust tcp sequence number.
> - A traffic redirector which redirect packet from a socket as 
> ingress
> traffic, and redirect outgress traffic to the socket which could 
> be
> polled by remote packet comparing thread.
>   Thoughts?
>
> Thanks
>
>>> Thanks
>>> zhangchen
>>> Hi, Jason.
>>> We consider your suggestion to split/decouple
>>> the reusable parts from the codes.
>>> Due to filter plugin are traversed one by one in order
>>> we will split colo-proxy to three filters in each side.
>>>
>>> But in this plan,primary and secondary both have socket
>>> server,startup is a problem.
> I believe this issue could be solved by reusing socket chardev.
>
>>>  Primary qemu  
>>> Secondary qemu
>>> +--+  
>>> +---+
>>> | +-+  |   
>>> | 
>>> +--+ |
>>> | | |  |   
>>> | 
>>> |  | |
>>> | |guest|  |   
>>> | 
>>> |guest | |
>>> | | |  |   
>>> | 
>>> |  | |
>>> | +---^--+--+  |   
>>> | 
>>> +-++---+ |
>>> | |  | |  
>>> |^| |
>>> | |  | |  
>>> ||| |
>>> | +-+ 
>>> ||| |
>>> |  netfilter  |  | ||  
>>> |  
>>> netfilter|| |
>>> | +-+  ||  
>>> | 
>>> +--+ |
>>> | |   |  | filter excute order  |  ||  
>>> | 
>>> | ||  filter excute order  | |
>>> | |   |  |+---> |  ||  
>>> | 
>>> | || +---> | |
>>> | |   |  |  |  ||  
>>> | 
>>> | ||   TCP | |
>>> | | 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/22/2016 03:42 PM, Jason Wang wrote:
> 
> 
> On 01/22/2016 02:47 PM, Wen Congyang wrote:
>> On 01/22/2016 02:21 PM, Jason Wang wrote:
>>>
>>> On 01/22/2016 01:56 PM, Wen Congyang wrote:
 On 01/22/2016 01:41 PM, Jason Wang wrote:
>>
>> On 01/22/2016 11:28 AM, Wen Congyang wrote:
 On 01/22/2016 11:15 AM, Jason Wang wrote:
>> On 01/20/2016 06:30 PM, Wen Congyang wrote:
 On 01/20/2016 06:19 PM, Jason Wang wrote:
 On 01/20/2016 06:01 PM, Wen Congyang wrote:
 On 01/20/2016 02:54 PM, Jason Wang wrote:
 On 01/20/2016 11:29 AM, Zhang Chen wrote:
 Sure.

 Two main comments/suggestions:

 - TCP analysis is missed in current version, 
 maybe you point a git tree
 (or another version of RFC) to me for a better 
 understanding of the
 design. (Just a skeleton for TCP should be 
 sufficient to discuss).
 - I prefer to make the code as reusable as 
 possible. So it's better to
 split/decouple the reusable parts from the 
 codes. So a vague idea is:

 1) Decouple the packet comparing from the 
 netfilter. You've achieved
 this 99% since the work has been done in a 
 thread. Just let the thread
 poll sockets directly, then the comparing have 
 the possibility to be
 reused by other kinds of dataplane.
 2) Implement traffic mirror/redirector as 
 filter.
 3) Implement TCP seq rewriting as a filter.

 Then, in primary node, you need just a traffic 
 mirror, which did:
 - mirror ingress traffic to secondary node
 - mirror outgress traffic to packet comparing 
 thread

 And in secondadry node, you need two filters:
 - A TCP seq rewriter which adjust tcp sequence 
 number.
 - A traffic redirector which redirect packet 
 from a socket as ingress
 traffic, and redirect outgress traffic to the 
 socket which could be
 polled by remote packet comparing thread.
   Thoughts?

 Thanks

 Thanks
 zhangchen
 Hi, Jason.
 We consider your suggestion to split/decouple
 the reusable parts from the codes.
 Due to filter plugin are traversed one by one in 
 order
 we will split colo-proxy to three filters in each 
 side.

 But in this plan,primary and secondary both have 
 socket
 server,startup is a problem.
 I believe this issue could be solved by reusing socket 
 chardev.

  Primary qemu  
 
 Secondary qemu
 +--+
   
 +---+
 | 
 +-+
   |   | 
 +--+
  |
 | |
  |  |   | 
 |  

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/20/2016 03:44 PM, Wen Congyang wrote:
>> > 
>> > ...
>> > -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait
>> > -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait
>> > -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait
>> > -netdev tap,id=hn0
>> > -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0
>> > -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1
>> > ...
>> > 
>> > packet comparer compares the packets from two chardev: comparer0 and
>> > comparer1.
>> > traffic-mirrorer mirror tx to secondary node through chardev mirrorer0,
>> > and mirror rx to packet comparer through chardev comparer0.
>> > 
>> > In secondary node:
>> > 
>> > ...
>> > -chardev socket,id=redirector0,host=ip_primary,port=Y
>> > -chardev socket,id=redirector1,host=ip_primary,port=Z
>> > -netdev tap,id=hn0
>> > -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1
>> > -colo-rewriter netdev=hn0,id=c0
>> > ...
>> > 
>> > traffic-redirector redirect the rx traffic from primary node through
>> > redirector0 and redirect the tx traffic to promary node through 
>> > redirector1.
>> > colo-rewriter rewrite seq number as a normal netfilter.
> What are traffic-mirrorer and colo-comparer, traffic-redirector, 
> colo-rewriter?
> A netfilter driver?

traffic-mirrorer/redirector is a type of netfilter that just
mirror/redirect packets between netdev and chardev (just the mirror
client/sever and redirect client/sever in the above graph)
colo-rewriter is a type of netfilter that did ack/seq adjust (just the
TCP rewriter in the above graph)
colo-comparer is a thread object that did packet comparing (similar to
"compare" in the above graph but not a netfiler)

>
> If not, how to get the packet from the netdev, and send back the packet to
> the netdev?
>
> Thanks
> Wen Congyang
>

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/20/2016 05:20 PM, Jason Wang wrote:
> 
> 
> On 01/20/2016 03:44 PM, Wen Congyang wrote:

 ...
 -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait
 -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait
 -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait
 -netdev tap,id=hn0
 -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0
 -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1
 ...

 packet comparer compares the packets from two chardev: comparer0 and
 comparer1.
 traffic-mirrorer mirror tx to secondary node through chardev mirrorer0,
 and mirror rx to packet comparer through chardev comparer0.

 In secondary node:

 ...
 -chardev socket,id=redirector0,host=ip_primary,port=Y
 -chardev socket,id=redirector1,host=ip_primary,port=Z
 -netdev tap,id=hn0
 -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1
 -colo-rewriter netdev=hn0,id=c0
 ...

 traffic-redirector redirect the rx traffic from primary node through
 redirector0 and redirect the tx traffic to promary node through 
 redirector1.
 colo-rewriter rewrite seq number as a normal netfilter.
>> What are traffic-mirrorer and colo-comparer, traffic-redirector, 
>> colo-rewriter?
>> A netfilter driver?
> 
> traffic-mirrorer/redirector is a type of netfilter that just
> mirror/redirect packets between netdev and chardev (just the mirror
> client/sever and redirect client/sever in the above graph)
> colo-rewriter is a type of netfilter that did ack/seq adjust (just the
> TCP rewriter in the above graph)
> colo-comparer is a thread object that did packet comparing (similar to
> "compare" in the above graph but not a netfiler)

Thanks. I have another question:
IIRC, both rx and tx packets walk through all netfilter objects in the same 
order.

tx packet(sent to the guest): we want that redirector hanldes it first
rx packet(sent from the guest): we want that colo-rewriter handles it first
Change the order or use two traffic-redirectors?

Thanks
Wen Congyang

> 
>>
>> If not, how to get the packet from the netdev, and send back the packet to
>> the netdev?
>>
>> Thanks
>> Wen Congyang
>>
> 
> 
> 
> .
> 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/20/2016 05:49 PM, Wen Congyang wrote:
> On 01/20/2016 05:20 PM, Jason Wang wrote:
>>
>> On 01/20/2016 03:44 PM, Wen Congyang wrote:
> ...
> -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait
> -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait
> -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait
> -netdev tap,id=hn0
> -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0
> -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1
> ...
>
> packet comparer compares the packets from two chardev: comparer0 and
> comparer1.
> traffic-mirrorer mirror tx to secondary node through chardev mirrorer0,
> and mirror rx to packet comparer through chardev comparer0.
>
> In secondary node:
>
> ...
> -chardev socket,id=redirector0,host=ip_primary,port=Y
> -chardev socket,id=redirector1,host=ip_primary,port=Z
> -netdev tap,id=hn0
> -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1
> -colo-rewriter netdev=hn0,id=c0
> ...
>
> traffic-redirector redirect the rx traffic from primary node through
> redirector0 and redirect the tx traffic to promary node through 
> redirector1.
> colo-rewriter rewrite seq number as a normal netfilter.
>>> What are traffic-mirrorer and colo-comparer, traffic-redirector, 
>>> colo-rewriter?
>>> A netfilter driver?
>> traffic-mirrorer/redirector is a type of netfilter that just
>> mirror/redirect packets between netdev and chardev (just the mirror
>> client/sever and redirect client/sever in the above graph)
>> colo-rewriter is a type of netfilter that did ack/seq adjust (just the
>> TCP rewriter in the above graph)
>> colo-comparer is a thread object that did packet comparing (similar to
>> "compare" in the above graph but not a netfiler)
> Thanks. I have another question:
> IIRC, both rx and tx packets walk through all netfilter objects in the same 
> order.
>
> tx packet(sent to the guest): we want that redirector hanldes it first
> rx packet(sent from the guest): we want that colo-rewriter handles it first
> Change the order or use two traffic-redirectors?
>
> Thanks
> Wen Congyang

Interesting question.

Two redirectors sounds ok or maybe we can go through rx filters in a
reverse order?

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/20/2016 02:54 PM, Jason Wang wrote:
> 
> 
> On 01/20/2016 11:29 AM, Zhang Chen wrote:
>>
>>> Sure.
>>>
>>> Two main comments/suggestions:
>>>
>>> - TCP analysis is missed in current version, maybe you point a git tree
>>> (or another version of RFC) to me for a better understanding of the
>>> design. (Just a skeleton for TCP should be sufficient to discuss).
>>> - I prefer to make the code as reusable as possible. So it's better to
>>> split/decouple the reusable parts from the codes. So a vague idea is:
>>>
>>> 1) Decouple the packet comparing from the netfilter. You've achieved
>>> this 99% since the work has been done in a thread. Just let the thread
>>> poll sockets directly, then the comparing have the possibility to be
>>> reused by other kinds of dataplane.
>>> 2) Implement traffic mirror/redirector as filter.
>>> 3) Implement TCP seq rewriting as a filter.
>>>
>>> Then, in primary node, you need just a traffic mirror, which did:
>>> - mirror ingress traffic to secondary node
>>> - mirror outgress traffic to packet comparing thread
>>>
>>> And in secondadry node, you need two filters:
>>> - A TCP seq rewriter which adjust tcp sequence number.
>>> - A traffic redirector which redirect packet from a socket as ingress
>>> traffic, and redirect outgress traffic to the socket which could be
>>> polled by remote packet comparing thread.
>>>   Thoughts?
>>>
>>> Thanks
>>>
 Thanks
 zhangchen
>>>
>>
>>
>> Hi, Jason.
>> We consider your suggestion to split/decouple
>> the reusable parts from the codes.
>> Due to filter plugin are traversed one by one in order
>> we will split colo-proxy to three filters in each side.
>>
>> But in this plan,primary and secondary both have socket
>> server,startup is a problem.
> 
> I believe this issue could be solved by reusing socket chardev.
> 
>>
>>
>>  Primary qemu  
>> Secondary qemu
>> +--+  
>> +---+
>> | +-+  |   | 
>> +--+ |
>> | | |  |   | 
>> |  | |
>> | |guest|  |   | 
>> |guest | |
>> | | |  |   | 
>> |  | |
>> | +---^--+--+  |   | 
>> +-++---+ |
>> | |  | |  
>> |^| |
>> | |  | |  
>> ||| |
>> | +-+ 
>> ||| |
>> |  netfilter  |  | ||  |  
>> netfilter|| |
>> | +-+  ||  | 
>> +--+ |
>> | |   |  | filter excute order  |  ||  | 
>> | ||  filter excute order  | |
>> | |   |  |+---> |  ||  | 
>> | || +---> | |
>> | |   |  |  |  ||  | 
>> | ||   TCP | |
>> | | +-+-+ +--v-+++ +-+  |  ||  | 
>> | +---+   +---++---v+rewriter+  ++ | |
>> | | |   | ||||  |  ||  | 
>> | |   |   || |  || | |
>> | | |  mirror   | |  redirect  +>  compare   |  |  |   
>> +> mirror   +---> adjust |   adjust+-->redirect| | |
>> | | |  client   | |  server|||  |  |   | 
>> | |  server   |   | ack|   seq   |  |client  | | |
>> | | |   | ||||  |  |   | 
>> | |   |   || |  || | |
>> | | +^--+ +^---++-+--+  |  |   | 
>> | +---+   ++-+  ++---+ | |
>> | |  | tx  |  rx  | rx  |  |   | 
>> |txall   |  rx | |
>> | +-+  |   | 
>> +--+ |
>> ||
>> 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/20/2016 06:01 PM, Wen Congyang wrote:
> On 01/20/2016 02:54 PM, Jason Wang wrote:
>>
>> On 01/20/2016 11:29 AM, Zhang Chen wrote:
 Sure.

 Two main comments/suggestions:

 - TCP analysis is missed in current version, maybe you point a git tree
 (or another version of RFC) to me for a better understanding of the
 design. (Just a skeleton for TCP should be sufficient to discuss).
 - I prefer to make the code as reusable as possible. So it's better to
 split/decouple the reusable parts from the codes. So a vague idea is:

 1) Decouple the packet comparing from the netfilter. You've achieved
 this 99% since the work has been done in a thread. Just let the thread
 poll sockets directly, then the comparing have the possibility to be
 reused by other kinds of dataplane.
 2) Implement traffic mirror/redirector as filter.
 3) Implement TCP seq rewriting as a filter.

 Then, in primary node, you need just a traffic mirror, which did:
 - mirror ingress traffic to secondary node
 - mirror outgress traffic to packet comparing thread

 And in secondadry node, you need two filters:
 - A TCP seq rewriter which adjust tcp sequence number.
 - A traffic redirector which redirect packet from a socket as ingress
 traffic, and redirect outgress traffic to the socket which could be
 polled by remote packet comparing thread.
   Thoughts?

 Thanks

> Thanks
> zhangchen
>>>
>>> Hi, Jason.
>>> We consider your suggestion to split/decouple
>>> the reusable parts from the codes.
>>> Due to filter plugin are traversed one by one in order
>>> we will split colo-proxy to three filters in each side.
>>>
>>> But in this plan,primary and secondary both have socket
>>> server,startup is a problem.
>> I believe this issue could be solved by reusing socket chardev.
>>
>>>
>>>  Primary qemu  
>>> Secondary qemu
>>> +--+  
>>> +---+
>>> | +-+  |   | 
>>> +--+ |
>>> | | |  |   | 
>>> |  | |
>>> | |guest|  |   | 
>>> |guest | |
>>> | | |  |   | 
>>> |  | |
>>> | +---^--+--+  |   | 
>>> +-++---+ |
>>> | |  | |  
>>> |^| |
>>> | |  | |  
>>> ||| |
>>> | +-+ 
>>> ||| |
>>> |  netfilter  |  | ||  |  
>>> netfilter|| |
>>> | +-+  ||  | 
>>> +--+ |
>>> | |   |  | filter excute order  |  ||  | 
>>> | ||  filter excute order  | |
>>> | |   |  |+---> |  ||  | 
>>> | || +---> | |
>>> | |   |  |  |  ||  | 
>>> | ||   TCP | |
>>> | | +-+-+ +--v-+++ +-+  |  ||  | 
>>> | +---+   +---++---v+rewriter+  ++ | |
>>> | | |   | ||||  |  ||  | 
>>> | |   |   || |  || | |
>>> | | |  mirror   | |  redirect  +>  compare   |  |  |   
>>> +> mirror   +---> adjust |   adjust+-->redirect| | |
>>> | | |  client   | |  server|||  |  |   | 
>>> | |  server   |   | ack|   seq   |  |client  | | |
>>> | | |   | ||||  |  |   | 
>>> | |   |   || |  || | |
>>> | | +^--+ +^---++-+--+  |  |   | 
>>> | +---+   ++-+  ++---+ | |
>>> | |  | tx  |  rx  | rx  |  |   | 
>>> |txall   |  rx | |
>>> | +-+  |   | 
>>> 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/20/2016 06:03 PM, Jason Wang wrote:
> 
> 
> On 01/20/2016 05:49 PM, Wen Congyang wrote:
>> On 01/20/2016 05:20 PM, Jason Wang wrote:
>>>
>>> On 01/20/2016 03:44 PM, Wen Congyang wrote:
>> ...
>> -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait
>> -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait
>> -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait
>> -netdev tap,id=hn0
>> -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0
>> -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1
>> ...
>>
>> packet comparer compares the packets from two chardev: comparer0 and
>> comparer1.
>> traffic-mirrorer mirror tx to secondary node through chardev mirrorer0,
>> and mirror rx to packet comparer through chardev comparer0.
>>
>> In secondary node:
>>
>> ...
>> -chardev socket,id=redirector0,host=ip_primary,port=Y
>> -chardev socket,id=redirector1,host=ip_primary,port=Z
>> -netdev tap,id=hn0
>> -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1
>> -colo-rewriter netdev=hn0,id=c0
>> ...
>>
>> traffic-redirector redirect the rx traffic from primary node through
>> redirector0 and redirect the tx traffic to promary node through 
>> redirector1.
>> colo-rewriter rewrite seq number as a normal netfilter.
 What are traffic-mirrorer and colo-comparer, traffic-redirector, 
 colo-rewriter?
 A netfilter driver?
>>> traffic-mirrorer/redirector is a type of netfilter that just
>>> mirror/redirect packets between netdev and chardev (just the mirror
>>> client/sever and redirect client/sever in the above graph)
>>> colo-rewriter is a type of netfilter that did ack/seq adjust (just the
>>> TCP rewriter in the above graph)
>>> colo-comparer is a thread object that did packet comparing (similar to
>>> "compare" in the above graph but not a netfiler)
>> Thanks. I have another question:
>> IIRC, both rx and tx packets walk through all netfilter objects in the same 
>> order.
>>
>> tx packet(sent to the guest): we want that redirector hanldes it first
>> rx packet(sent from the guest): we want that colo-rewriter handles it first
>> Change the order or use two traffic-redirectors?
>>
>> Thanks
>> Wen Congyang
> 
> Interesting question.
> 
> Two redirectors sounds ok or maybe we can go through rx filters in a
> reverse order?

netdev <---> filter1 <> filter2 <>  <> emulated device <> 
guest
So I think we can go through rx filters in a reverse order. But it changes
the behavior. So I am not sure if we can do it.

Thanks
Wen Congyang

> 
> 
> .
> 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/20/2016 06:19 PM, Jason Wang wrote:
> 
> 
> On 01/20/2016 06:01 PM, Wen Congyang wrote:
>> On 01/20/2016 02:54 PM, Jason Wang wrote:
>>>
>>> On 01/20/2016 11:29 AM, Zhang Chen wrote:
> Sure.
>
> Two main comments/suggestions:
>
> - TCP analysis is missed in current version, maybe you point a git tree
> (or another version of RFC) to me for a better understanding of the
> design. (Just a skeleton for TCP should be sufficient to discuss).
> - I prefer to make the code as reusable as possible. So it's better to
> split/decouple the reusable parts from the codes. So a vague idea is:
>
> 1) Decouple the packet comparing from the netfilter. You've achieved
> this 99% since the work has been done in a thread. Just let the thread
> poll sockets directly, then the comparing have the possibility to be
> reused by other kinds of dataplane.
> 2) Implement traffic mirror/redirector as filter.
> 3) Implement TCP seq rewriting as a filter.
>
> Then, in primary node, you need just a traffic mirror, which did:
> - mirror ingress traffic to secondary node
> - mirror outgress traffic to packet comparing thread
>
> And in secondadry node, you need two filters:
> - A TCP seq rewriter which adjust tcp sequence number.
> - A traffic redirector which redirect packet from a socket as ingress
> traffic, and redirect outgress traffic to the socket which could be
> polled by remote packet comparing thread.
>   Thoughts?
>
> Thanks
>
>> Thanks
>> zhangchen

 Hi, Jason.
 We consider your suggestion to split/decouple
 the reusable parts from the codes.
 Due to filter plugin are traversed one by one in order
 we will split colo-proxy to three filters in each side.

 But in this plan,primary and secondary both have socket
 server,startup is a problem.
>>> I believe this issue could be solved by reusing socket chardev.
>>>

  Primary qemu  
 Secondary qemu
 +--+  
 +---+
 | +-+  |   | 
 +--+ |
 | | |  |   | 
 |  | |
 | |guest|  |   | 
 |guest | |
 | | |  |   | 
 |  | |
 | +---^--+--+  |   | 
 +-++---+ |
 | |  | |  
 |^| |
 | |  | |  
 ||| |
 | +-+ 
 ||| |
 |  netfilter  |  | ||  |  
 netfilter|| |
 | +-+  ||  | 
 +--+ |
 | |   |  | filter excute order  |  ||  | 
 | ||  filter excute order  | |
 | |   |  |+---> |  ||  | 
 | || +---> | |
 | |   |  |  |  ||  | 
 | ||   TCP | |
 | | +-+-+ +--v-+++ +-+  |  ||  | 
 | +---+   +---++---v+rewriter+  ++ | |
 | | |   | ||||  |  ||  | 
 | |   |   || |  || | |
 | | |  mirror   | |  redirect  +>  compare   |  |  |   
 +> mirror   +---> adjust |   adjust+-->redirect| | |
 | | |  client   | |  server|||  |  |   | 
 | |  server   |   | ack|   seq   |  |client  | | |
 | | |   | ||||  |  |   | 
 | |   |   || |  || | |
 | | +^--+ +^---++-+--+  |  |   | 
 | +---+   ++-+  ++---+ | |
 | |  | tx  |  rx  | rx  |  |   | 
 |txall   

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Zhang Chen]

Sure.

Two main comments/suggestions:

- TCP analysis is missed in current version, maybe you point a git tree
(or another version of RFC) to me for a better understanding of the
design. (Just a skeleton for TCP should be sufficient to discuss).
- I prefer to make the code as reusable as possible. So it's better to
split/decouple the reusable parts from the codes. So a vague idea is:

1) Decouple the packet comparing from the netfilter. You've achieved
this 99% since the work has been done in a thread. Just let the thread
poll sockets directly, then the comparing have the possibility to be
reused by other kinds of dataplane.
2) Implement traffic mirror/redirector as filter.
3) Implement TCP seq rewriting as a filter.

Then, in primary node, you need just a traffic mirror, which did:
- mirror ingress traffic to secondary node
- mirror outgress traffic to packet comparing thread

And in secondadry node, you need two filters:
- A TCP seq rewriter which adjust tcp sequence number.
- A traffic redirector which redirect packet from a socket as ingress
traffic, and redirect outgress traffic to the socket which could be
polled by remote packet comparing thread.
  
Thoughts?


Thanks


Thanks
zhangchen





Hi, Jason.
We consider your suggestion to split/decouple
the reusable parts from the codes.
Due to filter plugin are traversed one by one in order
we will split colo-proxy to three filters in each side.

But in this plan,primary and secondary both have socket
server,startup is a problem.


 Primary qemu   Secondary 
qemu
+--+   
+---+
| +-+  |   |  
+--+ |
| | |  |   |  | 
 | |
| |guest|  |   |  | 
   guest | |
| | |  |   |  | 
 | |
| +---^--+--+  |   |  
+-++---+ |
| |  | |   |
^| |
| |  | |   |
|| |
| +-+  |
|| |
|  netfilter  |  | ||  |   
netfilter|| |
| +-+  ||  |  
+--+ |
| |   |  | filter excute order  |  ||  |  | 
||  filter excute order  | |
| |   |  |+---> |  ||  |  |
 || +---> | |
| |   |  |  |  ||  |  | 
||   TCP | |
| | +-+-+ +--v-+++ +-+  |  ||  |  | 
+---+   +---++---v+rewriter+  ++ | |
| | |   | ||||  |  ||  |  | |   
|   || |  || | |
| | |  mirror   | |  redirect  +>  compare   |  |  |+> mirror   
+---> adjust |   adjust+-->redirect| | |
| | |  client   | |  server|||  |  |   |  | |  
server   |   | ack|   seq   |  |client  | | |
| | |   | ||||  |  |   |  | |   
|   || |  || | |
| | +^--+ +^---++-+--+  |  |   |  | 
+---+   ++-+  ++---+ | |
| |  | tx  |  rx  | rx  |  |   |  | 
   txall   |  rx | |
| +-+  |   |  
+--+ |
|| 
+---+
   |
||||   |
   |
+--+   
+---+
 ||
 |guest receive   |guest send
 |

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/20/2016 11:29 AM, Zhang Chen wrote:
>
>> Sure.
>>
>> Two main comments/suggestions:
>>
>> - TCP analysis is missed in current version, maybe you point a git tree
>> (or another version of RFC) to me for a better understanding of the
>> design. (Just a skeleton for TCP should be sufficient to discuss).
>> - I prefer to make the code as reusable as possible. So it's better to
>> split/decouple the reusable parts from the codes. So a vague idea is:
>>
>> 1) Decouple the packet comparing from the netfilter. You've achieved
>> this 99% since the work has been done in a thread. Just let the thread
>> poll sockets directly, then the comparing have the possibility to be
>> reused by other kinds of dataplane.
>> 2) Implement traffic mirror/redirector as filter.
>> 3) Implement TCP seq rewriting as a filter.
>>
>> Then, in primary node, you need just a traffic mirror, which did:
>> - mirror ingress traffic to secondary node
>> - mirror outgress traffic to packet comparing thread
>>
>> And in secondadry node, you need two filters:
>> - A TCP seq rewriter which adjust tcp sequence number.
>> - A traffic redirector which redirect packet from a socket as ingress
>> traffic, and redirect outgress traffic to the socket which could be
>> polled by remote packet comparing thread.
>>   Thoughts?
>>
>> Thanks
>>
>>> Thanks
>>> zhangchen
>>
>
>
> Hi, Jason.
> We consider your suggestion to split/decouple
> the reusable parts from the codes.
> Due to filter plugin are traversed one by one in order
> we will split colo-proxy to three filters in each side.
>
> But in this plan,primary and secondary both have socket
> server,startup is a problem.

I believe this issue could be solved by reusing socket chardev.

>
>
>  Primary qemu  
> Secondary qemu
> +--+  
> +---+
> | +-+  |   | 
> +--+ |
> | | |  |   | 
> |  | |
> | |guest|  |   | 
> |guest | |
> | | |  |   | 
> |  | |
> | +---^--+--+  |   | 
> +-++---+ |
> | |  | |  
> |^| |
> | |  | |  
> ||| |
> | +-+ 
> ||| |
> |  netfilter  |  | ||  |  
> netfilter|| |
> | +-+  ||  | 
> +--+ |
> | |   |  | filter excute order  |  ||  | 
> | ||  filter excute order  | |
> | |   |  |+---> |  ||  | 
> | || +---> | |
> | |   |  |  |  ||  | 
> | ||   TCP | |
> | | +-+-+ +--v-+++ +-+  |  ||  | 
> | +---+   +---++---v+rewriter+  ++ | |
> | | |   | ||||  |  ||  | 
> | |   |   || |  || | |
> | | |  mirror   | |  redirect  +>  compare   |  |  |   
> +> mirror   +---> adjust |   adjust+-->redirect| | |
> | | |  client   | |  server|||  |  |   | 
> | |  server   |   | ack|   seq   |  |client  | | |
> | | |   | ||||  |  |   | 
> | |   |   || |  || | |
> | | +^--+ +^---++-+--+  |  |   | 
> | +---+   ++-+  ++---+ | |
> | |  | tx  |  rx  | rx  |  |   | 
> |txall   |  rx | |
> | +-+  |   | 
> +--+ |
> ||
> +---+
>   
> |
> ||||  
> |

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Wen Congyang]

On 01/20/2016 02:54 PM, Jason Wang wrote:
> 
> 
> On 01/20/2016 11:29 AM, Zhang Chen wrote:
>>
>>> Sure.
>>>
>>> Two main comments/suggestions:
>>>
>>> - TCP analysis is missed in current version, maybe you point a git tree
>>> (or another version of RFC) to me for a better understanding of the
>>> design. (Just a skeleton for TCP should be sufficient to discuss).
>>> - I prefer to make the code as reusable as possible. So it's better to
>>> split/decouple the reusable parts from the codes. So a vague idea is:
>>>
>>> 1) Decouple the packet comparing from the netfilter. You've achieved
>>> this 99% since the work has been done in a thread. Just let the thread
>>> poll sockets directly, then the comparing have the possibility to be
>>> reused by other kinds of dataplane.
>>> 2) Implement traffic mirror/redirector as filter.
>>> 3) Implement TCP seq rewriting as a filter.
>>>
>>> Then, in primary node, you need just a traffic mirror, which did:
>>> - mirror ingress traffic to secondary node
>>> - mirror outgress traffic to packet comparing thread
>>>
>>> And in secondadry node, you need two filters:
>>> - A TCP seq rewriter which adjust tcp sequence number.
>>> - A traffic redirector which redirect packet from a socket as ingress
>>> traffic, and redirect outgress traffic to the socket which could be
>>> polled by remote packet comparing thread.
>>>   Thoughts?
>>>
>>> Thanks
>>>
 Thanks
 zhangchen
>>>
>>
>>
>> Hi, Jason.
>> We consider your suggestion to split/decouple
>> the reusable parts from the codes.
>> Due to filter plugin are traversed one by one in order
>> we will split colo-proxy to three filters in each side.
>>
>> But in this plan,primary and secondary both have socket
>> server,startup is a problem.
> 
> I believe this issue could be solved by reusing socket chardev.
> 
>>
>>
>>  Primary qemu  
>> Secondary qemu
>> +--+  
>> +---+
>> | +-+  |   | 
>> +--+ |
>> | | |  |   | 
>> |  | |
>> | |guest|  |   | 
>> |guest | |
>> | | |  |   | 
>> |  | |
>> | +---^--+--+  |   | 
>> +-++---+ |
>> | |  | |  
>> |^| |
>> | |  | |  
>> ||| |
>> | +-+ 
>> ||| |
>> |  netfilter  |  | ||  |  
>> netfilter|| |
>> | +-+  ||  | 
>> +--+ |
>> | |   |  | filter excute order  |  ||  | 
>> | ||  filter excute order  | |
>> | |   |  |+---> |  ||  | 
>> | || +---> | |
>> | |   |  |  |  ||  | 
>> | ||   TCP | |
>> | | +-+-+ +--v-+++ +-+  |  ||  | 
>> | +---+   +---++---v+rewriter+  ++ | |
>> | | |   | ||||  |  ||  | 
>> | |   |   || |  || | |
>> | | |  mirror   | |  redirect  +>  compare   |  |  |   
>> +> mirror   +---> adjust |   adjust+-->redirect| | |
>> | | |  client   | |  server|||  |  |   | 
>> | |  server   |   | ack|   seq   |  |client  | | |
>> | | |   | ||||  |  |   | 
>> | |   |   || |  || | |
>> | | +^--+ +^---++-+--+  |  |   | 
>> | +---+   ++-+  ++---+ | |
>> | |  | tx  |  rx  | rx  |  |   | 
>> |txall   |  rx | |
>> | +-+  |   | 
>> +--+ |
>> ||
>> 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/18/2016 03:05 PM, Zhang Chen wrote:
>
>
> On 01/06/2016 01:16 PM, Jason Wang wrote:
>>
>> On 01/04/2016 07:17 PM, Zhang Chen wrote:
>>>
>>> On 01/04/2016 05:46 PM, Jason Wang wrote:
 On 01/04/2016 04:16 PM, Zhang Chen wrote:
> On 01/04/2016 01:37 PM, Jason Wang wrote:
>> On 12/31/2015 04:40 PM, Zhang Chen wrote:
>>> On 12/31/2015 10:36 AM, Jason Wang wrote:
 On 12/22/2015 06:42 PM, Zhang Chen wrote:
> From: zhangchen 
>
> Hi,all
>
> This patch add an colo-proxy object, COLO-Proxy is a part of
> COLO,
> based on qemu netfilter and it's a plugin for qemu netfilter. the
> function
> keep Secondary VM connect normal to Primary VM and compare
> packets
> sent by PVM to sent by SVM.if the packet difference,notify
> COLO do
> checkpoint and send all primary packet has queued.
 Thanks for the work. I don't object this method but still not
 convinced
 that qemu is the best place for this.

 As been raised in the past discussion, it's almost impossible to
 cooperate with vhost backends. If we want this to be used in
 production
 environment, need to think of a solution for vhost. There's no
 such
 worry if we decouple this from qemu.

> You can also get the series from:
>
> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
>
>
>
>
>
> Usage:
>
> primary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object
> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
>
> secondary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object
> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
>
 Have a quick glance at how secondary mode work. What it does is
 just
 forwarding packets between a nic and a socket, qemu socket
 backend did
 exact the same job. You could even use socket in primary node and
 let
 packet compare module talk to both primary and secondary node.
>>> If we use qemu socket backend , the same netdev will used by qemu
>>> socket and
>>> qemu netfilter. this will against qemu net design. and then, when
>>> colo
>>> do failover,
>>> secondary do not have backend to use. that's the real problem.
>> Then, maybe it's time to implement changing the netdev of a nic. The
>> point here is that what secondary mode did is in fact a netdev
>> backend
>> instead of a filter ...
> Currently, you are right. in colo-proxy V2 code, I just compare IP
> packet to
> decide whether to do checkpoint.
> But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide
> it.
> because that can reduce frequency of checkpoint and improve
> performance. To keep tcp connection well, colo secondary need to
> record
> primary guest's init seq and adjust secondary guest's ack. if colo do
> failover,
> secondary also need do this to old tcp connection. qemu socket
> can't do this job.
 So a question here: is it a must to do things (e.g TCP analysis
 stuffs)
 at secondary? Looks like we could do this at primary node. And I saw
 you're doing packet comparing in primary node, any advantages of doing
 this in primary instead of secondary?
>>> We think must  to do this in secondary, because if colo do
>>> failover,secondary
>>> must continues do TCP analysis stuffs to before tcp connection(if not,
>>> tcp connection
>>> will disconnect in that time), in this time primary already down or
>>> disconnect to
>>> secondary.so we can't make primary do this  TCP analysis stuffs.it can
>>> not ensure
>>> FT function.
>>>
>>> Thanks
>>> zhangchen
>> Makes sense.
>>
>> Thanks
>
> Hi~, Jason.
> No news for a week.
> Can you give me some comments for code.
> Let's make colo-proxy work well.

Sure.

Two main comments/suggestions:

- TCP analysis is missed in current version, maybe you point a git tree
(or another version of RFC) to me for a better understanding of the
design. (Just a skeleton for TCP should be sufficient to discuss).
- I prefer to make the code as reusable as possible. So it's better to
split/decouple the reusable parts from the codes. So a vague idea is:

1) Decouple the packet comparing from the netfilter. You've achieved
this 99% since the work has been done in a thread. Just let the thread
poll sockets directly, then the comparing have the possibility to be
reused by other kinds of dataplane.
2) Implement traffic mirror/redirector as filter.
3) Implement TCP seq rewriting as a filter.

Then, in primary node, you need just a traffic mirror, which did:
- mirror ingress traffic to secondary node
- mirror 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Zhang Chen]

On 01/06/2016 01:16 PM, Jason Wang wrote:


On 01/04/2016 07:17 PM, Zhang Chen wrote:


On 01/04/2016 05:46 PM, Jason Wang wrote:

On 01/04/2016 04:16 PM, Zhang Chen wrote:

On 01/04/2016 01:37 PM, Jason Wang wrote:

On 12/31/2015 04:40 PM, Zhang Chen wrote:

On 12/31/2015 10:36 AM, Jason Wang wrote:

On 12/22/2015 06:42 PM, Zhang Chen wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the
function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.

Thanks for the work. I don't object this method but still not
convinced
that qemu is the best place for this.

As been raised in the past discussion, it's almost impossible to
cooperate with vhost backends. If we want this to be used in
production
environment, need to think of a solution for vhost. There's no such
worry if we decouple this from qemu.


You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2




Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object
colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object
colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port

Have a quick glance at how secondary mode work. What it does is just
forwarding packets between a nic and a socket, qemu socket
backend did
exact the same job. You could even use socket in primary node and
let
packet compare module talk to both primary and secondary node.

If we use qemu socket backend , the same netdev will used by qemu
socket and
qemu netfilter. this will against qemu net design. and then, when
colo
do failover,
secondary do not have backend to use. that's the real problem.

Then, maybe it's time to implement changing the netdev of a nic. The
point here is that what secondary mode did is in fact a netdev backend
instead of a filter ...

Currently, you are right. in colo-proxy V2 code, I just compare IP
packet to
decide whether to do checkpoint.
But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it.
because that can reduce frequency of checkpoint and improve
performance. To keep tcp connection well, colo secondary need to record
primary guest's init seq and adjust secondary guest's ack. if colo do
failover,
secondary also need do this to old tcp connection. qemu socket
can't do this job.

So a question here: is it a must to do things (e.g TCP analysis stuffs)
at secondary? Looks like we could do this at primary node. And I saw
you're doing packet comparing in primary node, any advantages of doing
this in primary instead of secondary?

We think must  to do this in secondary, because if colo do
failover,secondary
must continues do TCP analysis stuffs to before tcp connection(if not,
tcp connection
will disconnect in that time), in this time primary already down or
disconnect to
secondary.so we can't make primary do this  TCP analysis stuffs.it can
not ensure
FT function.

Thanks
zhangchen

Makes sense.

Thanks


Hi~, Jason.
No news for a week.
Can you give me some comments for code.
Let's make colo-proxy work well.

Thanks
zhangchen


and another problem is do failover, if we use qemu socket
to be backend in secondary, when colo do failover, I don't know how to
change
secondary be a normal qemu, if you know, please tell me.

Current qemu couldn't do this, but I mean we implement something like
nic_change_backend which can change nic's peer(s). With this, in
secondary, we can replace the socket backend with whatever you want (e.g
tap or other).

Thanks


Thanks for your revew
zhangchen


.




.



--
Thanks
zhangchen

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Zhang Chen]

On 01/11/2016 08:59 PM, Dr. David Alan Gilbert wrote:

* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote:


On 01/08/2016 07:19 PM, Dr. David Alan Gilbert wrote:

* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.

You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2

Are you sure that tag is correct? The series of commits doesn't seem to match
up with the set of commits posted.

Dave

Yes, it is. we have some code fix in other colo unrelated file.
in email, I just send colo related part.

That doesn't seem to be what's happening in that git tree.
For example, your patch '[RFC PATCH v2 01/10] Init colo-proxy object based on 
netfilter'
adds the colo-proxy object to qemu-options.hx, but in that git tree
it comes from Li Zhijian's 'add proxy prototype' patch.
If you're going to include a git link with a series then please
make sure it contains exactly the patches posted.
It's OK to add some more patches somewhere, e.g. on a different tag or
branch, but make sure the one that you post for the series matches
the series posted.

Dave



Make sense.
I have fix it.

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2

Thanks
zhangchen


zhangchen
Thanks for review


Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port

NOTE:
queue must set "all". See enum NetFilterDirection for detail.
colo-proxy need queue all packets
colo-proxy V2 just can compare ip packet


## Background

COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on
demand)is conducted.

Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0

COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking
called colo-proxy.

V2:
   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
   fix dave's comments
   fix wency's comments
   fix zhanghailiang's comments

v1:
   initial patch.



zhangchen (10):
   Init colo-proxy object based on netfilter
   Jhash: add linux kernel jhashtable in qemu
   Colo-proxy: add colo-proxy framework
   Colo-proxy: add data structure and jhash func
   net/colo-proxy: Add colo interface to use proxy
   net/colo-proxy: add socket used by forward func
   net/colo-proxy: Add packet enqueue & handle func
   net/colo-proxy: Handle packet and connection
   net/colo-proxy: Compare pri pkt to sec pkt
   net/colo-proxy: Colo-proxy do checkpoint and clear

  include/qemu/jhash.h |  61 
  net/Makefile.objs|   1 +
  net/colo-proxy.c | 939 +++
  net/colo-proxy.h |  24 ++
  qemu-options.hx  |   6 +
  trace-events |   8 +
  vl.c |   3 +-
  7 files changed, 1041 insertions(+), 1 deletion(-)
  create mode 100644 include/qemu/jhash.h
  create mode 100644 net/colo-proxy.c
  create mode 100644 net/colo-proxy.h

--
1.9.1




--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK


.


--
Thanks
zhangchen




--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK



.



--
Thanks
zhangchen

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Dr. David Alan Gilbert]

* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote:
> 
> 
> On 01/08/2016 07:19 PM, Dr. David Alan Gilbert wrote:
> >* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote:
> >>From: zhangchen 
> >>
> >>Hi,all
> >>
> >>This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
> >>based on qemu netfilter and it's a plugin for qemu netfilter. the function
> >>keep Secondary VM connect normal to Primary VM and compare packets
> >>sent by PVM to sent by SVM.if the packet difference,notify COLO do
> >>checkpoint and send all primary packet has queued.
> >>
> >>You can also get the series from:
> >>
> >>https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
> >Are you sure that tag is correct? The series of commits doesn't seem to match
> >up with the set of commits posted.
> >
> >Dave
> 
> Yes, it is. we have some code fix in other colo unrelated file.
> in email, I just send colo related part.

That doesn't seem to be what's happening in that git tree.
For example, your patch '[RFC PATCH v2 01/10] Init colo-proxy object based on 
netfilter'
adds the colo-proxy object to qemu-options.hx, but in that git tree
it comes from Li Zhijian's 'add proxy prototype' patch.
If you're going to include a git link with a series then please
make sure it contains exactly the patches posted.
It's OK to add some more patches somewhere, e.g. on a different tag or
branch, but make sure the one that you post for the series matches
the series posted.

Dave

> 
> zhangchen
> Thanks for review
> 
> >>Usage:
> >>
> >>primary:
> >>-netdev tap,id=bn0 -device e1000,netdev=bn0
> >>-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
> >>
> >>secondary:
> >>-netdev tap,id=bn0 -device e1000,netdev=bn0
> >>-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
> >>
> >>NOTE:
> >>queue must set "all". See enum NetFilterDirection for detail.
> >>colo-proxy need queue all packets
> >>colo-proxy V2 just can compare ip packet
> >>
> >>
> >>## Background
> >>
> >>COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop 
> >>Service)
> >>project is a high availability solution. Both Primary VM (PVM) and 
> >>Secondary VM
> >>(SVM) run in parallel. They receive the same request from client, and 
> >>generate
> >>responses in parallel too. If the response packets from PVM and SVM are
> >>identical, they are released immediately. Otherwise, a VM checkpoint (on
> >>demand)is conducted.
> >>
> >>Paper:
> >>http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
> >>
> >>COLO on Xen:
> >>http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> >>
> >>COLO on Qemu/KVM:
> >>http://wiki.qemu.org/Features/COLO
> >>
> >>By the needs of capturing response packets from PVM and SVM and finding out
> >>whether they are identical, we introduce a new module to qemu networking
> >>called colo-proxy.
> >>
> >>V2:
> >>   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
> >>   fix dave's comments
> >>   fix wency's comments
> >>   fix zhanghailiang's comments
> >>
> >>v1:
> >>   initial patch.
> >>
> >>
> >>
> >>zhangchen (10):
> >>   Init colo-proxy object based on netfilter
> >>   Jhash: add linux kernel jhashtable in qemu
> >>   Colo-proxy: add colo-proxy framework
> >>   Colo-proxy: add data structure and jhash func
> >>   net/colo-proxy: Add colo interface to use proxy
> >>   net/colo-proxy: add socket used by forward func
> >>   net/colo-proxy: Add packet enqueue & handle func
> >>   net/colo-proxy: Handle packet and connection
> >>   net/colo-proxy: Compare pri pkt to sec pkt
> >>   net/colo-proxy: Colo-proxy do checkpoint and clear
> >>
> >>  include/qemu/jhash.h |  61 
> >>  net/Makefile.objs|   1 +
> >>  net/colo-proxy.c | 939 
> >> +++
> >>  net/colo-proxy.h |  24 ++
> >>  qemu-options.hx  |   6 +
> >>  trace-events |   8 +
> >>  vl.c |   3 +-
> >>  7 files changed, 1041 insertions(+), 1 deletion(-)
> >>  create mode 100644 include/qemu/jhash.h
> >>  create mode 100644 net/colo-proxy.c
> >>  create mode 100644 net/colo-proxy.h
> >>
> >>-- 
> >>1.9.1
> >>
> >>
> >>
> >--
> >Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
> >
> >
> >.
> >
> 
> -- 
> Thanks
> zhangchen
> 
> 
> 
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Zhang Chen]

On 01/08/2016 07:19 PM, Dr. David Alan Gilbert wrote:

* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.

You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2

Are you sure that tag is correct? The series of commits doesn't seem to match
up with the set of commits posted.

Dave


Yes, it is. we have some code fix in other colo unrelated file.
in email, I just send colo related part.

zhangchen
Thanks for review


Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port

NOTE:
queue must set "all". See enum NetFilterDirection for detail.
colo-proxy need queue all packets
colo-proxy V2 just can compare ip packet


## Background

COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on
demand)is conducted.

Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0

COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking
called colo-proxy.

V2:
   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
   fix dave's comments
   fix wency's comments
   fix zhanghailiang's comments

v1:
   initial patch.



zhangchen (10):
   Init colo-proxy object based on netfilter
   Jhash: add linux kernel jhashtable in qemu
   Colo-proxy: add colo-proxy framework
   Colo-proxy: add data structure and jhash func
   net/colo-proxy: Add colo interface to use proxy
   net/colo-proxy: add socket used by forward func
   net/colo-proxy: Add packet enqueue & handle func
   net/colo-proxy: Handle packet and connection
   net/colo-proxy: Compare pri pkt to sec pkt
   net/colo-proxy: Colo-proxy do checkpoint and clear

  include/qemu/jhash.h |  61 
  net/Makefile.objs|   1 +
  net/colo-proxy.c | 939 +++
  net/colo-proxy.h |  24 ++
  qemu-options.hx  |   6 +
  trace-events |   8 +
  vl.c |   3 +-
  7 files changed, 1041 insertions(+), 1 deletion(-)
  create mode 100644 include/qemu/jhash.h
  create mode 100644 net/colo-proxy.c
  create mode 100644 net/colo-proxy.h

--
1.9.1




--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK


.



--
Thanks
zhangchen

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Dr. David Alan Gilbert]

* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote:
> From: zhangchen 
> 
> Hi,all
> 
> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
> based on qemu netfilter and it's a plugin for qemu netfilter. the function
> keep Secondary VM connect normal to Primary VM and compare packets
> sent by PVM to sent by SVM.if the packet difference,notify COLO do
> checkpoint and send all primary packet has queued.
> 
> You can also get the series from:
> 
> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2

Are you sure that tag is correct? The series of commits doesn't seem to match
up with the set of commits posted.

Dave

> 
> Usage:
> 
> primary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
> 
> secondary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port 
> 
> NOTE:
> queue must set "all". See enum NetFilterDirection for detail.
> colo-proxy need queue all packets
> colo-proxy V2 just can compare ip packet
> 
> 
> ## Background
> 
> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
> project is a high availability solution. Both Primary VM (PVM) and Secondary 
> VM
> (SVM) run in parallel. They receive the same request from client, and generate
> responses in parallel too. If the response packets from PVM and SVM are
> identical, they are released immediately. Otherwise, a VM checkpoint (on
> demand)is conducted.
> 
> Paper:
> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
> 
> COLO on Xen:
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> 
> COLO on Qemu/KVM:
> http://wiki.qemu.org/Features/COLO
> 
> By the needs of capturing response packets from PVM and SVM and finding out
> whether they are identical, we introduce a new module to qemu networking
> called colo-proxy.
> 
> V2:
>   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
>   fix dave's comments
>   fix wency's comments
>   fix zhanghailiang's comments
> 
> v1:
>   initial patch.
> 
> 
> 
> zhangchen (10):
>   Init colo-proxy object based on netfilter
>   Jhash: add linux kernel jhashtable in qemu
>   Colo-proxy: add colo-proxy framework
>   Colo-proxy: add data structure and jhash func
>   net/colo-proxy: Add colo interface to use proxy
>   net/colo-proxy: add socket used by forward func
>   net/colo-proxy: Add packet enqueue & handle func
>   net/colo-proxy: Handle packet and connection
>   net/colo-proxy: Compare pri pkt to sec pkt
>   net/colo-proxy: Colo-proxy do checkpoint and clear
> 
>  include/qemu/jhash.h |  61 
>  net/Makefile.objs|   1 +
>  net/colo-proxy.c | 939 
> +++
>  net/colo-proxy.h |  24 ++
>  qemu-options.hx  |   6 +
>  trace-events |   8 +
>  vl.c |   3 +-
>  7 files changed, 1041 insertions(+), 1 deletion(-)
>  create mode 100644 include/qemu/jhash.h
>  create mode 100644 net/colo-proxy.c
>  create mode 100644 net/colo-proxy.h
> 
> -- 
> 1.9.1
> 
> 
> 
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Dr. David Alan Gilbert]

* Jason Wang (jasow...@redhat.com) wrote:
> 
> 
> On 01/05/2016 12:52 AM, Dr. David Alan Gilbert wrote:
> > * Jason Wang (jasow...@redhat.com) wrote:
> >>
> >> On 01/04/2016 04:16 PM, Zhang Chen wrote:
> >>>
> >>> On 01/04/2016 01:37 PM, Jason Wang wrote:
>  On 12/31/2015 04:40 PM, Zhang Chen wrote:
> > On 12/31/2015 10:36 AM, Jason Wang wrote:
> >> On 12/22/2015 06:42 PM, Zhang Chen wrote:
> >>> From: zhangchen 
> >>>
> >>> Hi,all
> >>>
> >>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
> >>> based on qemu netfilter and it's a plugin for qemu netfilter. the
> >>> function
> >>> keep Secondary VM connect normal to Primary VM and compare packets
> >>> sent by PVM to sent by SVM.if the packet difference,notify COLO do
> >>> checkpoint and send all primary packet has queued.
> >> Thanks for the work. I don't object this method but still not
> >> convinced
> >> that qemu is the best place for this.
> >>
> >> As been raised in the past discussion, it's almost impossible to
> >> cooperate with vhost backends. If we want this to be used in
> >> production
> >> environment, need to think of a solution for vhost. There's no such
> >> worry if we decouple this from qemu.
> >>
> >>> You can also get the series from:
> >>>
> >>> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
> >>>
> >>>
> >>>
> >>> Usage:
> >>>
> >>> primary:
> >>> -netdev tap,id=bn0 -device e1000,netdev=bn0
> >>> -object
> >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
> >>>
> >>> secondary:
> >>> -netdev tap,id=bn0 -device e1000,netdev=bn0
> >>> -object
> >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
> >> Have a quick glance at how secondary mode work. What it does is just
> >> forwarding packets between a nic and a socket, qemu socket backend did
> >> exact the same job. You could even use socket in primary node and let
> >> packet compare module talk to both primary and secondary node.
> > If we use qemu socket backend , the same netdev will used by qemu
> > socket and
> > qemu netfilter. this will against qemu net design. and then, when colo
> > do failover,
> > secondary do not have backend to use. that's the real problem.
>  Then, maybe it's time to implement changing the netdev of a nic. The
>  point here is that what secondary mode did is in fact a netdev backend
>  instead of a filter ...
> >>> Currently, you are right. in colo-proxy V2 code, I just compare IP
> >>> packet to
> >>> decide whether to do checkpoint.
> >>> But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it.
> >>> because that can reduce frequency of checkpoint and improve
> >>> performance. To keep tcp connection well, colo secondary need to record
> >>> primary guest's init seq and adjust secondary guest's ack. if colo do
> >>> failover,
> >>> secondary also need do this to old tcp connection. qemu socket
> >>> can't do this job.
> >> So a question here: is it a must to do things (e.g TCP analysis stuffs)
> >> at secondary? Looks like we could do this at primary node. And I saw
> >> you're doing packet comparing in primary node, any advantages of doing
> >> this in primary instead of secondary?
> > It needs to do this on the secondary; the trick is that things like TCP 
> > sequence
> > numbers are likely to be different on the primary and secondary; the kernel 
> > colo-proxy
> > implementation solved this problem by rewriting the sequence numbers on
> > the secondary to match the primary, after a failover, the secondary has
> > to keep doing that rewrite to ensure existing connections are OK.
> > Thus it's holding some state about the current connections.
> 
> I see.
> 
> > I think also, to be able to do a 2nd failover (i.e. recover from the 1st 
> > failure
> > and then sometime later have another) you'd have to sync this
> > state over to a new host, so again that says the state needs to be part of
> > qemu or at least easily available to it.
> >
> > Dave
> 
> Right, if it does thing like tcp seq rewrite (which is missed in current
> version), it works much more like a netfilter. Wonder if the function is
> generic enough for users other than colo.

I can imagine the sequence number rework might be, but I doubt the packet
comparison is.

Dave

> Thanks
> 
> >
> >>> and another problem is do failover, if we use qemu socket
> >>> to be backend in secondary, when colo do failover, I don't know how to
> >>> change
> >>> secondary be a normal qemu, if you know, please tell me.
> >> Current qemu couldn't do this, but I mean we implement something like
> >> nic_change_backend which can change nic's peer(s). With this, in
> >> secondary, we can replace the socket backend with whatever you want (e.g
> >> tap or other).
> >>

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/04/2016 07:17 PM, Zhang Chen wrote:
>
>
> On 01/04/2016 05:46 PM, Jason Wang wrote:
>>
>> On 01/04/2016 04:16 PM, Zhang Chen wrote:
>>>
>>> On 01/04/2016 01:37 PM, Jason Wang wrote:
 On 12/31/2015 04:40 PM, Zhang Chen wrote:
> On 12/31/2015 10:36 AM, Jason Wang wrote:
>> On 12/22/2015 06:42 PM, Zhang Chen wrote:
>>> From: zhangchen 
>>>
>>> Hi,all
>>>
>>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
>>> based on qemu netfilter and it's a plugin for qemu netfilter. the
>>> function
>>> keep Secondary VM connect normal to Primary VM and compare packets
>>> sent by PVM to sent by SVM.if the packet difference,notify COLO do
>>> checkpoint and send all primary packet has queued.
>> Thanks for the work. I don't object this method but still not
>> convinced
>> that qemu is the best place for this.
>>
>> As been raised in the past discussion, it's almost impossible to
>> cooperate with vhost backends. If we want this to be used in
>> production
>> environment, need to think of a solution for vhost. There's no such
>> worry if we decouple this from qemu.
>>
>>> You can also get the series from:
>>>
>>> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
>>>
>>>
>>>
>>>
>>> Usage:
>>>
>>> primary:
>>> -netdev tap,id=bn0 -device e1000,netdev=bn0
>>> -object
>>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
>>>
>>> secondary:
>>> -netdev tap,id=bn0 -device e1000,netdev=bn0
>>> -object
>>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
>> Have a quick glance at how secondary mode work. What it does is just
>> forwarding packets between a nic and a socket, qemu socket
>> backend did
>> exact the same job. You could even use socket in primary node and
>> let
>> packet compare module talk to both primary and secondary node.
> If we use qemu socket backend , the same netdev will used by qemu
> socket and
> qemu netfilter. this will against qemu net design. and then, when
> colo
> do failover,
> secondary do not have backend to use. that's the real problem.
 Then, maybe it's time to implement changing the netdev of a nic. The
 point here is that what secondary mode did is in fact a netdev backend
 instead of a filter ...
>>> Currently, you are right. in colo-proxy V2 code, I just compare IP
>>> packet to
>>> decide whether to do checkpoint.
>>> But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it.
>>> because that can reduce frequency of checkpoint and improve
>>> performance. To keep tcp connection well, colo secondary need to record
>>> primary guest's init seq and adjust secondary guest's ack. if colo do
>>> failover,
>>> secondary also need do this to old tcp connection. qemu socket
>>> can't do this job.
>> So a question here: is it a must to do things (e.g TCP analysis stuffs)
>> at secondary? Looks like we could do this at primary node. And I saw
>> you're doing packet comparing in primary node, any advantages of doing
>> this in primary instead of secondary?
>
> We think must  to do this in secondary, because if colo do
> failover,secondary
> must continues do TCP analysis stuffs to before tcp connection(if not,
> tcp connection
> will disconnect in that time), in this time primary already down or
> disconnect to
> secondary.so we can't make primary do this  TCP analysis stuffs.it can
> not ensure
> FT function.
>
> Thanks
> zhangchen

Makes sense.

Thanks

>
>>> and another problem is do failover, if we use qemu socket
>>> to be backend in secondary, when colo do failover, I don't know how to
>>> change
>>> secondary be a normal qemu, if you know, please tell me.
>> Current qemu couldn't do this, but I mean we implement something like
>> nic_change_backend which can change nic's peer(s). With this, in
>> secondary, we can replace the socket backend with whatever you want (e.g
>> tap or other).
>>
>> Thanks
>>
>>>
>>> Thanks for your revew
>>> zhangchen
>>
>>
>> .
>>
>

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/05/2016 12:52 AM, Dr. David Alan Gilbert wrote:
> * Jason Wang (jasow...@redhat.com) wrote:
>>
>> On 01/04/2016 04:16 PM, Zhang Chen wrote:
>>>
>>> On 01/04/2016 01:37 PM, Jason Wang wrote:
 On 12/31/2015 04:40 PM, Zhang Chen wrote:
> On 12/31/2015 10:36 AM, Jason Wang wrote:
>> On 12/22/2015 06:42 PM, Zhang Chen wrote:
>>> From: zhangchen 
>>>
>>> Hi,all
>>>
>>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
>>> based on qemu netfilter and it's a plugin for qemu netfilter. the
>>> function
>>> keep Secondary VM connect normal to Primary VM and compare packets
>>> sent by PVM to sent by SVM.if the packet difference,notify COLO do
>>> checkpoint and send all primary packet has queued.
>> Thanks for the work. I don't object this method but still not
>> convinced
>> that qemu is the best place for this.
>>
>> As been raised in the past discussion, it's almost impossible to
>> cooperate with vhost backends. If we want this to be used in
>> production
>> environment, need to think of a solution for vhost. There's no such
>> worry if we decouple this from qemu.
>>
>>> You can also get the series from:
>>>
>>> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
>>>
>>>
>>>
>>> Usage:
>>>
>>> primary:
>>> -netdev tap,id=bn0 -device e1000,netdev=bn0
>>> -object
>>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
>>>
>>> secondary:
>>> -netdev tap,id=bn0 -device e1000,netdev=bn0
>>> -object
>>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
>> Have a quick glance at how secondary mode work. What it does is just
>> forwarding packets between a nic and a socket, qemu socket backend did
>> exact the same job. You could even use socket in primary node and let
>> packet compare module talk to both primary and secondary node.
> If we use qemu socket backend , the same netdev will used by qemu
> socket and
> qemu netfilter. this will against qemu net design. and then, when colo
> do failover,
> secondary do not have backend to use. that's the real problem.
 Then, maybe it's time to implement changing the netdev of a nic. The
 point here is that what secondary mode did is in fact a netdev backend
 instead of a filter ...
>>> Currently, you are right. in colo-proxy V2 code, I just compare IP
>>> packet to
>>> decide whether to do checkpoint.
>>> But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it.
>>> because that can reduce frequency of checkpoint and improve
>>> performance. To keep tcp connection well, colo secondary need to record
>>> primary guest's init seq and adjust secondary guest's ack. if colo do
>>> failover,
>>> secondary also need do this to old tcp connection. qemu socket
>>> can't do this job.
>> So a question here: is it a must to do things (e.g TCP analysis stuffs)
>> at secondary? Looks like we could do this at primary node. And I saw
>> you're doing packet comparing in primary node, any advantages of doing
>> this in primary instead of secondary?
> It needs to do this on the secondary; the trick is that things like TCP 
> sequence
> numbers are likely to be different on the primary and secondary; the kernel 
> colo-proxy
> implementation solved this problem by rewriting the sequence numbers on
> the secondary to match the primary, after a failover, the secondary has
> to keep doing that rewrite to ensure existing connections are OK.
> Thus it's holding some state about the current connections.

I see.

> I think also, to be able to do a 2nd failover (i.e. recover from the 1st 
> failure
> and then sometime later have another) you'd have to sync this
> state over to a new host, so again that says the state needs to be part of
> qemu or at least easily available to it.
>
> Dave

Right, if it does thing like tcp seq rewrite (which is missed in current
version), it works much more like a netfilter. Wonder if the function is
generic enough for users other than colo.

Thanks

>
>>> and another problem is do failover, if we use qemu socket
>>> to be backend in secondary, when colo do failover, I don't know how to
>>> change
>>> secondary be a normal qemu, if you know, please tell me.
>> Current qemu couldn't do this, but I mean we implement something like
>> nic_change_backend which can change nic's peer(s). With this, in
>> secondary, we can replace the socket backend with whatever you want (e.g
>> tap or other).
>>
>> Thanks
>>
>>>
>>> Thanks for your revew
>>> zhangchen 
> --
> Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 01/04/2016 04:16 PM, Zhang Chen wrote:
>
>
> On 01/04/2016 01:37 PM, Jason Wang wrote:
>>
>> On 12/31/2015 04:40 PM, Zhang Chen wrote:
>>>
>>> On 12/31/2015 10:36 AM, Jason Wang wrote:
 On 12/22/2015 06:42 PM, Zhang Chen wrote:
> From: zhangchen 
>
> Hi,all
>
> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
> based on qemu netfilter and it's a plugin for qemu netfilter. the
> function
> keep Secondary VM connect normal to Primary VM and compare packets
> sent by PVM to sent by SVM.if the packet difference,notify COLO do
> checkpoint and send all primary packet has queued.
 Thanks for the work. I don't object this method but still not
 convinced
 that qemu is the best place for this.

 As been raised in the past discussion, it's almost impossible to
 cooperate with vhost backends. If we want this to be used in
 production
 environment, need to think of a solution for vhost. There's no such
 worry if we decouple this from qemu.

> You can also get the series from:
>
> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
>
>
>
> Usage:
>
> primary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object
> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
>
> secondary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object
> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
 Have a quick glance at how secondary mode work. What it does is just
 forwarding packets between a nic and a socket, qemu socket backend did
 exact the same job. You could even use socket in primary node and let
 packet compare module talk to both primary and secondary node.
>>> If we use qemu socket backend , the same netdev will used by qemu
>>> socket and
>>> qemu netfilter. this will against qemu net design. and then, when colo
>>> do failover,
>>> secondary do not have backend to use. that's the real problem.
>> Then, maybe it's time to implement changing the netdev of a nic. The
>> point here is that what secondary mode did is in fact a netdev backend
>> instead of a filter ...
>
> Currently, you are right. in colo-proxy V2 code, I just compare IP
> packet to
> decide whether to do checkpoint.
> But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it.
> because that can reduce frequency of checkpoint and improve
> performance. To keep tcp connection well, colo secondary need to record
> primary guest's init seq and adjust secondary guest's ack. if colo do
> failover,
> secondary also need do this to old tcp connection. qemu socket
> can't do this job.

So a question here: is it a must to do things (e.g TCP analysis stuffs)
at secondary? Looks like we could do this at primary node. And I saw
you're doing packet comparing in primary node, any advantages of doing
this in primary instead of secondary?

> and another problem is do failover, if we use qemu socket
> to be backend in secondary, when colo do failover, I don't know how to
> change
> secondary be a normal qemu, if you know, please tell me.

Current qemu couldn't do this, but I mean we implement something like
nic_change_backend which can change nic's peer(s). With this, in
secondary, we can replace the socket backend with whatever you want (e.g
tap or other).

Thanks

>
>
> Thanks for your revew
> zhangchen 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Zhang Chen]

On 01/04/2016 05:46 PM, Jason Wang wrote:


On 01/04/2016 04:16 PM, Zhang Chen wrote:


On 01/04/2016 01:37 PM, Jason Wang wrote:

On 12/31/2015 04:40 PM, Zhang Chen wrote:

On 12/31/2015 10:36 AM, Jason Wang wrote:

On 12/22/2015 06:42 PM, Zhang Chen wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the
function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.

Thanks for the work. I don't object this method but still not
convinced
that qemu is the best place for this.

As been raised in the past discussion, it's almost impossible to
cooperate with vhost backends. If we want this to be used in
production
environment, need to think of a solution for vhost. There's no such
worry if we decouple this from qemu.


You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2



Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object
colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object
colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port

Have a quick glance at how secondary mode work. What it does is just
forwarding packets between a nic and a socket, qemu socket backend did
exact the same job. You could even use socket in primary node and let
packet compare module talk to both primary and secondary node.

If we use qemu socket backend , the same netdev will used by qemu
socket and
qemu netfilter. this will against qemu net design. and then, when colo
do failover,
secondary do not have backend to use. that's the real problem.

Then, maybe it's time to implement changing the netdev of a nic. The
point here is that what secondary mode did is in fact a netdev backend
instead of a filter ...

Currently, you are right. in colo-proxy V2 code, I just compare IP
packet to
decide whether to do checkpoint.
But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it.
because that can reduce frequency of checkpoint and improve
performance. To keep tcp connection well, colo secondary need to record
primary guest's init seq and adjust secondary guest's ack. if colo do
failover,
secondary also need do this to old tcp connection. qemu socket
can't do this job.

So a question here: is it a must to do things (e.g TCP analysis stuffs)
at secondary? Looks like we could do this at primary node. And I saw
you're doing packet comparing in primary node, any advantages of doing
this in primary instead of secondary?


We think must  to do this in secondary, because if colo do 
failover,secondary
must continues do TCP analysis stuffs to before tcp connection(if not, 
tcp connection
will disconnect in that time), in this time primary already down or 
disconnect to
secondary.so we can't make primary do this  TCP analysis stuffs.it can 
not ensure

FT function.

Thanks
zhangchen


and another problem is do failover, if we use qemu socket
to be backend in secondary, when colo do failover, I don't know how to
change
secondary be a normal qemu, if you know, please tell me.

Current qemu couldn't do this, but I mean we implement something like
nic_change_backend which can change nic's peer(s). With this, in
secondary, we can replace the socket backend with whatever you want (e.g
tap or other).

Thanks



Thanks for your revew
zhangchen



.



--
Thanks
zhangchen

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Zhang Chen]

On 01/04/2016 01:37 PM, Jason Wang wrote:


On 12/31/2015 04:40 PM, Zhang Chen wrote:


On 12/31/2015 10:36 AM, Jason Wang wrote:

On 12/22/2015 06:42 PM, Zhang Chen wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the
function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.

Thanks for the work. I don't object this method but still not convinced
that qemu is the best place for this.

As been raised in the past discussion, it's almost impossible to
cooperate with vhost backends. If we want this to be used in production
environment, need to think of a solution for vhost. There's no such
worry if we decouple this from qemu.


You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2


Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object
colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object
colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port

Have a quick glance at how secondary mode work. What it does is just
forwarding packets between a nic and a socket, qemu socket backend did
exact the same job. You could even use socket in primary node and let
packet compare module talk to both primary and secondary node.

If we use qemu socket backend , the same netdev will used by qemu
socket and
qemu netfilter. this will against qemu net design. and then, when colo
do failover,
secondary do not have backend to use. that's the real problem.

Then, maybe it's time to implement changing the netdev of a nic. The
point here is that what secondary mode did is in fact a netdev backend
instead of a filter ...


Currently, you are right. in colo-proxy V2 code, I just compare IP packet to
decide whether to do checkpoint.
But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it.
because that can reduce frequency of checkpoint and improve
performance. To keep tcp connection well, colo secondary need to record
primary guest's init seq and adjust secondary guest's ack. if colo do 
failover,

secondary also need do this to old tcp connection. qemu socket
can't do this job. and another problem is do failover, if we use qemu socket
to be backend in secondary, when colo do failover, I don't know how to 
change

secondary be a normal qemu, if you know, please tell me.


Thanks for your revew
zhangchen



Thanks
zhangchen


NOTE:
queue must set "all". See enum NetFilterDirection for detail.
colo-proxy need queue all packets
colo-proxy V2 just can compare ip packet


## Background

COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
Service)
project is a high availability solution. Both Primary VM (PVM) and
Secondary VM
(SVM) run in parallel. They receive the same request from client,
and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint
(on
demand)is conducted.

Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0

COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and
finding out
whether they are identical, we introduce a new module to qemu
networking
called colo-proxy.

V2:
rebase colo-proxy with qemu-colo-v2.2-periodic-mode
fix dave's comments
fix wency's comments
fix zhanghailiang's comments

v1:
initial patch.



zhangchen (10):
Init colo-proxy object based on netfilter
Jhash: add linux kernel jhashtable in qemu
Colo-proxy: add colo-proxy framework
Colo-proxy: add data structure and jhash func
net/colo-proxy: Add colo interface to use proxy
net/colo-proxy: add socket used by forward func
net/colo-proxy: Add packet enqueue & handle func
net/colo-proxy: Handle packet and connection
net/colo-proxy: Compare pri pkt to sec pkt
net/colo-proxy: Colo-proxy do checkpoint and clear

   include/qemu/jhash.h |  61 
   net/Makefile.objs|   1 +
   net/colo-proxy.c | 939
+++
   net/colo-proxy.h |  24 ++
   qemu-options.hx  |   6 +
   trace-events |   8 +
   vl.c |   3 +-
   7 files changed, 1041 insertions(+), 1 deletion(-)
   create mode 100644 include/qemu/jhash.h
   create mode 100644 net/colo-proxy.c
   create mode 100644 net/colo-proxy.h



.





.



--
Thanks
zhangchen

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Dr. David Alan Gilbert]

* Jason Wang (jasow...@redhat.com) wrote:
> 
> 
> On 01/04/2016 04:16 PM, Zhang Chen wrote:
> >
> >
> > On 01/04/2016 01:37 PM, Jason Wang wrote:
> >>
> >> On 12/31/2015 04:40 PM, Zhang Chen wrote:
> >>>
> >>> On 12/31/2015 10:36 AM, Jason Wang wrote:
>  On 12/22/2015 06:42 PM, Zhang Chen wrote:
> > From: zhangchen 
> >
> > Hi,all
> >
> > This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
> > based on qemu netfilter and it's a plugin for qemu netfilter. the
> > function
> > keep Secondary VM connect normal to Primary VM and compare packets
> > sent by PVM to sent by SVM.if the packet difference,notify COLO do
> > checkpoint and send all primary packet has queued.
>  Thanks for the work. I don't object this method but still not
>  convinced
>  that qemu is the best place for this.
> 
>  As been raised in the past discussion, it's almost impossible to
>  cooperate with vhost backends. If we want this to be used in
>  production
>  environment, need to think of a solution for vhost. There's no such
>  worry if we decouple this from qemu.
> 
> > You can also get the series from:
> >
> > https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
> >
> >
> >
> > Usage:
> >
> > primary:
> > -netdev tap,id=bn0 -device e1000,netdev=bn0
> > -object
> > colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
> >
> > secondary:
> > -netdev tap,id=bn0 -device e1000,netdev=bn0
> > -object
> > colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
>  Have a quick glance at how secondary mode work. What it does is just
>  forwarding packets between a nic and a socket, qemu socket backend did
>  exact the same job. You could even use socket in primary node and let
>  packet compare module talk to both primary and secondary node.
> >>> If we use qemu socket backend , the same netdev will used by qemu
> >>> socket and
> >>> qemu netfilter. this will against qemu net design. and then, when colo
> >>> do failover,
> >>> secondary do not have backend to use. that's the real problem.
> >> Then, maybe it's time to implement changing the netdev of a nic. The
> >> point here is that what secondary mode did is in fact a netdev backend
> >> instead of a filter ...
> >
> > Currently, you are right. in colo-proxy V2 code, I just compare IP
> > packet to
> > decide whether to do checkpoint.
> > But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it.
> > because that can reduce frequency of checkpoint and improve
> > performance. To keep tcp connection well, colo secondary need to record
> > primary guest's init seq and adjust secondary guest's ack. if colo do
> > failover,
> > secondary also need do this to old tcp connection. qemu socket
> > can't do this job.
> 
> So a question here: is it a must to do things (e.g TCP analysis stuffs)
> at secondary? Looks like we could do this at primary node. And I saw
> you're doing packet comparing in primary node, any advantages of doing
> this in primary instead of secondary?

It needs to do this on the secondary; the trick is that things like TCP sequence
numbers are likely to be different on the primary and secondary; the kernel 
colo-proxy
implementation solved this problem by rewriting the sequence numbers on
the secondary to match the primary, after a failover, the secondary has
to keep doing that rewrite to ensure existing connections are OK.
Thus it's holding some state about the current connections.
I think also, to be able to do a 2nd failover (i.e. recover from the 1st failure
and then sometime later have another) you'd have to sync this
state over to a new host, so again that says the state needs to be part of
qemu or at least easily available to it.

Dave

> > and another problem is do failover, if we use qemu socket
> > to be backend in secondary, when colo do failover, I don't know how to
> > change
> > secondary be a normal qemu, if you know, please tell me.
> 
> Current qemu couldn't do this, but I mean we implement something like
> nic_change_backend which can change nic's peer(s). With this, in
> secondary, we can replace the socket backend with whatever you want (e.g
> tap or other).
> 
> Thanks
> 
> >
> >
> > Thanks for your revew
> > zhangchen 
> 
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 12/31/2015 04:02 PM, Li Zhijian wrote:
>
>
> On 12/31/2015 10:36 AM, Jason Wang wrote:
>>
>>
>> On 12/22/2015 06:42 PM, Zhang Chen wrote:
>>> From: zhangchen 
>>>
>>> Hi,all
>>>
>>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
>>> based on qemu netfilter and it's a plugin for qemu netfilter. the
>>> function
>>> keep Secondary VM connect normal to Primary VM and compare packets
>>> sent by PVM to sent by SVM.if the packet difference,notify COLO do
>>> checkpoint and send all primary packet has queued.
>>
>> Thanks for the work. I don't object this method but still not convinced
>> that qemu is the best place for this.
>>
>> As been raised in the past discussion, it's almost impossible to
>> cooperate with vhost backends. If we want this to be used in production
>> environment, need to think of a solution for vhost. There's no such
>> worry if we decouple this from qemu.
>
> Yes, I agree with you. But not everything is perfect at the beginning.
> If we implement proxy as kernel modules, maybe some one will ask how
> about the packet(e.g. vhost-user) that host kernel can not touch.

Then I think the best place is still in userspace but not qemu.  With
this the packet comparing module can easily accept the mirrored traffic
from both kernel and userspace. And qemu part can focus on the
infrastructures to support them (e.g mirroring traffic to a socket or
elsewhere).

>
> As you said, colo-proxy in qemu will not support vhost scene, but it
> can make
> colo more easier to be used so that more and more pepole will join us
> to test it.
> I'm looking forward to that day.

Agree, so moving this out of qemu can greatly reduce the complexity and
make it easier to be merged.

>
> if everything goes fine, I think colo can support vhost scene in other
> way(such as
> introduce extra proxy module in kernel) in the feature.

I believe we don't want duplicate codes & bugs(fixes) in two places.

>
> So, I think colo-proxy in qemu is a good choice for current COLO.
>
> Thanks
> Li 

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 12/31/2015 04:40 PM, Zhang Chen wrote:
>
>
> On 12/31/2015 10:36 AM, Jason Wang wrote:
>>
>> On 12/22/2015 06:42 PM, Zhang Chen wrote:
>>> From: zhangchen 
>>>
>>> Hi,all
>>>
>>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
>>> based on qemu netfilter and it's a plugin for qemu netfilter. the
>>> function
>>> keep Secondary VM connect normal to Primary VM and compare packets
>>> sent by PVM to sent by SVM.if the packet difference,notify COLO do
>>> checkpoint and send all primary packet has queued.
>> Thanks for the work. I don't object this method but still not convinced
>> that qemu is the best place for this.
>>
>> As been raised in the past discussion, it's almost impossible to
>> cooperate with vhost backends. If we want this to be used in production
>> environment, need to think of a solution for vhost. There's no such
>> worry if we decouple this from qemu.
>>
>>> You can also get the series from:
>>>
>>> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
>>>
>>>
>>> Usage:
>>>
>>> primary:
>>> -netdev tap,id=bn0 -device e1000,netdev=bn0
>>> -object
>>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
>>>
>>> secondary:
>>> -netdev tap,id=bn0 -device e1000,netdev=bn0
>>> -object
>>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
>> Have a quick glance at how secondary mode work. What it does is just
>> forwarding packets between a nic and a socket, qemu socket backend did
>> exact the same job. You could even use socket in primary node and let
>> packet compare module talk to both primary and secondary node.
>
> If we use qemu socket backend , the same netdev will used by qemu
> socket and
> qemu netfilter. this will against qemu net design. and then, when colo
> do failover,
> secondary do not have backend to use. that's the real problem.

Then, maybe it's time to implement changing the netdev of a nic. The
point here is that what secondary mode did is in fact a netdev backend
instead of a filter ...

>
>
> Thanks
> zhangchen
>
>>> NOTE:
>>> queue must set "all". See enum NetFilterDirection for detail.
>>> colo-proxy need queue all packets
>>> colo-proxy V2 just can compare ip packet
>>>
>>>
>>> ## Background
>>>
>>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
>>> Service)
>>> project is a high availability solution. Both Primary VM (PVM) and
>>> Secondary VM
>>> (SVM) run in parallel. They receive the same request from client,
>>> and generate
>>> responses in parallel too. If the response packets from PVM and SVM are
>>> identical, they are released immediately. Otherwise, a VM checkpoint
>>> (on
>>> demand)is conducted.
>>>
>>> Paper:
>>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
>>>
>>> COLO on Xen:
>>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>>>
>>> COLO on Qemu/KVM:
>>> http://wiki.qemu.org/Features/COLO
>>>
>>> By the needs of capturing response packets from PVM and SVM and
>>> finding out
>>> whether they are identical, we introduce a new module to qemu
>>> networking
>>> called colo-proxy.
>>>
>>> V2:
>>>rebase colo-proxy with qemu-colo-v2.2-periodic-mode
>>>fix dave's comments
>>>fix wency's comments
>>>fix zhanghailiang's comments
>>>
>>> v1:
>>>initial patch.
>>>
>>>
>>>
>>> zhangchen (10):
>>>Init colo-proxy object based on netfilter
>>>Jhash: add linux kernel jhashtable in qemu
>>>Colo-proxy: add colo-proxy framework
>>>Colo-proxy: add data structure and jhash func
>>>net/colo-proxy: Add colo interface to use proxy
>>>net/colo-proxy: add socket used by forward func
>>>net/colo-proxy: Add packet enqueue & handle func
>>>net/colo-proxy: Handle packet and connection
>>>net/colo-proxy: Compare pri pkt to sec pkt
>>>net/colo-proxy: Colo-proxy do checkpoint and clear
>>>
>>>   include/qemu/jhash.h |  61 
>>>   net/Makefile.objs|   1 +
>>>   net/colo-proxy.c | 939
>>> +++
>>>   net/colo-proxy.h |  24 ++
>>>   qemu-options.hx  |   6 +
>>>   trace-events |   8 +
>>>   vl.c |   3 +-
>>>   7 files changed, 1041 insertions(+), 1 deletion(-)
>>>   create mode 100644 include/qemu/jhash.h
>>>   create mode 100644 net/colo-proxy.c
>>>   create mode 100644 net/colo-proxy.h
>>>
>>
>>
>> .
>>
>

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Li Zhijian]

On 12/31/2015 10:36 AM, Jason Wang wrote:



On 12/22/2015 06:42 PM, Zhang Chen wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.


Thanks for the work. I don't object this method but still not convinced
that qemu is the best place for this.

As been raised in the past discussion, it's almost impossible to
cooperate with vhost backends. If we want this to be used in production
environment, need to think of a solution for vhost. There's no such
worry if we decouple this from qemu.


Yes, I agree with you. But not everything is perfect at the beginning.
If we implement proxy as kernel modules, maybe some one will ask how
about the packet(e.g. vhost-user) that host kernel can not touch.

As you said, colo-proxy in qemu will not support vhost scene, but it can make
colo more easier to be used so that more and more pepole will join us to test 
it.
I'm looking forward to that day.

if everything goes fine, I think colo can support vhost scene in other way(such 
as
introduce extra proxy module in kernel) in the feature.

So, I think colo-proxy in qemu is a good choice for current COLO.

Thanks
Li





You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2

Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port


Have a quick glance at how secondary mode work. What it does is just
forwarding packets between a nic and a socket, qemu socket backend did
exact the same job. You could even use socket in primary node and let
packet compare module talk to both primary and secondary node.



NOTE:
queue must set "all". See enum NetFilterDirection for detail.
colo-proxy need queue all packets
colo-proxy V2 just can compare ip packet


## Background

COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on
demand)is conducted.

Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0

COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking
called colo-proxy.

V2:
   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
   fix dave's comments
   fix wency's comments
   fix zhanghailiang's comments

v1:
   initial patch.



zhangchen (10):
   Init colo-proxy object based on netfilter
   Jhash: add linux kernel jhashtable in qemu
   Colo-proxy: add colo-proxy framework
   Colo-proxy: add data structure and jhash func
   net/colo-proxy: Add colo interface to use proxy
   net/colo-proxy: add socket used by forward func
   net/colo-proxy: Add packet enqueue & handle func
   net/colo-proxy: Handle packet and connection
   net/colo-proxy: Compare pri pkt to sec pkt
   net/colo-proxy: Colo-proxy do checkpoint and clear

  include/qemu/jhash.h |  61 
  net/Makefile.objs|   1 +
  net/colo-proxy.c | 939 +++
  net/colo-proxy.h |  24 ++
  qemu-options.hx  |   6 +
  trace-events |   8 +
  vl.c |   3 +-
  7 files changed, 1041 insertions(+), 1 deletion(-)
  create mode 100644 include/qemu/jhash.h
  create mode 100644 net/colo-proxy.c
  create mode 100644 net/colo-proxy.h






.



--
Best regards.
Li Zhijian (8555)

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Zhang Chen]

On 12/31/2015 10:36 AM, Jason Wang wrote:


On 12/22/2015 06:42 PM, Zhang Chen wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.

Thanks for the work. I don't object this method but still not convinced
that qemu is the best place for this.

As been raised in the past discussion, it's almost impossible to
cooperate with vhost backends. If we want this to be used in production
environment, need to think of a solution for vhost. There's no such
worry if we decouple this from qemu.


You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2

Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port

Have a quick glance at how secondary mode work. What it does is just
forwarding packets between a nic and a socket, qemu socket backend did
exact the same job. You could even use socket in primary node and let
packet compare module talk to both primary and secondary node.


If we use qemu socket backend , the same netdev will used by qemu socket 
and
qemu netfilter. this will against qemu net design. and then, when colo 
do failover,

secondary do not have backend to use. that's the real problem.


Thanks
zhangchen


NOTE:
queue must set "all". See enum NetFilterDirection for detail.
colo-proxy need queue all packets
colo-proxy V2 just can compare ip packet


## Background

COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on
demand)is conducted.

Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0

COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking
called colo-proxy.

V2:
   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
   fix dave's comments
   fix wency's comments
   fix zhanghailiang's comments

v1:
   initial patch.



zhangchen (10):
   Init colo-proxy object based on netfilter
   Jhash: add linux kernel jhashtable in qemu
   Colo-proxy: add colo-proxy framework
   Colo-proxy: add data structure and jhash func
   net/colo-proxy: Add colo interface to use proxy
   net/colo-proxy: add socket used by forward func
   net/colo-proxy: Add packet enqueue & handle func
   net/colo-proxy: Handle packet and connection
   net/colo-proxy: Compare pri pkt to sec pkt
   net/colo-proxy: Colo-proxy do checkpoint and clear

  include/qemu/jhash.h |  61 
  net/Makefile.objs|   1 +
  net/colo-proxy.c | 939 +++
  net/colo-proxy.h |  24 ++
  qemu-options.hx  |   6 +
  trace-events |   8 +
  vl.c |   3 +-
  7 files changed, 1041 insertions(+), 1 deletion(-)
  create mode 100644 include/qemu/jhash.h
  create mode 100644 net/colo-proxy.c
  create mode 100644 net/colo-proxy.h




.



--
Thanks
zhangchen

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 12/22/2015 06:42 PM, Zhang Chen wrote:
> From: zhangchen 
>
> Hi,all
>
> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
> based on qemu netfilter and it's a plugin for qemu netfilter. the function
> keep Secondary VM connect normal to Primary VM and compare packets
> sent by PVM to sent by SVM.if the packet difference,notify COLO do
> checkpoint and send all primary packet has queued.

Thanks for the work. I don't object this method but still not convinced
that qemu is the best place for this.

As been raised in the past discussion, it's almost impossible to
cooperate with vhost backends. If we want this to be used in production
environment, need to think of a solution for vhost. There's no such
worry if we decouple this from qemu.

>
> You can also get the series from:
>
> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
>
> Usage:
>
> primary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
>
> secondary:
> -netdev tap,id=bn0 -device e1000,netdev=bn0
> -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port 

Have a quick glance at how secondary mode work. What it does is just
forwarding packets between a nic and a socket, qemu socket backend did
exact the same job. You could even use socket in primary node and let
packet compare module talk to both primary and secondary node.

>
> NOTE:
> queue must set "all". See enum NetFilterDirection for detail.
> colo-proxy need queue all packets
> colo-proxy V2 just can compare ip packet
>
>
> ## Background
>
> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
> project is a high availability solution. Both Primary VM (PVM) and Secondary 
> VM
> (SVM) run in parallel. They receive the same request from client, and generate
> responses in parallel too. If the response packets from PVM and SVM are
> identical, they are released immediately. Otherwise, a VM checkpoint (on
> demand)is conducted.
>
> Paper:
> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
>
> COLO on Xen:
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>
> COLO on Qemu/KVM:
> http://wiki.qemu.org/Features/COLO
>
> By the needs of capturing response packets from PVM and SVM and finding out
> whether they are identical, we introduce a new module to qemu networking
> called colo-proxy.
>
> V2:
>   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
>   fix dave's comments
>   fix wency's comments
>   fix zhanghailiang's comments
>
> v1:
>   initial patch.
>
>
>
> zhangchen (10):
>   Init colo-proxy object based on netfilter
>   Jhash: add linux kernel jhashtable in qemu
>   Colo-proxy: add colo-proxy framework
>   Colo-proxy: add data structure and jhash func
>   net/colo-proxy: Add colo interface to use proxy
>   net/colo-proxy: add socket used by forward func
>   net/colo-proxy: Add packet enqueue & handle func
>   net/colo-proxy: Handle packet and connection
>   net/colo-proxy: Compare pri pkt to sec pkt
>   net/colo-proxy: Colo-proxy do checkpoint and clear
>
>  include/qemu/jhash.h |  61 
>  net/Makefile.objs|   1 +
>  net/colo-proxy.c | 939 
> +++
>  net/colo-proxy.h |  24 ++
>  qemu-options.hx  |   6 +
>  trace-events |   8 +
>  vl.c |   3 +-
>  7 files changed, 1041 insertions(+), 1 deletion(-)
>  create mode 100644 include/qemu/jhash.h
>  create mode 100644 net/colo-proxy.c
>  create mode 100644 net/colo-proxy.h
>

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Zhang Chen]

Hi~
Just a small ping...
No news for a week.
Colo proxy is a part of COLO project, we need review and comments.


Thanks
zhangchen


On 12/22/2015 06:42 PM, Zhang Chen wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.

You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2

Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port

NOTE:
queue must set "all". See enum NetFilterDirection for detail.
colo-proxy need queue all packets
colo-proxy V2 just can compare ip packet


## Background

COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on
demand)is conducted.

Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0

COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking
called colo-proxy.

V2:
   rebase colo-proxy with qemu-colo-v2.2-periodic-mode
   fix dave's comments
   fix wency's comments
   fix zhanghailiang's comments

v1:
   initial patch.



zhangchen (10):
   Init colo-proxy object based on netfilter
   Jhash: add linux kernel jhashtable in qemu
   Colo-proxy: add colo-proxy framework
   Colo-proxy: add data structure and jhash func
   net/colo-proxy: Add colo interface to use proxy
   net/colo-proxy: add socket used by forward func
   net/colo-proxy: Add packet enqueue & handle func
   net/colo-proxy: Handle packet and connection
   net/colo-proxy: Compare pri pkt to sec pkt
   net/colo-proxy: Colo-proxy do checkpoint and clear

  include/qemu/jhash.h |  61 
  net/Makefile.objs|   1 +
  net/colo-proxy.c | 939 +++
  net/colo-proxy.h |  24 ++
  qemu-options.hx  |   6 +
  trace-events |   8 +
  vl.c |   3 +-
  7 files changed, 1041 insertions(+), 1 deletion(-)
  create mode 100644 include/qemu/jhash.h
  create mode 100644 net/colo-proxy.c
  create mode 100644 net/colo-proxy.h



--
Thanks
zhangchen

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Zhang Chen]

On 12/29/2015 02:58 PM, Jason Wang wrote:


On 12/29/2015 02:31 PM, Zhang Chen wrote:

Hi~
Just a small ping...
No news for a week.
Colo proxy is a part of COLO project, we need review and comments.


Thanks
zhangchen

Hi, will find sometime to review this this week.

Thanks


Thanks very much for your help~~



On 12/22/2015 06:42 PM, Zhang Chen wrote:

From: zhangchen 

Hi,all

This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
based on qemu netfilter and it's a plugin for qemu netfilter. the
function
keep Secondary VM connect normal to Primary VM and compare packets
sent by PVM to sent by SVM.if the packet difference,notify COLO do
checkpoint and send all primary packet has queued.

You can also get the series from:

https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2


Usage:

primary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object
colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port

secondary:
-netdev tap,id=bn0 -device e1000,netdev=bn0
-object
colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port

NOTE:
queue must set "all". See enum NetFilterDirection for detail.
colo-proxy need queue all packets
colo-proxy V2 just can compare ip packet


## Background

COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
Service)
project is a high availability solution. Both Primary VM (PVM) and
Secondary VM
(SVM) run in parallel. They receive the same request from client, and
generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on
demand)is conducted.

Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0

COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and
finding out
whether they are identical, we introduce a new module to qemu networking
called colo-proxy.

V2:
rebase colo-proxy with qemu-colo-v2.2-periodic-mode
fix dave's comments
fix wency's comments
fix zhanghailiang's comments

v1:
initial patch.



zhangchen (10):
Init colo-proxy object based on netfilter
Jhash: add linux kernel jhashtable in qemu
Colo-proxy: add colo-proxy framework
Colo-proxy: add data structure and jhash func
net/colo-proxy: Add colo interface to use proxy
net/colo-proxy: add socket used by forward func
net/colo-proxy: Add packet enqueue & handle func
net/colo-proxy: Handle packet and connection
net/colo-proxy: Compare pri pkt to sec pkt
net/colo-proxy: Colo-proxy do checkpoint and clear

   include/qemu/jhash.h |  61 
   net/Makefile.objs|   1 +
   net/colo-proxy.c | 939
+++
   net/colo-proxy.h |  24 ++
   qemu-options.hx  |   6 +
   trace-events |   8 +
   vl.c |   3 +-
   7 files changed, 1041 insertions(+), 1 deletion(-)
   create mode 100644 include/qemu/jhash.h
   create mode 100644 net/colo-proxy.c
   create mode 100644 net/colo-proxy.h




.



--
Thanks
zhangchen

Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter

[Jason Wang]

On 12/29/2015 02:31 PM, Zhang Chen wrote:
> Hi~
> Just a small ping...
> No news for a week.
> Colo proxy is a part of COLO project, we need review and comments.
>
>
> Thanks
> zhangchen

Hi, will find sometime to review this this week.

Thanks

>
>
> On 12/22/2015 06:42 PM, Zhang Chen wrote:
>> From: zhangchen 
>>
>> Hi,all
>>
>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO,
>> based on qemu netfilter and it's a plugin for qemu netfilter. the
>> function
>> keep Secondary VM connect normal to Primary VM and compare packets
>> sent by PVM to sent by SVM.if the packet difference,notify COLO do
>> checkpoint and send all primary packet has queued.
>>
>> You can also get the series from:
>>
>> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2
>>
>>
>> Usage:
>>
>> primary:
>> -netdev tap,id=bn0 -device e1000,netdev=bn0
>> -object
>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port
>>
>> secondary:
>> -netdev tap,id=bn0 -device e1000,netdev=bn0
>> -object
>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port
>>
>> NOTE:
>> queue must set "all". See enum NetFilterDirection for detail.
>> colo-proxy need queue all packets
>> colo-proxy V2 just can compare ip packet
>>
>>
>> ## Background
>>
>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
>> Service)
>> project is a high availability solution. Both Primary VM (PVM) and
>> Secondary VM
>> (SVM) run in parallel. They receive the same request from client, and
>> generate
>> responses in parallel too. If the response packets from PVM and SVM are
>> identical, they are released immediately. Otherwise, a VM checkpoint (on
>> demand)is conducted.
>>
>> Paper:
>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
>>
>> COLO on Xen:
>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>>
>> COLO on Qemu/KVM:
>> http://wiki.qemu.org/Features/COLO
>>
>> By the needs of capturing response packets from PVM and SVM and
>> finding out
>> whether they are identical, we introduce a new module to qemu networking
>> called colo-proxy.
>>
>> V2:
>>rebase colo-proxy with qemu-colo-v2.2-periodic-mode
>>fix dave's comments
>>fix wency's comments
>>fix zhanghailiang's comments
>>
>> v1:
>>initial patch.
>>
>>
>>
>> zhangchen (10):
>>Init colo-proxy object based on netfilter
>>Jhash: add linux kernel jhashtable in qemu
>>Colo-proxy: add colo-proxy framework
>>Colo-proxy: add data structure and jhash func
>>net/colo-proxy: Add colo interface to use proxy
>>net/colo-proxy: add socket used by forward func
>>net/colo-proxy: Add packet enqueue & handle func
>>net/colo-proxy: Handle packet and connection
>>net/colo-proxy: Compare pri pkt to sec pkt
>>net/colo-proxy: Colo-proxy do checkpoint and clear
>>
>>   include/qemu/jhash.h |  61 
>>   net/Makefile.objs|   1 +
>>   net/colo-proxy.c | 939
>> +++
>>   net/colo-proxy.h |  24 ++
>>   qemu-options.hx  |   6 +
>>   trace-events |   8 +
>>   vl.c |   3 +-
>>   7 files changed, 1041 insertions(+), 1 deletion(-)
>>   create mode 100644 include/qemu/jhash.h
>>   create mode 100644 net/colo-proxy.c
>>   create mode 100644 net/colo-proxy.h
>>
>