Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
* Li Zhijian (lizhij...@cn.fujitsu.com) wrote: > > > On 03/01/2016 04:04 AM, Dr. David Alan Gilbert wrote: > >* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote: > >>From: zhangchen> >> > >>Hi,all > >> > >>This patch add an colo-proxy object, COLO-Proxy is a part of COLO, > >>based on qemu netfilter and it's a plugin for qemu netfilter. the function > >>keep Secondary VM connect normal to Primary VM and compare packets > >>sent by PVM to sent by SVM.if the packet difference,notify COLO do > >>checkpoint and send all primary packet has queued. > > > >Hi Zhangchen, > > How are you dealing with the IP 'identification' field? > >It's a very very random field, and not just the initial value in the > >connection. > >I can't see how the kernel colo-proxy dealt with it either; but I think it's > >comparison was after defragementation so probably ignored the identification > >field > You are right, most kernel colo-proxy code is working at mangle table (after > defrag). > and colo proxy only compare the contents of L4(TCP/UDP) excluding IP > identification. > > > - wouldn't that confuse a client at failover? > Err..., instersting question. > > for example, a COLO including primay(PVM) adn secondary(SVM) > 1. primay is sending a compared P_IP packet(identification=0x12345, split to > IP_s1, IP_s2..IP_s100) to client > 2. client is receiving the ip segment(but IP_s2, IP_s50, IP_s80..IP_s99 are > missing) >and primary host is down. > 3. secondary VM takeover, and send a S_IP packet(ip contents is always same > as at PVM) > > in step 3, > if the S_IP identification isn't 0x12345, client will drop the ip segment at > step 2 because of defrag timeout. So that triggers a timeout (30 second?) - hmm OK, a bit slow but OK. > if the S_IP identification isn 0x12345, client may mix the segment from PVM > or SVM(just like segment come > from different router), but that's Okay, because we have ensured the ip > contents is identical. Could the S_IP identification match a later/earlier fragment? Dave > so IMO, it will not confuse the client at failover. > > > > >Dave > > > >>You can also get the series from: > >> > >>https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 > >> > >>Usage: > >> > >>primary: > >>-netdev tap,id=bn0 -device e1000,netdev=bn0 > >>-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port > >> > >>secondary: > >>-netdev tap,id=bn0 -device e1000,netdev=bn0 > >>-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port > >> > >>NOTE: > >>queue must set "all". See enum NetFilterDirection for detail. > >>colo-proxy need queue all packets > >>colo-proxy V2 just can compare ip packet > >> > >> > >>## Background > >> > >>COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop > >>Service) > >>project is a high availability solution. Both Primary VM (PVM) and > >>Secondary VM > >>(SVM) run in parallel. They receive the same request from client, and > >>generate > >>responses in parallel too. If the response packets from PVM and SVM are > >>identical, they are released immediately. Otherwise, a VM checkpoint (on > >>demand)is conducted. > >> > >>Paper: > >>http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > >> > >>COLO on Xen: > >>http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > >> > >>COLO on Qemu/KVM: > >>http://wiki.qemu.org/Features/COLO > >> > >>By the needs of capturing response packets from PVM and SVM and finding out > >>whether they are identical, we introduce a new module to qemu networking > >>called colo-proxy. > >> > >>V2: > >> rebase colo-proxy with qemu-colo-v2.2-periodic-mode > >> fix dave's comments > >> fix wency's comments > >> fix zhanghailiang's comments > >> > >>v1: > >> initial patch. > >> > >> > >> > >>zhangchen (10): > >> Init colo-proxy object based on netfilter > >> Jhash: add linux kernel jhashtable in qemu > >> Colo-proxy: add colo-proxy framework > >> Colo-proxy: add data structure and jhash func > >> net/colo-proxy: Add colo interface to use proxy > >> net/colo-proxy: add socket used by forward func > >> net/colo-proxy: Add packet enqueue & handle func > >> net/colo-proxy: Handle packet and connection > >> net/colo-proxy: Compare pri pkt to sec pkt > >> net/colo-proxy: Colo-proxy do checkpoint and clear > >> > >> include/qemu/jhash.h | 61 > >> net/Makefile.objs| 1 + > >> net/colo-proxy.c | 939 > >> +++ > >> net/colo-proxy.h | 24 ++ > >> qemu-options.hx | 6 + > >> trace-events | 8 + > >> vl.c | 3 +- > >> 7 files changed, 1041 insertions(+), 1 deletion(-) > >> create mode 100644 include/qemu/jhash.h > >> create mode 100644 net/colo-proxy.c > >> create mode 100644 net/colo-proxy.h > >> > >>-- > >>1.9.1 > >> > >> > >> > >-- > >Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 03/01/2016 04:04 AM, Dr. David Alan Gilbert wrote: * Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. Hi Zhangchen, How are you dealing with the IP 'identification' field? It's a very very random field, and not just the initial value in the connection. I can't see how the kernel colo-proxy dealt with it either; but I think it's comparison was after defragementation so probably ignored the identification field You are right, most kernel colo-proxy code is working at mangle table (after defrag). and colo proxy only compare the contents of L4(TCP/UDP) excluding IP identification. - wouldn't that confuse a client at failover? Err..., instersting question. for example, a COLO including primay(PVM) adn secondary(SVM) 1. primay is sending a compared P_IP packet(identification=0x12345, split to IP_s1, IP_s2..IP_s100) to client 2. client is receiving the ip segment(but IP_s2, IP_s50, IP_s80..IP_s99 are missing) and primary host is down. 3. secondary VM takeover, and send a S_IP packet(ip contents is always same as at PVM) in step 3, if the S_IP identification isn't 0x12345, client will drop the ip segment at step 2 because of defrag timeout. if the S_IP identification isn 0x12345, client may mix the segment from PVM or SVM(just like segment come from different router), but that's Okay, because we have ensured the ip contents is identical. so IMO, it will not confuse the client at failover. Dave You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port NOTE: queue must set "all". See enum NetFilterDirection for detail. colo-proxy need queue all packets colo-proxy V2 just can compare ip packet ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand)is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. V2: rebase colo-proxy with qemu-colo-v2.2-periodic-mode fix dave's comments fix wency's comments fix zhanghailiang's comments v1: initial patch. zhangchen (10): Init colo-proxy object based on netfilter Jhash: add linux kernel jhashtable in qemu Colo-proxy: add colo-proxy framework Colo-proxy: add data structure and jhash func net/colo-proxy: Add colo interface to use proxy net/colo-proxy: add socket used by forward func net/colo-proxy: Add packet enqueue & handle func net/colo-proxy: Handle packet and connection net/colo-proxy: Compare pri pkt to sec pkt net/colo-proxy: Colo-proxy do checkpoint and clear include/qemu/jhash.h | 61 net/Makefile.objs| 1 + net/colo-proxy.c | 939 +++ net/colo-proxy.h | 24 ++ qemu-options.hx | 6 + trace-events | 8 + vl.c | 3 +- 7 files changed, 1041 insertions(+), 1 deletion(-) create mode 100644 include/qemu/jhash.h create mode 100644 net/colo-proxy.c create mode 100644 net/colo-proxy.h -- 1.9.1 -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK . -- Best regards. Li Zhijian (8555)
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote: > From: zhangchen> > Hi,all > > This patch add an colo-proxy object, COLO-Proxy is a part of COLO, > based on qemu netfilter and it's a plugin for qemu netfilter. the function > keep Secondary VM connect normal to Primary VM and compare packets > sent by PVM to sent by SVM.if the packet difference,notify COLO do > checkpoint and send all primary packet has queued. Hi Zhangchen, How are you dealing with the IP 'identification' field? It's a very very random field, and not just the initial value in the connection. I can't see how the kernel colo-proxy dealt with it either; but I think it's comparison was after defragementation so probably ignored the identification field - wouldn't that confuse a client at failover? Dave > You can also get the series from: > > https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 > > Usage: > > primary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port > > secondary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port > > NOTE: > queue must set "all". See enum NetFilterDirection for detail. > colo-proxy need queue all packets > colo-proxy V2 just can compare ip packet > > > ## Background > > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) > project is a high availability solution. Both Primary VM (PVM) and Secondary > VM > (SVM) run in parallel. They receive the same request from client, and generate > responses in parallel too. If the response packets from PVM and SVM are > identical, they are released immediately. Otherwise, a VM checkpoint (on > demand)is conducted. > > Paper: > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > > COLO on Xen: > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > > COLO on Qemu/KVM: > http://wiki.qemu.org/Features/COLO > > By the needs of capturing response packets from PVM and SVM and finding out > whether they are identical, we introduce a new module to qemu networking > called colo-proxy. > > V2: > rebase colo-proxy with qemu-colo-v2.2-periodic-mode > fix dave's comments > fix wency's comments > fix zhanghailiang's comments > > v1: > initial patch. > > > > zhangchen (10): > Init colo-proxy object based on netfilter > Jhash: add linux kernel jhashtable in qemu > Colo-proxy: add colo-proxy framework > Colo-proxy: add data structure and jhash func > net/colo-proxy: Add colo interface to use proxy > net/colo-proxy: add socket used by forward func > net/colo-proxy: Add packet enqueue & handle func > net/colo-proxy: Handle packet and connection > net/colo-proxy: Compare pri pkt to sec pkt > net/colo-proxy: Colo-proxy do checkpoint and clear > > include/qemu/jhash.h | 61 > net/Makefile.objs| 1 + > net/colo-proxy.c | 939 > +++ > net/colo-proxy.h | 24 ++ > qemu-options.hx | 6 + > trace-events | 8 + > vl.c | 3 +- > 7 files changed, 1041 insertions(+), 1 deletion(-) > create mode 100644 include/qemu/jhash.h > create mode 100644 net/colo-proxy.c > create mode 100644 net/colo-proxy.h > > -- > 1.9.1 > > > -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/22/2016 12:46 AM, Wen Congyang wrote: ... > On 01/20/2016 11:29 AM, Zhang Chen wrote: > Sure. Wow, that's a lot of wasted quoting. Your mail weighed in at 24k, even though... >> Thanks. I think I get the point. So if there's a difference, primary >> packet will only be sent after checkpoint and we could not assume the >> checkpoint itself is reliable. > > Yes. > >> >> Back to the filters design. We'd better still decouple packet comparing >> out of netdev. Maybe a little bit more tweak on what you've suggested: >> >> -netdev tap,id=hn0 >> -object traffic-mirrorer,id=f0,netdev=hn0,queue=tx,outdev=mirrorer0 >> -object >> traffic-redirector,id=f1,netdev=hn0,queue=rx,outdev=comparer0,indev=comparer2 >> -colo-comparer >> primary_traffic=comparer0,secondary_traffic=comparer1,outdev=comparer2 >> >> Just add one more socket for comparer for sending primary packet, and >> let f1 redirector its output to netdev? > > OK, I understand it now. > Thanks for your suggestion. ...content-wise, you only added about 100 bytes. It's okay to trim replies down to relevant portions, to make it easier for readers to get to the meat of your message. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/22/2016 11:15 AM, Jason Wang wrote: > > > On 01/20/2016 06:30 PM, Wen Congyang wrote: >> On 01/20/2016 06:19 PM, Jason Wang wrote: On 01/20/2016 06:01 PM, Wen Congyang wrote: >> On 01/20/2016 02:54 PM, Jason Wang wrote: On 01/20/2016 11:29 AM, Zhang Chen wrote: Sure. Two main comments/suggestions: - TCP analysis is missed in current version, maybe you point a git tree (or another version of RFC) to me for a better understanding of the design. (Just a skeleton for TCP should be sufficient to discuss). - I prefer to make the code as reusable as possible. So it's better to split/decouple the reusable parts from the codes. So a vague idea is: 1) Decouple the packet comparing from the netfilter. You've achieved this 99% since the work has been done in a thread. Just let the thread poll sockets directly, then the comparing have the possibility to be reused by other kinds of dataplane. 2) Implement traffic mirror/redirector as filter. 3) Implement TCP seq rewriting as a filter. Then, in primary node, you need just a traffic mirror, which did: - mirror ingress traffic to secondary node - mirror outgress traffic to packet comparing thread And in secondadry node, you need two filters: - A TCP seq rewriter which adjust tcp sequence number. - A traffic redirector which redirect packet from a socket as ingress traffic, and redirect outgress traffic to the socket which could be polled by remote packet comparing thread. Thoughts? Thanks >> Thanks >> zhangchen >> >> Hi, Jason. >> We consider your suggestion to split/decouple >> the reusable parts from the codes. >> Due to filter plugin are traversed one by one in order >> we will split colo-proxy to three filters in each side. >> >> But in this plan,primary and secondary both have socket >> server,startup is a problem. I believe this issue could be solved by reusing socket chardev. >> >> Primary qemu >> Secondary qemu >> +--+ >> +---+ >> | +-+ | | >> +--+ | >> | | | | | >> | | | >> | |guest| | | >> |guest | | >> | | | | | >> | | | >> | +---^--+--+ | | >> +-++---+ | >> | | | | >> |^| | >> | | | | >> ||| | >> | +-+ >> ||| | >> | netfilter | | || | >> >> netfilter|| | >> | +-+ || | >> +--+ | >> | | | | filter excute order | || | >> | || filter excute order | | >> | | | |+---> | || | >> | || +---> | | >> | | | | | || | >> | || TCP | | >> | | +-+-+ +--v-+++ +-+ | || | >> | +---+ +---++---v+rewriter+ ++ | | >> | | | | |||| | || | >> | | |
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 06:30 PM, Wen Congyang wrote: > On 01/20/2016 06:19 PM, Jason Wang wrote: >> > >> > >> > On 01/20/2016 06:01 PM, Wen Congyang wrote: >>> >> On 01/20/2016 02:54 PM, Jason Wang wrote: >>> >>> On 01/20/2016 11:29 AM, Zhang Chen wrote: >> > Sure. >> > >> > Two main comments/suggestions: >> > >> > - TCP analysis is missed in current version, maybe you point a git >> > tree >> > (or another version of RFC) to me for a better understanding of the >> > design. (Just a skeleton for TCP should be sufficient to discuss). >> > - I prefer to make the code as reusable as possible. So it's >> > better to >> > split/decouple the reusable parts from the codes. So a vague idea >> > is: >> > >> > 1) Decouple the packet comparing from the netfilter. You've >> > achieved >> > this 99% since the work has been done in a thread. Just let the >> > thread >> > poll sockets directly, then the comparing have the possibility to >> > be >> > reused by other kinds of dataplane. >> > 2) Implement traffic mirror/redirector as filter. >> > 3) Implement TCP seq rewriting as a filter. >> > >> > Then, in primary node, you need just a traffic mirror, which did: >> > - mirror ingress traffic to secondary node >> > - mirror outgress traffic to packet comparing thread >> > >> > And in secondadry node, you need two filters: >> > - A TCP seq rewriter which adjust tcp sequence number. >> > - A traffic redirector which redirect packet from a socket as >> > ingress >> > traffic, and redirect outgress traffic to the socket which could be >> > polled by remote packet comparing thread. >> > Thoughts? >> > >> > Thanks >> > >>> >> Thanks >>> >> zhangchen > > Hi, Jason. > We consider your suggestion to split/decouple > the reusable parts from the codes. > Due to filter plugin are traversed one by one in order > we will split colo-proxy to three filters in each side. > > But in this plan,primary and secondary both have socket > server,startup is a problem. >>> I believe this issue could be solved by reusing socket chardev. >>> > > Primary qemu > Secondary qemu > +--+ > +---+ > | +-+ | | > +--+ | > | | | | | > | | | > | |guest| | | > |guest | | > | | | | | > | | | > | +---^--+--+ | | > +-++---+ | > | | | | > |^| | > | | | | > ||| | > | +-+ > ||| | > | netfilter | | || | > > netfilter|| | > | +-+ || | > +--+ | > | | | | filter excute order | || | > | || filter excute order | | > | | | |+---> | || | > | || +---> | | > | | | | | || | > | || TCP | | > | | +-+-+ +--v-+++ +-+ | || | > | +---+ +---++---v+rewriter+ ++ | | > | | | | |||| | || | > | | | || | || | |
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/22/2016 01:56 PM, Wen Congyang wrote: > On 01/22/2016 01:41 PM, Jason Wang wrote: >> > >> > >> > On 01/22/2016 11:28 AM, Wen Congyang wrote: >>> >> On 01/22/2016 11:15 AM, Jason Wang wrote: >>> >>> On 01/20/2016 06:30 PM, Wen Congyang wrote: > On 01/20/2016 06:19 PM, Jason Wang wrote: >>> >> >>> >> On 01/20/2016 06:01 PM, Wen Congyang wrote: > On 01/20/2016 02:54 PM, Jason Wang wrote: >>> >> On 01/20/2016 11:29 AM, Zhang Chen wrote: >>> >> Sure. >>> >> >>> >> Two main comments/suggestions: >>> >> >>> >> - TCP analysis is missed in current version, >>> >> maybe you point a git tree >>> >> (or another version of RFC) to me for a better >>> >> understanding of the >>> >> design. (Just a skeleton for TCP should be >>> >> sufficient to discuss). >>> >> - I prefer to make the code as reusable as >>> >> possible. So it's better to >>> >> split/decouple the reusable parts from the >>> >> codes. So a vague idea is: >>> >> >>> >> 1) Decouple the packet comparing from the >>> >> netfilter. You've achieved >>> >> this 99% since the work has been done in a >>> >> thread. Just let the thread >>> >> poll sockets directly, then the comparing have >>> >> the possibility to be >>> >> reused by other kinds of dataplane. >>> >> 2) Implement traffic mirror/redirector as filter. >>> >> 3) Implement TCP seq rewriting as a filter. >>> >> >>> >> Then, in primary node, you need just a traffic >>> >> mirror, which did: >>> >> - mirror ingress traffic to secondary node >>> >> - mirror outgress traffic to packet comparing >>> >> thread >>> >> >>> >> And in secondadry node, you need two filters: >>> >> - A TCP seq rewriter which adjust tcp sequence >>> >> number. >>> >> - A traffic redirector which redirect packet >>> >> from a socket as ingress >>> >> traffic, and redirect outgress traffic to the >>> >> socket which could be >>> >> polled by remote packet comparing thread. >>> >> Thoughts? >>> >> >>> >> Thanks >>> >> > Thanks > zhangchen > Hi, Jason. > We consider your suggestion to split/decouple > the reusable parts from the codes. > Due to filter plugin are traversed one by one in > order > we will split colo-proxy to three filters in each > side. > > But in this plan,primary and secondary both have > socket > server,startup is a problem. >>> >> I believe this issue could be solved by reusing socket >>> >> chardev. >>> >> > Primary qemu > > Secondary qemu > +--+ > > +---+ > | > +-+ > | | > +--+ > | > | | > | | | > | > | | > | |guest > | | | > |guest > | | >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/22/2016 02:21 PM, Jason Wang wrote: > > > On 01/22/2016 01:56 PM, Wen Congyang wrote: >> On 01/22/2016 01:41 PM, Jason Wang wrote: On 01/22/2016 11:28 AM, Wen Congyang wrote: >> On 01/22/2016 11:15 AM, Jason Wang wrote: On 01/20/2016 06:30 PM, Wen Congyang wrote: >> On 01/20/2016 06:19 PM, Jason Wang wrote: >> >> On 01/20/2016 06:01 PM, Wen Congyang wrote: >> On 01/20/2016 02:54 PM, Jason Wang wrote: >> On 01/20/2016 11:29 AM, Zhang Chen wrote: >> Sure. >> >> Two main comments/suggestions: >> >> - TCP analysis is missed in current version, >> maybe you point a git tree >> (or another version of RFC) to me for a better >> understanding of the >> design. (Just a skeleton for TCP should be >> sufficient to discuss). >> - I prefer to make the code as reusable as >> possible. So it's better to >> split/decouple the reusable parts from the >> codes. So a vague idea is: >> >> 1) Decouple the packet comparing from the >> netfilter. You've achieved >> this 99% since the work has been done in a >> thread. Just let the thread >> poll sockets directly, then the comparing have >> the possibility to be >> reused by other kinds of dataplane. >> 2) Implement traffic mirror/redirector as filter. >> 3) Implement TCP seq rewriting as a filter. >> >> Then, in primary node, you need just a traffic >> mirror, which did: >> - mirror ingress traffic to secondary node >> - mirror outgress traffic to packet comparing >> thread >> >> And in secondadry node, you need two filters: >> - A TCP seq rewriter which adjust tcp sequence >> number. >> - A traffic redirector which redirect packet >> from a socket as ingress >> traffic, and redirect outgress traffic to the >> socket which could be >> polled by remote packet comparing thread. >> Thoughts? >> >> Thanks >> >> Thanks >> zhangchen >> Hi, Jason. >> We consider your suggestion to split/decouple >> the reusable parts from the codes. >> Due to filter plugin are traversed one by one in >> order >> we will split colo-proxy to three filters in each >> side. >> >> But in this plan,primary and secondary both have >> socket >> server,startup is a problem. >> I believe this issue could be solved by reusing socket >> chardev. >> >> Primary qemu >> >> Secondary qemu >> +--+ >> >> +---+ >> | >> +-+ >> | | >> +--+ >> | >> | | >>| | | >> | >> | | >> | |guest >>| | | >> |guest
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 06:34 PM, Wen Congyang wrote: > On 01/20/2016 06:03 PM, Jason Wang wrote: >> >> On 01/20/2016 05:49 PM, Wen Congyang wrote: >>> On 01/20/2016 05:20 PM, Jason Wang wrote: On 01/20/2016 03:44 PM, Wen Congyang wrote: >>> ... >>> -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait >>> -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait >>> -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait >>> -netdev tap,id=hn0 >>> -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0 >>> -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1 >>> ... >>> >>> packet comparer compares the packets from two chardev: comparer0 and >>> comparer1. >>> traffic-mirrorer mirror tx to secondary node through chardev mirrorer0, >>> and mirror rx to packet comparer through chardev comparer0. >>> >>> In secondary node: >>> >>> ... >>> -chardev socket,id=redirector0,host=ip_primary,port=Y >>> -chardev socket,id=redirector1,host=ip_primary,port=Z >>> -netdev tap,id=hn0 >>> -traffic-redirector >>> netdev=hn0,id,r0,indev=redirector0,outdev=redirector1 >>> -colo-rewriter netdev=hn0,id=c0 >>> ... >>> >>> traffic-redirector redirect the rx traffic from primary node through >>> redirector0 and redirect the tx traffic to promary node through >>> redirector1. >>> colo-rewriter rewrite seq number as a normal netfilter. > What are traffic-mirrorer and colo-comparer, traffic-redirector, > colo-rewriter? > A netfilter driver? traffic-mirrorer/redirector is a type of netfilter that just mirror/redirect packets between netdev and chardev (just the mirror client/sever and redirect client/sever in the above graph) colo-rewriter is a type of netfilter that did ack/seq adjust (just the TCP rewriter in the above graph) colo-comparer is a thread object that did packet comparing (similar to "compare" in the above graph but not a netfiler) >>> Thanks. I have another question: >>> IIRC, both rx and tx packets walk through all netfilter objects in the same >>> order. >>> >>> tx packet(sent to the guest): we want that redirector hanldes it first >>> rx packet(sent from the guest): we want that colo-rewriter handles it first >>> Change the order or use two traffic-redirectors? >>> >>> Thanks >>> Wen Congyang >> Interesting question. >> >> Two redirectors sounds ok or maybe we can go through rx filters in a >> reverse order? > netdev <---> filter1 <> filter2 <> <> emulated device <> > guest > So I think we can go through rx filters in a reverse order. But it changes > the behavior. So I am not sure if we can do it. I think we can. Both dump and buffer filter does not require strict order, so it's a good time and change to do this. > > Thanks > Wen Congyang > >> >> . >> > >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/22/2016 01:41 PM, Jason Wang wrote: > > > On 01/22/2016 11:28 AM, Wen Congyang wrote: >> On 01/22/2016 11:15 AM, Jason Wang wrote: >>> >>> On 01/20/2016 06:30 PM, Wen Congyang wrote: On 01/20/2016 06:19 PM, Jason Wang wrote: >> >> On 01/20/2016 06:01 PM, Wen Congyang wrote: On 01/20/2016 02:54 PM, Jason Wang wrote: >> On 01/20/2016 11:29 AM, Zhang Chen wrote: >> Sure. >> >> Two main comments/suggestions: >> >> - TCP analysis is missed in current version, maybe you point a >> git tree >> (or another version of RFC) to me for a better understanding of >> the >> design. (Just a skeleton for TCP should be sufficient to >> discuss). >> - I prefer to make the code as reusable as possible. So it's >> better to >> split/decouple the reusable parts from the codes. So a vague >> idea is: >> >> 1) Decouple the packet comparing from the netfilter. You've >> achieved >> this 99% since the work has been done in a thread. Just let the >> thread >> poll sockets directly, then the comparing have the possibility >> to be >> reused by other kinds of dataplane. >> 2) Implement traffic mirror/redirector as filter. >> 3) Implement TCP seq rewriting as a filter. >> >> Then, in primary node, you need just a traffic mirror, which did: >> - mirror ingress traffic to secondary node >> - mirror outgress traffic to packet comparing thread >> >> And in secondadry node, you need two filters: >> - A TCP seq rewriter which adjust tcp sequence number. >> - A traffic redirector which redirect packet from a socket as >> ingress >> traffic, and redirect outgress traffic to the socket which could >> be >> polled by remote packet comparing thread. >> Thoughts? >> >> Thanks >> Thanks zhangchen Hi, Jason. We consider your suggestion to split/decouple the reusable parts from the codes. Due to filter plugin are traversed one by one in order we will split colo-proxy to three filters in each side. But in this plan,primary and secondary both have socket server,startup is a problem. >> I believe this issue could be solved by reusing socket chardev. >> Primary qemu Secondary qemu +--+ +---+ | +-+ | | +--+ | | | | | | | | | | |guest| | | |guest | | | | | | | | | | | +---^--+--+ | | +-++---+ | | | | | |^| | | | | | ||| | | +-+ ||| | | netfilter | | || | netfilter|| | | +-+ || | +--+ | | | | | filter excute order | || | | || filter excute order | | | | | |+---> | || | | || +---> | | | |
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/22/2016 01:33 PM, Jason Wang wrote: > > > On 01/20/2016 06:34 PM, Wen Congyang wrote: >> On 01/20/2016 06:03 PM, Jason Wang wrote: >>> >>> On 01/20/2016 05:49 PM, Wen Congyang wrote: On 01/20/2016 05:20 PM, Jason Wang wrote: > On 01/20/2016 03:44 PM, Wen Congyang wrote: ... -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait -netdev tap,id=hn0 -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0 -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1 ... packet comparer compares the packets from two chardev: comparer0 and comparer1. traffic-mirrorer mirror tx to secondary node through chardev mirrorer0, and mirror rx to packet comparer through chardev comparer0. In secondary node: ... -chardev socket,id=redirector0,host=ip_primary,port=Y -chardev socket,id=redirector1,host=ip_primary,port=Z -netdev tap,id=hn0 -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1 -colo-rewriter netdev=hn0,id=c0 ... traffic-redirector redirect the rx traffic from primary node through redirector0 and redirect the tx traffic to promary node through redirector1. colo-rewriter rewrite seq number as a normal netfilter. >> What are traffic-mirrorer and colo-comparer, traffic-redirector, >> colo-rewriter? >> A netfilter driver? > traffic-mirrorer/redirector is a type of netfilter that just > mirror/redirect packets between netdev and chardev (just the mirror > client/sever and redirect client/sever in the above graph) > colo-rewriter is a type of netfilter that did ack/seq adjust (just the > TCP rewriter in the above graph) > colo-comparer is a thread object that did packet comparing (similar to > "compare" in the above graph but not a netfiler) Thanks. I have another question: IIRC, both rx and tx packets walk through all netfilter objects in the same order. tx packet(sent to the guest): we want that redirector hanldes it first rx packet(sent from the guest): we want that colo-rewriter handles it first Change the order or use two traffic-redirectors? Thanks Wen Congyang >>> Interesting question. >>> >>> Two redirectors sounds ok or maybe we can go through rx filters in a >>> reverse order? >> netdev <---> filter1 <> filter2 <> <> emulated device >> <> guest >> So I think we can go through rx filters in a reverse order. But it changes >> the behavior. So I am not sure if we can do it. > > I think we can. Both dump and buffer filter does not require strict > order, so it's a good time and change to do this. OK, we will do it. Thanks Wen Congyang > >> >> Thanks >> Wen Congyang >> >>> >>> . >>> >> >> > > > > . >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/22/2016 02:47 PM, Wen Congyang wrote: > On 01/22/2016 02:21 PM, Jason Wang wrote: >> >> On 01/22/2016 01:56 PM, Wen Congyang wrote: >>> On 01/22/2016 01:41 PM, Jason Wang wrote: > > On 01/22/2016 11:28 AM, Wen Congyang wrote: >>> On 01/22/2016 11:15 AM, Jason Wang wrote: > On 01/20/2016 06:30 PM, Wen Congyang wrote: >>> On 01/20/2016 06:19 PM, Jason Wang wrote: >>> On 01/20/2016 06:01 PM, Wen Congyang wrote: >>> On 01/20/2016 02:54 PM, Jason Wang wrote: >>> On 01/20/2016 11:29 AM, Zhang Chen wrote: >>> Sure. >>> >>> Two main comments/suggestions: >>> >>> - TCP analysis is missed in current version, >>> maybe you point a git tree >>> (or another version of RFC) to me for a better >>> understanding of the >>> design. (Just a skeleton for TCP should be >>> sufficient to discuss). >>> - I prefer to make the code as reusable as >>> possible. So it's better to >>> split/decouple the reusable parts from the >>> codes. So a vague idea is: >>> >>> 1) Decouple the packet comparing from the >>> netfilter. You've achieved >>> this 99% since the work has been done in a >>> thread. Just let the thread >>> poll sockets directly, then the comparing have >>> the possibility to be >>> reused by other kinds of dataplane. >>> 2) Implement traffic mirror/redirector as >>> filter. >>> 3) Implement TCP seq rewriting as a filter. >>> >>> Then, in primary node, you need just a traffic >>> mirror, which did: >>> - mirror ingress traffic to secondary node >>> - mirror outgress traffic to packet comparing >>> thread >>> >>> And in secondadry node, you need two filters: >>> - A TCP seq rewriter which adjust tcp sequence >>> number. >>> - A traffic redirector which redirect packet >>> from a socket as ingress >>> traffic, and redirect outgress traffic to the >>> socket which could be >>> polled by remote packet comparing thread. >>> Thoughts? >>> >>> Thanks >>> >>> Thanks >>> zhangchen >>> Hi, Jason. >>> We consider your suggestion to split/decouple >>> the reusable parts from the codes. >>> Due to filter plugin are traversed one by one in >>> order >>> we will split colo-proxy to three filters in each >>> side. >>> >>> But in this plan,primary and secondary both have >>> socket >>> server,startup is a problem. >>> I believe this issue could be solved by reusing socket >>> chardev. >>> >>> Primary qemu >>> >>> Secondary qemu >>> +--+ >>> >>> +---+ >>> | >>> +-+ >>> | | >>> +--+ >>> | >>> | | >>> | | | >>> | >>>| | >>> | |guest
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/22/2016 11:28 AM, Wen Congyang wrote: > On 01/22/2016 11:15 AM, Jason Wang wrote: >> >> On 01/20/2016 06:30 PM, Wen Congyang wrote: >>> On 01/20/2016 06:19 PM, Jason Wang wrote: > > On 01/20/2016 06:01 PM, Wen Congyang wrote: >>> On 01/20/2016 02:54 PM, Jason Wang wrote: > On 01/20/2016 11:29 AM, Zhang Chen wrote: > Sure. > > Two main comments/suggestions: > > - TCP analysis is missed in current version, maybe you point a > git tree > (or another version of RFC) to me for a better understanding of > the > design. (Just a skeleton for TCP should be sufficient to discuss). > - I prefer to make the code as reusable as possible. So it's > better to > split/decouple the reusable parts from the codes. So a vague idea > is: > > 1) Decouple the packet comparing from the netfilter. You've > achieved > this 99% since the work has been done in a thread. Just let the > thread > poll sockets directly, then the comparing have the possibility to > be > reused by other kinds of dataplane. > 2) Implement traffic mirror/redirector as filter. > 3) Implement TCP seq rewriting as a filter. > > Then, in primary node, you need just a traffic mirror, which did: > - mirror ingress traffic to secondary node > - mirror outgress traffic to packet comparing thread > > And in secondadry node, you need two filters: > - A TCP seq rewriter which adjust tcp sequence number. > - A traffic redirector which redirect packet from a socket as > ingress > traffic, and redirect outgress traffic to the socket which could > be > polled by remote packet comparing thread. > Thoughts? > > Thanks > >>> Thanks >>> zhangchen >>> Hi, Jason. >>> We consider your suggestion to split/decouple >>> the reusable parts from the codes. >>> Due to filter plugin are traversed one by one in order >>> we will split colo-proxy to three filters in each side. >>> >>> But in this plan,primary and secondary both have socket >>> server,startup is a problem. > I believe this issue could be solved by reusing socket chardev. > >>> Primary qemu >>> Secondary qemu >>> +--+ >>> +---+ >>> | +-+ | >>> | >>> +--+ | >>> | | | | >>> | >>> | | | >>> | |guest| | >>> | >>> |guest | | >>> | | | | >>> | >>> | | | >>> | +---^--+--+ | >>> | >>> +-++---+ | >>> | | | | >>> |^| | >>> | | | | >>> ||| | >>> | +-+ >>> ||| | >>> | netfilter | | || >>> | >>> netfilter|| | >>> | +-+ || >>> | >>> +--+ | >>> | | | | filter excute order | || >>> | >>> | || filter excute order | | >>> | | | |+---> | || >>> | >>> | || +---> | | >>> | | | | | || >>> | >>> | || TCP | | >>> | |
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/22/2016 03:42 PM, Jason Wang wrote: > > > On 01/22/2016 02:47 PM, Wen Congyang wrote: >> On 01/22/2016 02:21 PM, Jason Wang wrote: >>> >>> On 01/22/2016 01:56 PM, Wen Congyang wrote: On 01/22/2016 01:41 PM, Jason Wang wrote: >> >> On 01/22/2016 11:28 AM, Wen Congyang wrote: On 01/22/2016 11:15 AM, Jason Wang wrote: >> On 01/20/2016 06:30 PM, Wen Congyang wrote: On 01/20/2016 06:19 PM, Jason Wang wrote: On 01/20/2016 06:01 PM, Wen Congyang wrote: On 01/20/2016 02:54 PM, Jason Wang wrote: On 01/20/2016 11:29 AM, Zhang Chen wrote: Sure. Two main comments/suggestions: - TCP analysis is missed in current version, maybe you point a git tree (or another version of RFC) to me for a better understanding of the design. (Just a skeleton for TCP should be sufficient to discuss). - I prefer to make the code as reusable as possible. So it's better to split/decouple the reusable parts from the codes. So a vague idea is: 1) Decouple the packet comparing from the netfilter. You've achieved this 99% since the work has been done in a thread. Just let the thread poll sockets directly, then the comparing have the possibility to be reused by other kinds of dataplane. 2) Implement traffic mirror/redirector as filter. 3) Implement TCP seq rewriting as a filter. Then, in primary node, you need just a traffic mirror, which did: - mirror ingress traffic to secondary node - mirror outgress traffic to packet comparing thread And in secondadry node, you need two filters: - A TCP seq rewriter which adjust tcp sequence number. - A traffic redirector which redirect packet from a socket as ingress traffic, and redirect outgress traffic to the socket which could be polled by remote packet comparing thread. Thoughts? Thanks Thanks zhangchen Hi, Jason. We consider your suggestion to split/decouple the reusable parts from the codes. Due to filter plugin are traversed one by one in order we will split colo-proxy to three filters in each side. But in this plan,primary and secondary both have socket server,startup is a problem. I believe this issue could be solved by reusing socket chardev. Primary qemu Secondary qemu +--+ +---+ | +-+ | | +--+ | | | | | | |
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 03:44 PM, Wen Congyang wrote: >> > >> > ... >> > -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait >> > -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait >> > -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait >> > -netdev tap,id=hn0 >> > -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0 >> > -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1 >> > ... >> > >> > packet comparer compares the packets from two chardev: comparer0 and >> > comparer1. >> > traffic-mirrorer mirror tx to secondary node through chardev mirrorer0, >> > and mirror rx to packet comparer through chardev comparer0. >> > >> > In secondary node: >> > >> > ... >> > -chardev socket,id=redirector0,host=ip_primary,port=Y >> > -chardev socket,id=redirector1,host=ip_primary,port=Z >> > -netdev tap,id=hn0 >> > -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1 >> > -colo-rewriter netdev=hn0,id=c0 >> > ... >> > >> > traffic-redirector redirect the rx traffic from primary node through >> > redirector0 and redirect the tx traffic to promary node through >> > redirector1. >> > colo-rewriter rewrite seq number as a normal netfilter. > What are traffic-mirrorer and colo-comparer, traffic-redirector, > colo-rewriter? > A netfilter driver? traffic-mirrorer/redirector is a type of netfilter that just mirror/redirect packets between netdev and chardev (just the mirror client/sever and redirect client/sever in the above graph) colo-rewriter is a type of netfilter that did ack/seq adjust (just the TCP rewriter in the above graph) colo-comparer is a thread object that did packet comparing (similar to "compare" in the above graph but not a netfiler) > > If not, how to get the packet from the netdev, and send back the packet to > the netdev? > > Thanks > Wen Congyang >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 05:20 PM, Jason Wang wrote: > > > On 01/20/2016 03:44 PM, Wen Congyang wrote: ... -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait -netdev tap,id=hn0 -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0 -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1 ... packet comparer compares the packets from two chardev: comparer0 and comparer1. traffic-mirrorer mirror tx to secondary node through chardev mirrorer0, and mirror rx to packet comparer through chardev comparer0. In secondary node: ... -chardev socket,id=redirector0,host=ip_primary,port=Y -chardev socket,id=redirector1,host=ip_primary,port=Z -netdev tap,id=hn0 -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1 -colo-rewriter netdev=hn0,id=c0 ... traffic-redirector redirect the rx traffic from primary node through redirector0 and redirect the tx traffic to promary node through redirector1. colo-rewriter rewrite seq number as a normal netfilter. >> What are traffic-mirrorer and colo-comparer, traffic-redirector, >> colo-rewriter? >> A netfilter driver? > > traffic-mirrorer/redirector is a type of netfilter that just > mirror/redirect packets between netdev and chardev (just the mirror > client/sever and redirect client/sever in the above graph) > colo-rewriter is a type of netfilter that did ack/seq adjust (just the > TCP rewriter in the above graph) > colo-comparer is a thread object that did packet comparing (similar to > "compare" in the above graph but not a netfiler) Thanks. I have another question: IIRC, both rx and tx packets walk through all netfilter objects in the same order. tx packet(sent to the guest): we want that redirector hanldes it first rx packet(sent from the guest): we want that colo-rewriter handles it first Change the order or use two traffic-redirectors? Thanks Wen Congyang > >> >> If not, how to get the packet from the netdev, and send back the packet to >> the netdev? >> >> Thanks >> Wen Congyang >> > > > > . >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 05:49 PM, Wen Congyang wrote: > On 01/20/2016 05:20 PM, Jason Wang wrote: >> >> On 01/20/2016 03:44 PM, Wen Congyang wrote: > ... > -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait > -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait > -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait > -netdev tap,id=hn0 > -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0 > -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1 > ... > > packet comparer compares the packets from two chardev: comparer0 and > comparer1. > traffic-mirrorer mirror tx to secondary node through chardev mirrorer0, > and mirror rx to packet comparer through chardev comparer0. > > In secondary node: > > ... > -chardev socket,id=redirector0,host=ip_primary,port=Y > -chardev socket,id=redirector1,host=ip_primary,port=Z > -netdev tap,id=hn0 > -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1 > -colo-rewriter netdev=hn0,id=c0 > ... > > traffic-redirector redirect the rx traffic from primary node through > redirector0 and redirect the tx traffic to promary node through > redirector1. > colo-rewriter rewrite seq number as a normal netfilter. >>> What are traffic-mirrorer and colo-comparer, traffic-redirector, >>> colo-rewriter? >>> A netfilter driver? >> traffic-mirrorer/redirector is a type of netfilter that just >> mirror/redirect packets between netdev and chardev (just the mirror >> client/sever and redirect client/sever in the above graph) >> colo-rewriter is a type of netfilter that did ack/seq adjust (just the >> TCP rewriter in the above graph) >> colo-comparer is a thread object that did packet comparing (similar to >> "compare" in the above graph but not a netfiler) > Thanks. I have another question: > IIRC, both rx and tx packets walk through all netfilter objects in the same > order. > > tx packet(sent to the guest): we want that redirector hanldes it first > rx packet(sent from the guest): we want that colo-rewriter handles it first > Change the order or use two traffic-redirectors? > > Thanks > Wen Congyang Interesting question. Two redirectors sounds ok or maybe we can go through rx filters in a reverse order?
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 02:54 PM, Jason Wang wrote: > > > On 01/20/2016 11:29 AM, Zhang Chen wrote: >> >>> Sure. >>> >>> Two main comments/suggestions: >>> >>> - TCP analysis is missed in current version, maybe you point a git tree >>> (or another version of RFC) to me for a better understanding of the >>> design. (Just a skeleton for TCP should be sufficient to discuss). >>> - I prefer to make the code as reusable as possible. So it's better to >>> split/decouple the reusable parts from the codes. So a vague idea is: >>> >>> 1) Decouple the packet comparing from the netfilter. You've achieved >>> this 99% since the work has been done in a thread. Just let the thread >>> poll sockets directly, then the comparing have the possibility to be >>> reused by other kinds of dataplane. >>> 2) Implement traffic mirror/redirector as filter. >>> 3) Implement TCP seq rewriting as a filter. >>> >>> Then, in primary node, you need just a traffic mirror, which did: >>> - mirror ingress traffic to secondary node >>> - mirror outgress traffic to packet comparing thread >>> >>> And in secondadry node, you need two filters: >>> - A TCP seq rewriter which adjust tcp sequence number. >>> - A traffic redirector which redirect packet from a socket as ingress >>> traffic, and redirect outgress traffic to the socket which could be >>> polled by remote packet comparing thread. >>> Thoughts? >>> >>> Thanks >>> Thanks zhangchen >>> >> >> >> Hi, Jason. >> We consider your suggestion to split/decouple >> the reusable parts from the codes. >> Due to filter plugin are traversed one by one in order >> we will split colo-proxy to three filters in each side. >> >> But in this plan,primary and secondary both have socket >> server,startup is a problem. > > I believe this issue could be solved by reusing socket chardev. > >> >> >> Primary qemu >> Secondary qemu >> +--+ >> +---+ >> | +-+ | | >> +--+ | >> | | | | | >> | | | >> | |guest| | | >> |guest | | >> | | | | | >> | | | >> | +---^--+--+ | | >> +-++---+ | >> | | | | >> |^| | >> | | | | >> ||| | >> | +-+ >> ||| | >> | netfilter | | || | >> netfilter|| | >> | +-+ || | >> +--+ | >> | | | | filter excute order | || | >> | || filter excute order | | >> | | | |+---> | || | >> | || +---> | | >> | | | | | || | >> | || TCP | | >> | | +-+-+ +--v-+++ +-+ | || | >> | +---+ +---++---v+rewriter+ ++ | | >> | | | | |||| | || | >> | | | || | || | | >> | | | mirror | | redirect +> compare | | | >> +> mirror +---> adjust | adjust+-->redirect| | | >> | | | client | | server||| | | | >> | | server | | ack| seq | |client | | | >> | | | | |||| | | | >> | | | || | || | | >> | | +^--+ +^---++-+--+ | | | >> | +---+ ++-+ ++---+ | | >> | | | tx | rx | rx | | | >> |txall | rx | | >> | +-+ | | >> +--+ | >> || >>
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 06:01 PM, Wen Congyang wrote: > On 01/20/2016 02:54 PM, Jason Wang wrote: >> >> On 01/20/2016 11:29 AM, Zhang Chen wrote: Sure. Two main comments/suggestions: - TCP analysis is missed in current version, maybe you point a git tree (or another version of RFC) to me for a better understanding of the design. (Just a skeleton for TCP should be sufficient to discuss). - I prefer to make the code as reusable as possible. So it's better to split/decouple the reusable parts from the codes. So a vague idea is: 1) Decouple the packet comparing from the netfilter. You've achieved this 99% since the work has been done in a thread. Just let the thread poll sockets directly, then the comparing have the possibility to be reused by other kinds of dataplane. 2) Implement traffic mirror/redirector as filter. 3) Implement TCP seq rewriting as a filter. Then, in primary node, you need just a traffic mirror, which did: - mirror ingress traffic to secondary node - mirror outgress traffic to packet comparing thread And in secondadry node, you need two filters: - A TCP seq rewriter which adjust tcp sequence number. - A traffic redirector which redirect packet from a socket as ingress traffic, and redirect outgress traffic to the socket which could be polled by remote packet comparing thread. Thoughts? Thanks > Thanks > zhangchen >>> >>> Hi, Jason. >>> We consider your suggestion to split/decouple >>> the reusable parts from the codes. >>> Due to filter plugin are traversed one by one in order >>> we will split colo-proxy to three filters in each side. >>> >>> But in this plan,primary and secondary both have socket >>> server,startup is a problem. >> I believe this issue could be solved by reusing socket chardev. >> >>> >>> Primary qemu >>> Secondary qemu >>> +--+ >>> +---+ >>> | +-+ | | >>> +--+ | >>> | | | | | >>> | | | >>> | |guest| | | >>> |guest | | >>> | | | | | >>> | | | >>> | +---^--+--+ | | >>> +-++---+ | >>> | | | | >>> |^| | >>> | | | | >>> ||| | >>> | +-+ >>> ||| | >>> | netfilter | | || | >>> netfilter|| | >>> | +-+ || | >>> +--+ | >>> | | | | filter excute order | || | >>> | || filter excute order | | >>> | | | |+---> | || | >>> | || +---> | | >>> | | | | | || | >>> | || TCP | | >>> | | +-+-+ +--v-+++ +-+ | || | >>> | +---+ +---++---v+rewriter+ ++ | | >>> | | | | |||| | || | >>> | | | || | || | | >>> | | | mirror | | redirect +> compare | | | >>> +> mirror +---> adjust | adjust+-->redirect| | | >>> | | | client | | server||| | | | >>> | | server | | ack| seq | |client | | | >>> | | | | |||| | | | >>> | | | || | || | | >>> | | +^--+ +^---++-+--+ | | | >>> | +---+ ++-+ ++---+ | | >>> | | | tx | rx | rx | | | >>> |txall | rx | | >>> | +-+ | | >>>
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 06:03 PM, Jason Wang wrote: > > > On 01/20/2016 05:49 PM, Wen Congyang wrote: >> On 01/20/2016 05:20 PM, Jason Wang wrote: >>> >>> On 01/20/2016 03:44 PM, Wen Congyang wrote: >> ... >> -chardev socket,id=comparer0,host=ip_primary,port=X,server,nowait >> -chardev socket,id=comparer1,host=ip_primary,port=Y,server,nowait >> -chardev socket,id=mirrorer0,host=ip_primary,port=Z,server,nowait >> -netdev tap,id=hn0 >> -traffic-mirrorer netdev=hn0,id=t0,indev=comparer0,outdev=mirrorer0 >> -colo-comparer primary_traffic=comparer0,secondary_traffic=comparer1 >> ... >> >> packet comparer compares the packets from two chardev: comparer0 and >> comparer1. >> traffic-mirrorer mirror tx to secondary node through chardev mirrorer0, >> and mirror rx to packet comparer through chardev comparer0. >> >> In secondary node: >> >> ... >> -chardev socket,id=redirector0,host=ip_primary,port=Y >> -chardev socket,id=redirector1,host=ip_primary,port=Z >> -netdev tap,id=hn0 >> -traffic-redirector netdev=hn0,id,r0,indev=redirector0,outdev=redirector1 >> -colo-rewriter netdev=hn0,id=c0 >> ... >> >> traffic-redirector redirect the rx traffic from primary node through >> redirector0 and redirect the tx traffic to promary node through >> redirector1. >> colo-rewriter rewrite seq number as a normal netfilter. What are traffic-mirrorer and colo-comparer, traffic-redirector, colo-rewriter? A netfilter driver? >>> traffic-mirrorer/redirector is a type of netfilter that just >>> mirror/redirect packets between netdev and chardev (just the mirror >>> client/sever and redirect client/sever in the above graph) >>> colo-rewriter is a type of netfilter that did ack/seq adjust (just the >>> TCP rewriter in the above graph) >>> colo-comparer is a thread object that did packet comparing (similar to >>> "compare" in the above graph but not a netfiler) >> Thanks. I have another question: >> IIRC, both rx and tx packets walk through all netfilter objects in the same >> order. >> >> tx packet(sent to the guest): we want that redirector hanldes it first >> rx packet(sent from the guest): we want that colo-rewriter handles it first >> Change the order or use two traffic-redirectors? >> >> Thanks >> Wen Congyang > > Interesting question. > > Two redirectors sounds ok or maybe we can go through rx filters in a > reverse order? netdev <---> filter1 <> filter2 <> <> emulated device <> guest So I think we can go through rx filters in a reverse order. But it changes the behavior. So I am not sure if we can do it. Thanks Wen Congyang > > > . >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 06:19 PM, Jason Wang wrote: > > > On 01/20/2016 06:01 PM, Wen Congyang wrote: >> On 01/20/2016 02:54 PM, Jason Wang wrote: >>> >>> On 01/20/2016 11:29 AM, Zhang Chen wrote: > Sure. > > Two main comments/suggestions: > > - TCP analysis is missed in current version, maybe you point a git tree > (or another version of RFC) to me for a better understanding of the > design. (Just a skeleton for TCP should be sufficient to discuss). > - I prefer to make the code as reusable as possible. So it's better to > split/decouple the reusable parts from the codes. So a vague idea is: > > 1) Decouple the packet comparing from the netfilter. You've achieved > this 99% since the work has been done in a thread. Just let the thread > poll sockets directly, then the comparing have the possibility to be > reused by other kinds of dataplane. > 2) Implement traffic mirror/redirector as filter. > 3) Implement TCP seq rewriting as a filter. > > Then, in primary node, you need just a traffic mirror, which did: > - mirror ingress traffic to secondary node > - mirror outgress traffic to packet comparing thread > > And in secondadry node, you need two filters: > - A TCP seq rewriter which adjust tcp sequence number. > - A traffic redirector which redirect packet from a socket as ingress > traffic, and redirect outgress traffic to the socket which could be > polled by remote packet comparing thread. > Thoughts? > > Thanks > >> Thanks >> zhangchen Hi, Jason. We consider your suggestion to split/decouple the reusable parts from the codes. Due to filter plugin are traversed one by one in order we will split colo-proxy to three filters in each side. But in this plan,primary and secondary both have socket server,startup is a problem. >>> I believe this issue could be solved by reusing socket chardev. >>> Primary qemu Secondary qemu +--+ +---+ | +-+ | | +--+ | | | | | | | | | | |guest| | | |guest | | | | | | | | | | | +---^--+--+ | | +-++---+ | | | | | |^| | | | | | ||| | | +-+ ||| | | netfilter | | || | netfilter|| | | +-+ || | +--+ | | | | | filter excute order | || | | || filter excute order | | | | | |+---> | || | | || +---> | | | | | | | || | | || TCP | | | | +-+-+ +--v-+++ +-+ | || | | +---+ +---++---v+rewriter+ ++ | | | | | | |||| | || | | | | || | || | | | | | mirror | | redirect +> compare | | | +> mirror +---> adjust | adjust+-->redirect| | | | | | client | | server||| | | | | | server | | ack| seq | |client | | | | | | | |||| | | | | | | || | || | | | | +^--+ +^---++-+--+ | | | | +---+ ++-+ ++---+ | | | | | tx | rx | rx | | | |txall
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
Sure. Two main comments/suggestions: - TCP analysis is missed in current version, maybe you point a git tree (or another version of RFC) to me for a better understanding of the design. (Just a skeleton for TCP should be sufficient to discuss). - I prefer to make the code as reusable as possible. So it's better to split/decouple the reusable parts from the codes. So a vague idea is: 1) Decouple the packet comparing from the netfilter. You've achieved this 99% since the work has been done in a thread. Just let the thread poll sockets directly, then the comparing have the possibility to be reused by other kinds of dataplane. 2) Implement traffic mirror/redirector as filter. 3) Implement TCP seq rewriting as a filter. Then, in primary node, you need just a traffic mirror, which did: - mirror ingress traffic to secondary node - mirror outgress traffic to packet comparing thread And in secondadry node, you need two filters: - A TCP seq rewriter which adjust tcp sequence number. - A traffic redirector which redirect packet from a socket as ingress traffic, and redirect outgress traffic to the socket which could be polled by remote packet comparing thread. Thoughts? Thanks Thanks zhangchen Hi, Jason. We consider your suggestion to split/decouple the reusable parts from the codes. Due to filter plugin are traversed one by one in order we will split colo-proxy to three filters in each side. But in this plan,primary and secondary both have socket server,startup is a problem. Primary qemu Secondary qemu +--+ +---+ | +-+ | | +--+ | | | | | | | | | | |guest| | | | guest | | | | | | | | | | | +---^--+--+ | | +-++---+ | | | | | | ^| | | | | | | || | | +-+ | || | | netfilter | | || | netfilter|| | | +-+ || | +--+ | | | | | filter excute order | || | | || filter excute order | | | | | |+---> | || | | || +---> | | | | | | | || | | || TCP | | | | +-+-+ +--v-+++ +-+ | || | | +---+ +---++---v+rewriter+ ++ | | | | | | |||| | || | | | | || | || | | | | | mirror | | redirect +> compare | | |+> mirror +---> adjust | adjust+-->redirect| | | | | | client | | server||| | | | | | server | | ack| seq | |client | | | | | | | |||| | | | | | | || | || | | | | +^--+ +^---++-+--+ | | | | +---+ ++-+ ++---+ | | | | | tx | rx | rx | | | | txall | rx | | | +-+ | | +--+ | || +---+ | |||| | | +--+ +---+ || |guest receive |guest send |
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 11:29 AM, Zhang Chen wrote: > >> Sure. >> >> Two main comments/suggestions: >> >> - TCP analysis is missed in current version, maybe you point a git tree >> (or another version of RFC) to me for a better understanding of the >> design. (Just a skeleton for TCP should be sufficient to discuss). >> - I prefer to make the code as reusable as possible. So it's better to >> split/decouple the reusable parts from the codes. So a vague idea is: >> >> 1) Decouple the packet comparing from the netfilter. You've achieved >> this 99% since the work has been done in a thread. Just let the thread >> poll sockets directly, then the comparing have the possibility to be >> reused by other kinds of dataplane. >> 2) Implement traffic mirror/redirector as filter. >> 3) Implement TCP seq rewriting as a filter. >> >> Then, in primary node, you need just a traffic mirror, which did: >> - mirror ingress traffic to secondary node >> - mirror outgress traffic to packet comparing thread >> >> And in secondadry node, you need two filters: >> - A TCP seq rewriter which adjust tcp sequence number. >> - A traffic redirector which redirect packet from a socket as ingress >> traffic, and redirect outgress traffic to the socket which could be >> polled by remote packet comparing thread. >> Thoughts? >> >> Thanks >> >>> Thanks >>> zhangchen >> > > > Hi, Jason. > We consider your suggestion to split/decouple > the reusable parts from the codes. > Due to filter plugin are traversed one by one in order > we will split colo-proxy to three filters in each side. > > But in this plan,primary and secondary both have socket > server,startup is a problem. I believe this issue could be solved by reusing socket chardev. > > > Primary qemu > Secondary qemu > +--+ > +---+ > | +-+ | | > +--+ | > | | | | | > | | | > | |guest| | | > |guest | | > | | | | | > | | | > | +---^--+--+ | | > +-++---+ | > | | | | > |^| | > | | | | > ||| | > | +-+ > ||| | > | netfilter | | || | > netfilter|| | > | +-+ || | > +--+ | > | | | | filter excute order | || | > | || filter excute order | | > | | | |+---> | || | > | || +---> | | > | | | | | || | > | || TCP | | > | | +-+-+ +--v-+++ +-+ | || | > | +---+ +---++---v+rewriter+ ++ | | > | | | | |||| | || | > | | | || | || | | > | | | mirror | | redirect +> compare | | | > +> mirror +---> adjust | adjust+-->redirect| | | > | | | client | | server||| | | | > | | server | | ack| seq | |client | | | > | | | | |||| | | | > | | | || | || | | > | | +^--+ +^---++-+--+ | | | > | +---+ ++-+ ++---+ | | > | | | tx | rx | rx | | | > |txall | rx | | > | +-+ | | > +--+ | > || > +---+ > > | > |||| > |
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/20/2016 02:54 PM, Jason Wang wrote: > > > On 01/20/2016 11:29 AM, Zhang Chen wrote: >> >>> Sure. >>> >>> Two main comments/suggestions: >>> >>> - TCP analysis is missed in current version, maybe you point a git tree >>> (or another version of RFC) to me for a better understanding of the >>> design. (Just a skeleton for TCP should be sufficient to discuss). >>> - I prefer to make the code as reusable as possible. So it's better to >>> split/decouple the reusable parts from the codes. So a vague idea is: >>> >>> 1) Decouple the packet comparing from the netfilter. You've achieved >>> this 99% since the work has been done in a thread. Just let the thread >>> poll sockets directly, then the comparing have the possibility to be >>> reused by other kinds of dataplane. >>> 2) Implement traffic mirror/redirector as filter. >>> 3) Implement TCP seq rewriting as a filter. >>> >>> Then, in primary node, you need just a traffic mirror, which did: >>> - mirror ingress traffic to secondary node >>> - mirror outgress traffic to packet comparing thread >>> >>> And in secondadry node, you need two filters: >>> - A TCP seq rewriter which adjust tcp sequence number. >>> - A traffic redirector which redirect packet from a socket as ingress >>> traffic, and redirect outgress traffic to the socket which could be >>> polled by remote packet comparing thread. >>> Thoughts? >>> >>> Thanks >>> Thanks zhangchen >>> >> >> >> Hi, Jason. >> We consider your suggestion to split/decouple >> the reusable parts from the codes. >> Due to filter plugin are traversed one by one in order >> we will split colo-proxy to three filters in each side. >> >> But in this plan,primary and secondary both have socket >> server,startup is a problem. > > I believe this issue could be solved by reusing socket chardev. > >> >> >> Primary qemu >> Secondary qemu >> +--+ >> +---+ >> | +-+ | | >> +--+ | >> | | | | | >> | | | >> | |guest| | | >> |guest | | >> | | | | | >> | | | >> | +---^--+--+ | | >> +-++---+ | >> | | | | >> |^| | >> | | | | >> ||| | >> | +-+ >> ||| | >> | netfilter | | || | >> netfilter|| | >> | +-+ || | >> +--+ | >> | | | | filter excute order | || | >> | || filter excute order | | >> | | | |+---> | || | >> | || +---> | | >> | | | | | || | >> | || TCP | | >> | | +-+-+ +--v-+++ +-+ | || | >> | +---+ +---++---v+rewriter+ ++ | | >> | | | | |||| | || | >> | | | || | || | | >> | | | mirror | | redirect +> compare | | | >> +> mirror +---> adjust | adjust+-->redirect| | | >> | | | client | | server||| | | | >> | | server | | ack| seq | |client | | | >> | | | | |||| | | | >> | | | || | || | | >> | | +^--+ +^---++-+--+ | | | >> | +---+ ++-+ ++---+ | | >> | | | tx | rx | rx | | | >> |txall | rx | | >> | +-+ | | >> +--+ | >> || >>
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/18/2016 03:05 PM, Zhang Chen wrote: > > > On 01/06/2016 01:16 PM, Jason Wang wrote: >> >> On 01/04/2016 07:17 PM, Zhang Chen wrote: >>> >>> On 01/04/2016 05:46 PM, Jason Wang wrote: On 01/04/2016 04:16 PM, Zhang Chen wrote: > On 01/04/2016 01:37 PM, Jason Wang wrote: >> On 12/31/2015 04:40 PM, Zhang Chen wrote: >>> On 12/31/2015 10:36 AM, Jason Wang wrote: On 12/22/2015 06:42 PM, Zhang Chen wrote: > From: zhangchen> > Hi,all > > This patch add an colo-proxy object, COLO-Proxy is a part of > COLO, > based on qemu netfilter and it's a plugin for qemu netfilter. the > function > keep Secondary VM connect normal to Primary VM and compare > packets > sent by PVM to sent by SVM.if the packet difference,notify > COLO do > checkpoint and send all primary packet has queued. Thanks for the work. I don't object this method but still not convinced that qemu is the best place for this. As been raised in the past discussion, it's almost impossible to cooperate with vhost backends. If we want this to be used in production environment, need to think of a solution for vhost. There's no such worry if we decouple this from qemu. > You can also get the series from: > > https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 > > > > > > Usage: > > primary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object > colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port > > secondary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object > colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port > Have a quick glance at how secondary mode work. What it does is just forwarding packets between a nic and a socket, qemu socket backend did exact the same job. You could even use socket in primary node and let packet compare module talk to both primary and secondary node. >>> If we use qemu socket backend , the same netdev will used by qemu >>> socket and >>> qemu netfilter. this will against qemu net design. and then, when >>> colo >>> do failover, >>> secondary do not have backend to use. that's the real problem. >> Then, maybe it's time to implement changing the netdev of a nic. The >> point here is that what secondary mode did is in fact a netdev >> backend >> instead of a filter ... > Currently, you are right. in colo-proxy V2 code, I just compare IP > packet to > decide whether to do checkpoint. > But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide > it. > because that can reduce frequency of checkpoint and improve > performance. To keep tcp connection well, colo secondary need to > record > primary guest's init seq and adjust secondary guest's ack. if colo do > failover, > secondary also need do this to old tcp connection. qemu socket > can't do this job. So a question here: is it a must to do things (e.g TCP analysis stuffs) at secondary? Looks like we could do this at primary node. And I saw you're doing packet comparing in primary node, any advantages of doing this in primary instead of secondary? >>> We think must to do this in secondary, because if colo do >>> failover,secondary >>> must continues do TCP analysis stuffs to before tcp connection(if not, >>> tcp connection >>> will disconnect in that time), in this time primary already down or >>> disconnect to >>> secondary.so we can't make primary do this TCP analysis stuffs.it can >>> not ensure >>> FT function. >>> >>> Thanks >>> zhangchen >> Makes sense. >> >> Thanks > > Hi~, Jason. > No news for a week. > Can you give me some comments for code. > Let's make colo-proxy work well. Sure. Two main comments/suggestions: - TCP analysis is missed in current version, maybe you point a git tree (or another version of RFC) to me for a better understanding of the design. (Just a skeleton for TCP should be sufficient to discuss). - I prefer to make the code as reusable as possible. So it's better to split/decouple the reusable parts from the codes. So a vague idea is: 1) Decouple the packet comparing from the netfilter. You've achieved this 99% since the work has been done in a thread. Just let the thread poll sockets directly, then the comparing have the possibility to be reused by other kinds of dataplane. 2) Implement traffic mirror/redirector as filter. 3) Implement TCP seq rewriting as a filter. Then, in primary node, you need just a traffic mirror, which did: - mirror ingress traffic to secondary node - mirror
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/06/2016 01:16 PM, Jason Wang wrote: On 01/04/2016 07:17 PM, Zhang Chen wrote: On 01/04/2016 05:46 PM, Jason Wang wrote: On 01/04/2016 04:16 PM, Zhang Chen wrote: On 01/04/2016 01:37 PM, Jason Wang wrote: On 12/31/2015 04:40 PM, Zhang Chen wrote: On 12/31/2015 10:36 AM, Jason Wang wrote: On 12/22/2015 06:42 PM, Zhang Chen wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. Thanks for the work. I don't object this method but still not convinced that qemu is the best place for this. As been raised in the past discussion, it's almost impossible to cooperate with vhost backends. If we want this to be used in production environment, need to think of a solution for vhost. There's no such worry if we decouple this from qemu. You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port Have a quick glance at how secondary mode work. What it does is just forwarding packets between a nic and a socket, qemu socket backend did exact the same job. You could even use socket in primary node and let packet compare module talk to both primary and secondary node. If we use qemu socket backend , the same netdev will used by qemu socket and qemu netfilter. this will against qemu net design. and then, when colo do failover, secondary do not have backend to use. that's the real problem. Then, maybe it's time to implement changing the netdev of a nic. The point here is that what secondary mode did is in fact a netdev backend instead of a filter ... Currently, you are right. in colo-proxy V2 code, I just compare IP packet to decide whether to do checkpoint. But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it. because that can reduce frequency of checkpoint and improve performance. To keep tcp connection well, colo secondary need to record primary guest's init seq and adjust secondary guest's ack. if colo do failover, secondary also need do this to old tcp connection. qemu socket can't do this job. So a question here: is it a must to do things (e.g TCP analysis stuffs) at secondary? Looks like we could do this at primary node. And I saw you're doing packet comparing in primary node, any advantages of doing this in primary instead of secondary? We think must to do this in secondary, because if colo do failover,secondary must continues do TCP analysis stuffs to before tcp connection(if not, tcp connection will disconnect in that time), in this time primary already down or disconnect to secondary.so we can't make primary do this TCP analysis stuffs.it can not ensure FT function. Thanks zhangchen Makes sense. Thanks Hi~, Jason. No news for a week. Can you give me some comments for code. Let's make colo-proxy work well. Thanks zhangchen and another problem is do failover, if we use qemu socket to be backend in secondary, when colo do failover, I don't know how to change secondary be a normal qemu, if you know, please tell me. Current qemu couldn't do this, but I mean we implement something like nic_change_backend which can change nic's peer(s). With this, in secondary, we can replace the socket backend with whatever you want (e.g tap or other). Thanks Thanks for your revew zhangchen . . -- Thanks zhangchen
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/11/2016 08:59 PM, Dr. David Alan Gilbert wrote: * Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote: On 01/08/2016 07:19 PM, Dr. David Alan Gilbert wrote: * Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Are you sure that tag is correct? The series of commits doesn't seem to match up with the set of commits posted. Dave Yes, it is. we have some code fix in other colo unrelated file. in email, I just send colo related part. That doesn't seem to be what's happening in that git tree. For example, your patch '[RFC PATCH v2 01/10] Init colo-proxy object based on netfilter' adds the colo-proxy object to qemu-options.hx, but in that git tree it comes from Li Zhijian's 'add proxy prototype' patch. If you're going to include a git link with a series then please make sure it contains exactly the patches posted. It's OK to add some more patches somewhere, e.g. on a different tag or branch, but make sure the one that you post for the series matches the series posted. Dave Make sense. I have fix it. https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Thanks zhangchen zhangchen Thanks for review Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port NOTE: queue must set "all". See enum NetFilterDirection for detail. colo-proxy need queue all packets colo-proxy V2 just can compare ip packet ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand)is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. V2: rebase colo-proxy with qemu-colo-v2.2-periodic-mode fix dave's comments fix wency's comments fix zhanghailiang's comments v1: initial patch. zhangchen (10): Init colo-proxy object based on netfilter Jhash: add linux kernel jhashtable in qemu Colo-proxy: add colo-proxy framework Colo-proxy: add data structure and jhash func net/colo-proxy: Add colo interface to use proxy net/colo-proxy: add socket used by forward func net/colo-proxy: Add packet enqueue & handle func net/colo-proxy: Handle packet and connection net/colo-proxy: Compare pri pkt to sec pkt net/colo-proxy: Colo-proxy do checkpoint and clear include/qemu/jhash.h | 61 net/Makefile.objs| 1 + net/colo-proxy.c | 939 +++ net/colo-proxy.h | 24 ++ qemu-options.hx | 6 + trace-events | 8 + vl.c | 3 +- 7 files changed, 1041 insertions(+), 1 deletion(-) create mode 100644 include/qemu/jhash.h create mode 100644 net/colo-proxy.c create mode 100644 net/colo-proxy.h -- 1.9.1 -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK . -- Thanks zhangchen -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK . -- Thanks zhangchen
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote: > > > On 01/08/2016 07:19 PM, Dr. David Alan Gilbert wrote: > >* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote: > >>From: zhangchen> >> > >>Hi,all > >> > >>This patch add an colo-proxy object, COLO-Proxy is a part of COLO, > >>based on qemu netfilter and it's a plugin for qemu netfilter. the function > >>keep Secondary VM connect normal to Primary VM and compare packets > >>sent by PVM to sent by SVM.if the packet difference,notify COLO do > >>checkpoint and send all primary packet has queued. > >> > >>You can also get the series from: > >> > >>https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 > >Are you sure that tag is correct? The series of commits doesn't seem to match > >up with the set of commits posted. > > > >Dave > > Yes, it is. we have some code fix in other colo unrelated file. > in email, I just send colo related part. That doesn't seem to be what's happening in that git tree. For example, your patch '[RFC PATCH v2 01/10] Init colo-proxy object based on netfilter' adds the colo-proxy object to qemu-options.hx, but in that git tree it comes from Li Zhijian's 'add proxy prototype' patch. If you're going to include a git link with a series then please make sure it contains exactly the patches posted. It's OK to add some more patches somewhere, e.g. on a different tag or branch, but make sure the one that you post for the series matches the series posted. Dave > > zhangchen > Thanks for review > > >>Usage: > >> > >>primary: > >>-netdev tap,id=bn0 -device e1000,netdev=bn0 > >>-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port > >> > >>secondary: > >>-netdev tap,id=bn0 -device e1000,netdev=bn0 > >>-object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port > >> > >>NOTE: > >>queue must set "all". See enum NetFilterDirection for detail. > >>colo-proxy need queue all packets > >>colo-proxy V2 just can compare ip packet > >> > >> > >>## Background > >> > >>COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop > >>Service) > >>project is a high availability solution. Both Primary VM (PVM) and > >>Secondary VM > >>(SVM) run in parallel. They receive the same request from client, and > >>generate > >>responses in parallel too. If the response packets from PVM and SVM are > >>identical, they are released immediately. Otherwise, a VM checkpoint (on > >>demand)is conducted. > >> > >>Paper: > >>http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > >> > >>COLO on Xen: > >>http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > >> > >>COLO on Qemu/KVM: > >>http://wiki.qemu.org/Features/COLO > >> > >>By the needs of capturing response packets from PVM and SVM and finding out > >>whether they are identical, we introduce a new module to qemu networking > >>called colo-proxy. > >> > >>V2: > >> rebase colo-proxy with qemu-colo-v2.2-periodic-mode > >> fix dave's comments > >> fix wency's comments > >> fix zhanghailiang's comments > >> > >>v1: > >> initial patch. > >> > >> > >> > >>zhangchen (10): > >> Init colo-proxy object based on netfilter > >> Jhash: add linux kernel jhashtable in qemu > >> Colo-proxy: add colo-proxy framework > >> Colo-proxy: add data structure and jhash func > >> net/colo-proxy: Add colo interface to use proxy > >> net/colo-proxy: add socket used by forward func > >> net/colo-proxy: Add packet enqueue & handle func > >> net/colo-proxy: Handle packet and connection > >> net/colo-proxy: Compare pri pkt to sec pkt > >> net/colo-proxy: Colo-proxy do checkpoint and clear > >> > >> include/qemu/jhash.h | 61 > >> net/Makefile.objs| 1 + > >> net/colo-proxy.c | 939 > >> +++ > >> net/colo-proxy.h | 24 ++ > >> qemu-options.hx | 6 + > >> trace-events | 8 + > >> vl.c | 3 +- > >> 7 files changed, 1041 insertions(+), 1 deletion(-) > >> create mode 100644 include/qemu/jhash.h > >> create mode 100644 net/colo-proxy.c > >> create mode 100644 net/colo-proxy.h > >> > >>-- > >>1.9.1 > >> > >> > >> > >-- > >Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK > > > > > >. > > > > -- > Thanks > zhangchen > > > -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/08/2016 07:19 PM, Dr. David Alan Gilbert wrote: * Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Are you sure that tag is correct? The series of commits doesn't seem to match up with the set of commits posted. Dave Yes, it is. we have some code fix in other colo unrelated file. in email, I just send colo related part. zhangchen Thanks for review Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port NOTE: queue must set "all". See enum NetFilterDirection for detail. colo-proxy need queue all packets colo-proxy V2 just can compare ip packet ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand)is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. V2: rebase colo-proxy with qemu-colo-v2.2-periodic-mode fix dave's comments fix wency's comments fix zhanghailiang's comments v1: initial patch. zhangchen (10): Init colo-proxy object based on netfilter Jhash: add linux kernel jhashtable in qemu Colo-proxy: add colo-proxy framework Colo-proxy: add data structure and jhash func net/colo-proxy: Add colo interface to use proxy net/colo-proxy: add socket used by forward func net/colo-proxy: Add packet enqueue & handle func net/colo-proxy: Handle packet and connection net/colo-proxy: Compare pri pkt to sec pkt net/colo-proxy: Colo-proxy do checkpoint and clear include/qemu/jhash.h | 61 net/Makefile.objs| 1 + net/colo-proxy.c | 939 +++ net/colo-proxy.h | 24 ++ qemu-options.hx | 6 + trace-events | 8 + vl.c | 3 +- 7 files changed, 1041 insertions(+), 1 deletion(-) create mode 100644 include/qemu/jhash.h create mode 100644 net/colo-proxy.c create mode 100644 net/colo-proxy.h -- 1.9.1 -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK . -- Thanks zhangchen
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
* Zhang Chen (zhangchen.f...@cn.fujitsu.com) wrote: > From: zhangchen> > Hi,all > > This patch add an colo-proxy object, COLO-Proxy is a part of COLO, > based on qemu netfilter and it's a plugin for qemu netfilter. the function > keep Secondary VM connect normal to Primary VM and compare packets > sent by PVM to sent by SVM.if the packet difference,notify COLO do > checkpoint and send all primary packet has queued. > > You can also get the series from: > > https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Are you sure that tag is correct? The series of commits doesn't seem to match up with the set of commits posted. Dave > > Usage: > > primary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port > > secondary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port > > NOTE: > queue must set "all". See enum NetFilterDirection for detail. > colo-proxy need queue all packets > colo-proxy V2 just can compare ip packet > > > ## Background > > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) > project is a high availability solution. Both Primary VM (PVM) and Secondary > VM > (SVM) run in parallel. They receive the same request from client, and generate > responses in parallel too. If the response packets from PVM and SVM are > identical, they are released immediately. Otherwise, a VM checkpoint (on > demand)is conducted. > > Paper: > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > > COLO on Xen: > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > > COLO on Qemu/KVM: > http://wiki.qemu.org/Features/COLO > > By the needs of capturing response packets from PVM and SVM and finding out > whether they are identical, we introduce a new module to qemu networking > called colo-proxy. > > V2: > rebase colo-proxy with qemu-colo-v2.2-periodic-mode > fix dave's comments > fix wency's comments > fix zhanghailiang's comments > > v1: > initial patch. > > > > zhangchen (10): > Init colo-proxy object based on netfilter > Jhash: add linux kernel jhashtable in qemu > Colo-proxy: add colo-proxy framework > Colo-proxy: add data structure and jhash func > net/colo-proxy: Add colo interface to use proxy > net/colo-proxy: add socket used by forward func > net/colo-proxy: Add packet enqueue & handle func > net/colo-proxy: Handle packet and connection > net/colo-proxy: Compare pri pkt to sec pkt > net/colo-proxy: Colo-proxy do checkpoint and clear > > include/qemu/jhash.h | 61 > net/Makefile.objs| 1 + > net/colo-proxy.c | 939 > +++ > net/colo-proxy.h | 24 ++ > qemu-options.hx | 6 + > trace-events | 8 + > vl.c | 3 +- > 7 files changed, 1041 insertions(+), 1 deletion(-) > create mode 100644 include/qemu/jhash.h > create mode 100644 net/colo-proxy.c > create mode 100644 net/colo-proxy.h > > -- > 1.9.1 > > > -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
* Jason Wang (jasow...@redhat.com) wrote: > > > On 01/05/2016 12:52 AM, Dr. David Alan Gilbert wrote: > > * Jason Wang (jasow...@redhat.com) wrote: > >> > >> On 01/04/2016 04:16 PM, Zhang Chen wrote: > >>> > >>> On 01/04/2016 01:37 PM, Jason Wang wrote: > On 12/31/2015 04:40 PM, Zhang Chen wrote: > > On 12/31/2015 10:36 AM, Jason Wang wrote: > >> On 12/22/2015 06:42 PM, Zhang Chen wrote: > >>> From: zhangchen> >>> > >>> Hi,all > >>> > >>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO, > >>> based on qemu netfilter and it's a plugin for qemu netfilter. the > >>> function > >>> keep Secondary VM connect normal to Primary VM and compare packets > >>> sent by PVM to sent by SVM.if the packet difference,notify COLO do > >>> checkpoint and send all primary packet has queued. > >> Thanks for the work. I don't object this method but still not > >> convinced > >> that qemu is the best place for this. > >> > >> As been raised in the past discussion, it's almost impossible to > >> cooperate with vhost backends. If we want this to be used in > >> production > >> environment, need to think of a solution for vhost. There's no such > >> worry if we decouple this from qemu. > >> > >>> You can also get the series from: > >>> > >>> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 > >>> > >>> > >>> > >>> Usage: > >>> > >>> primary: > >>> -netdev tap,id=bn0 -device e1000,netdev=bn0 > >>> -object > >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port > >>> > >>> secondary: > >>> -netdev tap,id=bn0 -device e1000,netdev=bn0 > >>> -object > >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port > >> Have a quick glance at how secondary mode work. What it does is just > >> forwarding packets between a nic and a socket, qemu socket backend did > >> exact the same job. You could even use socket in primary node and let > >> packet compare module talk to both primary and secondary node. > > If we use qemu socket backend , the same netdev will used by qemu > > socket and > > qemu netfilter. this will against qemu net design. and then, when colo > > do failover, > > secondary do not have backend to use. that's the real problem. > Then, maybe it's time to implement changing the netdev of a nic. The > point here is that what secondary mode did is in fact a netdev backend > instead of a filter ... > >>> Currently, you are right. in colo-proxy V2 code, I just compare IP > >>> packet to > >>> decide whether to do checkpoint. > >>> But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it. > >>> because that can reduce frequency of checkpoint and improve > >>> performance. To keep tcp connection well, colo secondary need to record > >>> primary guest's init seq and adjust secondary guest's ack. if colo do > >>> failover, > >>> secondary also need do this to old tcp connection. qemu socket > >>> can't do this job. > >> So a question here: is it a must to do things (e.g TCP analysis stuffs) > >> at secondary? Looks like we could do this at primary node. And I saw > >> you're doing packet comparing in primary node, any advantages of doing > >> this in primary instead of secondary? > > It needs to do this on the secondary; the trick is that things like TCP > > sequence > > numbers are likely to be different on the primary and secondary; the kernel > > colo-proxy > > implementation solved this problem by rewriting the sequence numbers on > > the secondary to match the primary, after a failover, the secondary has > > to keep doing that rewrite to ensure existing connections are OK. > > Thus it's holding some state about the current connections. > > I see. > > > I think also, to be able to do a 2nd failover (i.e. recover from the 1st > > failure > > and then sometime later have another) you'd have to sync this > > state over to a new host, so again that says the state needs to be part of > > qemu or at least easily available to it. > > > > Dave > > Right, if it does thing like tcp seq rewrite (which is missed in current > version), it works much more like a netfilter. Wonder if the function is > generic enough for users other than colo. I can imagine the sequence number rework might be, but I doubt the packet comparison is. Dave > Thanks > > > > >>> and another problem is do failover, if we use qemu socket > >>> to be backend in secondary, when colo do failover, I don't know how to > >>> change > >>> secondary be a normal qemu, if you know, please tell me. > >> Current qemu couldn't do this, but I mean we implement something like > >> nic_change_backend which can change nic's peer(s). With this, in > >> secondary, we can replace the socket backend with whatever you want (e.g > >> tap or other). > >>
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/04/2016 07:17 PM, Zhang Chen wrote: > > > On 01/04/2016 05:46 PM, Jason Wang wrote: >> >> On 01/04/2016 04:16 PM, Zhang Chen wrote: >>> >>> On 01/04/2016 01:37 PM, Jason Wang wrote: On 12/31/2015 04:40 PM, Zhang Chen wrote: > On 12/31/2015 10:36 AM, Jason Wang wrote: >> On 12/22/2015 06:42 PM, Zhang Chen wrote: >>> From: zhangchen>>> >>> Hi,all >>> >>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO, >>> based on qemu netfilter and it's a plugin for qemu netfilter. the >>> function >>> keep Secondary VM connect normal to Primary VM and compare packets >>> sent by PVM to sent by SVM.if the packet difference,notify COLO do >>> checkpoint and send all primary packet has queued. >> Thanks for the work. I don't object this method but still not >> convinced >> that qemu is the best place for this. >> >> As been raised in the past discussion, it's almost impossible to >> cooperate with vhost backends. If we want this to be used in >> production >> environment, need to think of a solution for vhost. There's no such >> worry if we decouple this from qemu. >> >>> You can also get the series from: >>> >>> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 >>> >>> >>> >>> >>> Usage: >>> >>> primary: >>> -netdev tap,id=bn0 -device e1000,netdev=bn0 >>> -object >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port >>> >>> secondary: >>> -netdev tap,id=bn0 -device e1000,netdev=bn0 >>> -object >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port >> Have a quick glance at how secondary mode work. What it does is just >> forwarding packets between a nic and a socket, qemu socket >> backend did >> exact the same job. You could even use socket in primary node and >> let >> packet compare module talk to both primary and secondary node. > If we use qemu socket backend , the same netdev will used by qemu > socket and > qemu netfilter. this will against qemu net design. and then, when > colo > do failover, > secondary do not have backend to use. that's the real problem. Then, maybe it's time to implement changing the netdev of a nic. The point here is that what secondary mode did is in fact a netdev backend instead of a filter ... >>> Currently, you are right. in colo-proxy V2 code, I just compare IP >>> packet to >>> decide whether to do checkpoint. >>> But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it. >>> because that can reduce frequency of checkpoint and improve >>> performance. To keep tcp connection well, colo secondary need to record >>> primary guest's init seq and adjust secondary guest's ack. if colo do >>> failover, >>> secondary also need do this to old tcp connection. qemu socket >>> can't do this job. >> So a question here: is it a must to do things (e.g TCP analysis stuffs) >> at secondary? Looks like we could do this at primary node. And I saw >> you're doing packet comparing in primary node, any advantages of doing >> this in primary instead of secondary? > > We think must to do this in secondary, because if colo do > failover,secondary > must continues do TCP analysis stuffs to before tcp connection(if not, > tcp connection > will disconnect in that time), in this time primary already down or > disconnect to > secondary.so we can't make primary do this TCP analysis stuffs.it can > not ensure > FT function. > > Thanks > zhangchen Makes sense. Thanks > >>> and another problem is do failover, if we use qemu socket >>> to be backend in secondary, when colo do failover, I don't know how to >>> change >>> secondary be a normal qemu, if you know, please tell me. >> Current qemu couldn't do this, but I mean we implement something like >> nic_change_backend which can change nic's peer(s). With this, in >> secondary, we can replace the socket backend with whatever you want (e.g >> tap or other). >> >> Thanks >> >>> >>> Thanks for your revew >>> zhangchen >> >> >> . >> >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/05/2016 12:52 AM, Dr. David Alan Gilbert wrote: > * Jason Wang (jasow...@redhat.com) wrote: >> >> On 01/04/2016 04:16 PM, Zhang Chen wrote: >>> >>> On 01/04/2016 01:37 PM, Jason Wang wrote: On 12/31/2015 04:40 PM, Zhang Chen wrote: > On 12/31/2015 10:36 AM, Jason Wang wrote: >> On 12/22/2015 06:42 PM, Zhang Chen wrote: >>> From: zhangchen>>> >>> Hi,all >>> >>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO, >>> based on qemu netfilter and it's a plugin for qemu netfilter. the >>> function >>> keep Secondary VM connect normal to Primary VM and compare packets >>> sent by PVM to sent by SVM.if the packet difference,notify COLO do >>> checkpoint and send all primary packet has queued. >> Thanks for the work. I don't object this method but still not >> convinced >> that qemu is the best place for this. >> >> As been raised in the past discussion, it's almost impossible to >> cooperate with vhost backends. If we want this to be used in >> production >> environment, need to think of a solution for vhost. There's no such >> worry if we decouple this from qemu. >> >>> You can also get the series from: >>> >>> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 >>> >>> >>> >>> Usage: >>> >>> primary: >>> -netdev tap,id=bn0 -device e1000,netdev=bn0 >>> -object >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port >>> >>> secondary: >>> -netdev tap,id=bn0 -device e1000,netdev=bn0 >>> -object >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port >> Have a quick glance at how secondary mode work. What it does is just >> forwarding packets between a nic and a socket, qemu socket backend did >> exact the same job. You could even use socket in primary node and let >> packet compare module talk to both primary and secondary node. > If we use qemu socket backend , the same netdev will used by qemu > socket and > qemu netfilter. this will against qemu net design. and then, when colo > do failover, > secondary do not have backend to use. that's the real problem. Then, maybe it's time to implement changing the netdev of a nic. The point here is that what secondary mode did is in fact a netdev backend instead of a filter ... >>> Currently, you are right. in colo-proxy V2 code, I just compare IP >>> packet to >>> decide whether to do checkpoint. >>> But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it. >>> because that can reduce frequency of checkpoint and improve >>> performance. To keep tcp connection well, colo secondary need to record >>> primary guest's init seq and adjust secondary guest's ack. if colo do >>> failover, >>> secondary also need do this to old tcp connection. qemu socket >>> can't do this job. >> So a question here: is it a must to do things (e.g TCP analysis stuffs) >> at secondary? Looks like we could do this at primary node. And I saw >> you're doing packet comparing in primary node, any advantages of doing >> this in primary instead of secondary? > It needs to do this on the secondary; the trick is that things like TCP > sequence > numbers are likely to be different on the primary and secondary; the kernel > colo-proxy > implementation solved this problem by rewriting the sequence numbers on > the secondary to match the primary, after a failover, the secondary has > to keep doing that rewrite to ensure existing connections are OK. > Thus it's holding some state about the current connections. I see. > I think also, to be able to do a 2nd failover (i.e. recover from the 1st > failure > and then sometime later have another) you'd have to sync this > state over to a new host, so again that says the state needs to be part of > qemu or at least easily available to it. > > Dave Right, if it does thing like tcp seq rewrite (which is missed in current version), it works much more like a netfilter. Wonder if the function is generic enough for users other than colo. Thanks > >>> and another problem is do failover, if we use qemu socket >>> to be backend in secondary, when colo do failover, I don't know how to >>> change >>> secondary be a normal qemu, if you know, please tell me. >> Current qemu couldn't do this, but I mean we implement something like >> nic_change_backend which can change nic's peer(s). With this, in >> secondary, we can replace the socket backend with whatever you want (e.g >> tap or other). >> >> Thanks >> >>> >>> Thanks for your revew >>> zhangchen > -- > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/04/2016 04:16 PM, Zhang Chen wrote: > > > On 01/04/2016 01:37 PM, Jason Wang wrote: >> >> On 12/31/2015 04:40 PM, Zhang Chen wrote: >>> >>> On 12/31/2015 10:36 AM, Jason Wang wrote: On 12/22/2015 06:42 PM, Zhang Chen wrote: > From: zhangchen> > Hi,all > > This patch add an colo-proxy object, COLO-Proxy is a part of COLO, > based on qemu netfilter and it's a plugin for qemu netfilter. the > function > keep Secondary VM connect normal to Primary VM and compare packets > sent by PVM to sent by SVM.if the packet difference,notify COLO do > checkpoint and send all primary packet has queued. Thanks for the work. I don't object this method but still not convinced that qemu is the best place for this. As been raised in the past discussion, it's almost impossible to cooperate with vhost backends. If we want this to be used in production environment, need to think of a solution for vhost. There's no such worry if we decouple this from qemu. > You can also get the series from: > > https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 > > > > Usage: > > primary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object > colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port > > secondary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object > colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port Have a quick glance at how secondary mode work. What it does is just forwarding packets between a nic and a socket, qemu socket backend did exact the same job. You could even use socket in primary node and let packet compare module talk to both primary and secondary node. >>> If we use qemu socket backend , the same netdev will used by qemu >>> socket and >>> qemu netfilter. this will against qemu net design. and then, when colo >>> do failover, >>> secondary do not have backend to use. that's the real problem. >> Then, maybe it's time to implement changing the netdev of a nic. The >> point here is that what secondary mode did is in fact a netdev backend >> instead of a filter ... > > Currently, you are right. in colo-proxy V2 code, I just compare IP > packet to > decide whether to do checkpoint. > But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it. > because that can reduce frequency of checkpoint and improve > performance. To keep tcp connection well, colo secondary need to record > primary guest's init seq and adjust secondary guest's ack. if colo do > failover, > secondary also need do this to old tcp connection. qemu socket > can't do this job. So a question here: is it a must to do things (e.g TCP analysis stuffs) at secondary? Looks like we could do this at primary node. And I saw you're doing packet comparing in primary node, any advantages of doing this in primary instead of secondary? > and another problem is do failover, if we use qemu socket > to be backend in secondary, when colo do failover, I don't know how to > change > secondary be a normal qemu, if you know, please tell me. Current qemu couldn't do this, but I mean we implement something like nic_change_backend which can change nic's peer(s). With this, in secondary, we can replace the socket backend with whatever you want (e.g tap or other). Thanks > > > Thanks for your revew > zhangchen
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/04/2016 05:46 PM, Jason Wang wrote: On 01/04/2016 04:16 PM, Zhang Chen wrote: On 01/04/2016 01:37 PM, Jason Wang wrote: On 12/31/2015 04:40 PM, Zhang Chen wrote: On 12/31/2015 10:36 AM, Jason Wang wrote: On 12/22/2015 06:42 PM, Zhang Chen wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. Thanks for the work. I don't object this method but still not convinced that qemu is the best place for this. As been raised in the past discussion, it's almost impossible to cooperate with vhost backends. If we want this to be used in production environment, need to think of a solution for vhost. There's no such worry if we decouple this from qemu. You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port Have a quick glance at how secondary mode work. What it does is just forwarding packets between a nic and a socket, qemu socket backend did exact the same job. You could even use socket in primary node and let packet compare module talk to both primary and secondary node. If we use qemu socket backend , the same netdev will used by qemu socket and qemu netfilter. this will against qemu net design. and then, when colo do failover, secondary do not have backend to use. that's the real problem. Then, maybe it's time to implement changing the netdev of a nic. The point here is that what secondary mode did is in fact a netdev backend instead of a filter ... Currently, you are right. in colo-proxy V2 code, I just compare IP packet to decide whether to do checkpoint. But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it. because that can reduce frequency of checkpoint and improve performance. To keep tcp connection well, colo secondary need to record primary guest's init seq and adjust secondary guest's ack. if colo do failover, secondary also need do this to old tcp connection. qemu socket can't do this job. So a question here: is it a must to do things (e.g TCP analysis stuffs) at secondary? Looks like we could do this at primary node. And I saw you're doing packet comparing in primary node, any advantages of doing this in primary instead of secondary? We think must to do this in secondary, because if colo do failover,secondary must continues do TCP analysis stuffs to before tcp connection(if not, tcp connection will disconnect in that time), in this time primary already down or disconnect to secondary.so we can't make primary do this TCP analysis stuffs.it can not ensure FT function. Thanks zhangchen and another problem is do failover, if we use qemu socket to be backend in secondary, when colo do failover, I don't know how to change secondary be a normal qemu, if you know, please tell me. Current qemu couldn't do this, but I mean we implement something like nic_change_backend which can change nic's peer(s). With this, in secondary, we can replace the socket backend with whatever you want (e.g tap or other). Thanks Thanks for your revew zhangchen . -- Thanks zhangchen
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 01/04/2016 01:37 PM, Jason Wang wrote: On 12/31/2015 04:40 PM, Zhang Chen wrote: On 12/31/2015 10:36 AM, Jason Wang wrote: On 12/22/2015 06:42 PM, Zhang Chen wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. Thanks for the work. I don't object this method but still not convinced that qemu is the best place for this. As been raised in the past discussion, it's almost impossible to cooperate with vhost backends. If we want this to be used in production environment, need to think of a solution for vhost. There's no such worry if we decouple this from qemu. You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port Have a quick glance at how secondary mode work. What it does is just forwarding packets between a nic and a socket, qemu socket backend did exact the same job. You could even use socket in primary node and let packet compare module talk to both primary and secondary node. If we use qemu socket backend , the same netdev will used by qemu socket and qemu netfilter. this will against qemu net design. and then, when colo do failover, secondary do not have backend to use. that's the real problem. Then, maybe it's time to implement changing the netdev of a nic. The point here is that what secondary mode did is in fact a netdev backend instead of a filter ... Currently, you are right. in colo-proxy V2 code, I just compare IP packet to decide whether to do checkpoint. But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it. because that can reduce frequency of checkpoint and improve performance. To keep tcp connection well, colo secondary need to record primary guest's init seq and adjust secondary guest's ack. if colo do failover, secondary also need do this to old tcp connection. qemu socket can't do this job. and another problem is do failover, if we use qemu socket to be backend in secondary, when colo do failover, I don't know how to change secondary be a normal qemu, if you know, please tell me. Thanks for your revew zhangchen Thanks zhangchen NOTE: queue must set "all". See enum NetFilterDirection for detail. colo-proxy need queue all packets colo-proxy V2 just can compare ip packet ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand)is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. V2: rebase colo-proxy with qemu-colo-v2.2-periodic-mode fix dave's comments fix wency's comments fix zhanghailiang's comments v1: initial patch. zhangchen (10): Init colo-proxy object based on netfilter Jhash: add linux kernel jhashtable in qemu Colo-proxy: add colo-proxy framework Colo-proxy: add data structure and jhash func net/colo-proxy: Add colo interface to use proxy net/colo-proxy: add socket used by forward func net/colo-proxy: Add packet enqueue & handle func net/colo-proxy: Handle packet and connection net/colo-proxy: Compare pri pkt to sec pkt net/colo-proxy: Colo-proxy do checkpoint and clear include/qemu/jhash.h | 61 net/Makefile.objs| 1 + net/colo-proxy.c | 939 +++ net/colo-proxy.h | 24 ++ qemu-options.hx | 6 + trace-events | 8 + vl.c | 3 +- 7 files changed, 1041 insertions(+), 1 deletion(-) create mode 100644 include/qemu/jhash.h create mode 100644 net/colo-proxy.c create mode 100644 net/colo-proxy.h . . -- Thanks zhangchen
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
* Jason Wang (jasow...@redhat.com) wrote: > > > On 01/04/2016 04:16 PM, Zhang Chen wrote: > > > > > > On 01/04/2016 01:37 PM, Jason Wang wrote: > >> > >> On 12/31/2015 04:40 PM, Zhang Chen wrote: > >>> > >>> On 12/31/2015 10:36 AM, Jason Wang wrote: > On 12/22/2015 06:42 PM, Zhang Chen wrote: > > From: zhangchen> > > > Hi,all > > > > This patch add an colo-proxy object, COLO-Proxy is a part of COLO, > > based on qemu netfilter and it's a plugin for qemu netfilter. the > > function > > keep Secondary VM connect normal to Primary VM and compare packets > > sent by PVM to sent by SVM.if the packet difference,notify COLO do > > checkpoint and send all primary packet has queued. > Thanks for the work. I don't object this method but still not > convinced > that qemu is the best place for this. > > As been raised in the past discussion, it's almost impossible to > cooperate with vhost backends. If we want this to be used in > production > environment, need to think of a solution for vhost. There's no such > worry if we decouple this from qemu. > > > You can also get the series from: > > > > https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 > > > > > > > > Usage: > > > > primary: > > -netdev tap,id=bn0 -device e1000,netdev=bn0 > > -object > > colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port > > > > secondary: > > -netdev tap,id=bn0 -device e1000,netdev=bn0 > > -object > > colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port > Have a quick glance at how secondary mode work. What it does is just > forwarding packets between a nic and a socket, qemu socket backend did > exact the same job. You could even use socket in primary node and let > packet compare module talk to both primary and secondary node. > >>> If we use qemu socket backend , the same netdev will used by qemu > >>> socket and > >>> qemu netfilter. this will against qemu net design. and then, when colo > >>> do failover, > >>> secondary do not have backend to use. that's the real problem. > >> Then, maybe it's time to implement changing the netdev of a nic. The > >> point here is that what secondary mode did is in fact a netdev backend > >> instead of a filter ... > > > > Currently, you are right. in colo-proxy V2 code, I just compare IP > > packet to > > decide whether to do checkpoint. > > But, in colo-proxy V3 I will compare tcp,icmp,udp packet to decide it. > > because that can reduce frequency of checkpoint and improve > > performance. To keep tcp connection well, colo secondary need to record > > primary guest's init seq and adjust secondary guest's ack. if colo do > > failover, > > secondary also need do this to old tcp connection. qemu socket > > can't do this job. > > So a question here: is it a must to do things (e.g TCP analysis stuffs) > at secondary? Looks like we could do this at primary node. And I saw > you're doing packet comparing in primary node, any advantages of doing > this in primary instead of secondary? It needs to do this on the secondary; the trick is that things like TCP sequence numbers are likely to be different on the primary and secondary; the kernel colo-proxy implementation solved this problem by rewriting the sequence numbers on the secondary to match the primary, after a failover, the secondary has to keep doing that rewrite to ensure existing connections are OK. Thus it's holding some state about the current connections. I think also, to be able to do a 2nd failover (i.e. recover from the 1st failure and then sometime later have another) you'd have to sync this state over to a new host, so again that says the state needs to be part of qemu or at least easily available to it. Dave > > and another problem is do failover, if we use qemu socket > > to be backend in secondary, when colo do failover, I don't know how to > > change > > secondary be a normal qemu, if you know, please tell me. > > Current qemu couldn't do this, but I mean we implement something like > nic_change_backend which can change nic's peer(s). With this, in > secondary, we can replace the socket backend with whatever you want (e.g > tap or other). > > Thanks > > > > > > > Thanks for your revew > > zhangchen > -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 12/31/2015 04:02 PM, Li Zhijian wrote: > > > On 12/31/2015 10:36 AM, Jason Wang wrote: >> >> >> On 12/22/2015 06:42 PM, Zhang Chen wrote: >>> From: zhangchen>>> >>> Hi,all >>> >>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO, >>> based on qemu netfilter and it's a plugin for qemu netfilter. the >>> function >>> keep Secondary VM connect normal to Primary VM and compare packets >>> sent by PVM to sent by SVM.if the packet difference,notify COLO do >>> checkpoint and send all primary packet has queued. >> >> Thanks for the work. I don't object this method but still not convinced >> that qemu is the best place for this. >> >> As been raised in the past discussion, it's almost impossible to >> cooperate with vhost backends. If we want this to be used in production >> environment, need to think of a solution for vhost. There's no such >> worry if we decouple this from qemu. > > Yes, I agree with you. But not everything is perfect at the beginning. > If we implement proxy as kernel modules, maybe some one will ask how > about the packet(e.g. vhost-user) that host kernel can not touch. Then I think the best place is still in userspace but not qemu. With this the packet comparing module can easily accept the mirrored traffic from both kernel and userspace. And qemu part can focus on the infrastructures to support them (e.g mirroring traffic to a socket or elsewhere). > > As you said, colo-proxy in qemu will not support vhost scene, but it > can make > colo more easier to be used so that more and more pepole will join us > to test it. > I'm looking forward to that day. Agree, so moving this out of qemu can greatly reduce the complexity and make it easier to be merged. > > if everything goes fine, I think colo can support vhost scene in other > way(such as > introduce extra proxy module in kernel) in the feature. I believe we don't want duplicate codes & bugs(fixes) in two places. > > So, I think colo-proxy in qemu is a good choice for current COLO. > > Thanks > Li
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 12/31/2015 04:40 PM, Zhang Chen wrote: > > > On 12/31/2015 10:36 AM, Jason Wang wrote: >> >> On 12/22/2015 06:42 PM, Zhang Chen wrote: >>> From: zhangchen>>> >>> Hi,all >>> >>> This patch add an colo-proxy object, COLO-Proxy is a part of COLO, >>> based on qemu netfilter and it's a plugin for qemu netfilter. the >>> function >>> keep Secondary VM connect normal to Primary VM and compare packets >>> sent by PVM to sent by SVM.if the packet difference,notify COLO do >>> checkpoint and send all primary packet has queued. >> Thanks for the work. I don't object this method but still not convinced >> that qemu is the best place for this. >> >> As been raised in the past discussion, it's almost impossible to >> cooperate with vhost backends. If we want this to be used in production >> environment, need to think of a solution for vhost. There's no such >> worry if we decouple this from qemu. >> >>> You can also get the series from: >>> >>> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 >>> >>> >>> Usage: >>> >>> primary: >>> -netdev tap,id=bn0 -device e1000,netdev=bn0 >>> -object >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port >>> >>> secondary: >>> -netdev tap,id=bn0 -device e1000,netdev=bn0 >>> -object >>> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port >> Have a quick glance at how secondary mode work. What it does is just >> forwarding packets between a nic and a socket, qemu socket backend did >> exact the same job. You could even use socket in primary node and let >> packet compare module talk to both primary and secondary node. > > If we use qemu socket backend , the same netdev will used by qemu > socket and > qemu netfilter. this will against qemu net design. and then, when colo > do failover, > secondary do not have backend to use. that's the real problem. Then, maybe it's time to implement changing the netdev of a nic. The point here is that what secondary mode did is in fact a netdev backend instead of a filter ... > > > Thanks > zhangchen > >>> NOTE: >>> queue must set "all". See enum NetFilterDirection for detail. >>> colo-proxy need queue all packets >>> colo-proxy V2 just can compare ip packet >>> >>> >>> ## Background >>> >>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >>> Service) >>> project is a high availability solution. Both Primary VM (PVM) and >>> Secondary VM >>> (SVM) run in parallel. They receive the same request from client, >>> and generate >>> responses in parallel too. If the response packets from PVM and SVM are >>> identical, they are released immediately. Otherwise, a VM checkpoint >>> (on >>> demand)is conducted. >>> >>> Paper: >>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >>> >>> COLO on Xen: >>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >>> >>> COLO on Qemu/KVM: >>> http://wiki.qemu.org/Features/COLO >>> >>> By the needs of capturing response packets from PVM and SVM and >>> finding out >>> whether they are identical, we introduce a new module to qemu >>> networking >>> called colo-proxy. >>> >>> V2: >>>rebase colo-proxy with qemu-colo-v2.2-periodic-mode >>>fix dave's comments >>>fix wency's comments >>>fix zhanghailiang's comments >>> >>> v1: >>>initial patch. >>> >>> >>> >>> zhangchen (10): >>>Init colo-proxy object based on netfilter >>>Jhash: add linux kernel jhashtable in qemu >>>Colo-proxy: add colo-proxy framework >>>Colo-proxy: add data structure and jhash func >>>net/colo-proxy: Add colo interface to use proxy >>>net/colo-proxy: add socket used by forward func >>>net/colo-proxy: Add packet enqueue & handle func >>>net/colo-proxy: Handle packet and connection >>>net/colo-proxy: Compare pri pkt to sec pkt >>>net/colo-proxy: Colo-proxy do checkpoint and clear >>> >>> include/qemu/jhash.h | 61 >>> net/Makefile.objs| 1 + >>> net/colo-proxy.c | 939 >>> +++ >>> net/colo-proxy.h | 24 ++ >>> qemu-options.hx | 6 + >>> trace-events | 8 + >>> vl.c | 3 +- >>> 7 files changed, 1041 insertions(+), 1 deletion(-) >>> create mode 100644 include/qemu/jhash.h >>> create mode 100644 net/colo-proxy.c >>> create mode 100644 net/colo-proxy.h >>> >> >> >> . >> >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 12/31/2015 10:36 AM, Jason Wang wrote: On 12/22/2015 06:42 PM, Zhang Chen wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. Thanks for the work. I don't object this method but still not convinced that qemu is the best place for this. As been raised in the past discussion, it's almost impossible to cooperate with vhost backends. If we want this to be used in production environment, need to think of a solution for vhost. There's no such worry if we decouple this from qemu. Yes, I agree with you. But not everything is perfect at the beginning. If we implement proxy as kernel modules, maybe some one will ask how about the packet(e.g. vhost-user) that host kernel can not touch. As you said, colo-proxy in qemu will not support vhost scene, but it can make colo more easier to be used so that more and more pepole will join us to test it. I'm looking forward to that day. if everything goes fine, I think colo can support vhost scene in other way(such as introduce extra proxy module in kernel) in the feature. So, I think colo-proxy in qemu is a good choice for current COLO. Thanks Li You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port Have a quick glance at how secondary mode work. What it does is just forwarding packets between a nic and a socket, qemu socket backend did exact the same job. You could even use socket in primary node and let packet compare module talk to both primary and secondary node. NOTE: queue must set "all". See enum NetFilterDirection for detail. colo-proxy need queue all packets colo-proxy V2 just can compare ip packet ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand)is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. V2: rebase colo-proxy with qemu-colo-v2.2-periodic-mode fix dave's comments fix wency's comments fix zhanghailiang's comments v1: initial patch. zhangchen (10): Init colo-proxy object based on netfilter Jhash: add linux kernel jhashtable in qemu Colo-proxy: add colo-proxy framework Colo-proxy: add data structure and jhash func net/colo-proxy: Add colo interface to use proxy net/colo-proxy: add socket used by forward func net/colo-proxy: Add packet enqueue & handle func net/colo-proxy: Handle packet and connection net/colo-proxy: Compare pri pkt to sec pkt net/colo-proxy: Colo-proxy do checkpoint and clear include/qemu/jhash.h | 61 net/Makefile.objs| 1 + net/colo-proxy.c | 939 +++ net/colo-proxy.h | 24 ++ qemu-options.hx | 6 + trace-events | 8 + vl.c | 3 +- 7 files changed, 1041 insertions(+), 1 deletion(-) create mode 100644 include/qemu/jhash.h create mode 100644 net/colo-proxy.c create mode 100644 net/colo-proxy.h . -- Best regards. Li Zhijian (8555)
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 12/31/2015 10:36 AM, Jason Wang wrote: On 12/22/2015 06:42 PM, Zhang Chen wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. Thanks for the work. I don't object this method but still not convinced that qemu is the best place for this. As been raised in the past discussion, it's almost impossible to cooperate with vhost backends. If we want this to be used in production environment, need to think of a solution for vhost. There's no such worry if we decouple this from qemu. You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port Have a quick glance at how secondary mode work. What it does is just forwarding packets between a nic and a socket, qemu socket backend did exact the same job. You could even use socket in primary node and let packet compare module talk to both primary and secondary node. If we use qemu socket backend , the same netdev will used by qemu socket and qemu netfilter. this will against qemu net design. and then, when colo do failover, secondary do not have backend to use. that's the real problem. Thanks zhangchen NOTE: queue must set "all". See enum NetFilterDirection for detail. colo-proxy need queue all packets colo-proxy V2 just can compare ip packet ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand)is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. V2: rebase colo-proxy with qemu-colo-v2.2-periodic-mode fix dave's comments fix wency's comments fix zhanghailiang's comments v1: initial patch. zhangchen (10): Init colo-proxy object based on netfilter Jhash: add linux kernel jhashtable in qemu Colo-proxy: add colo-proxy framework Colo-proxy: add data structure and jhash func net/colo-proxy: Add colo interface to use proxy net/colo-proxy: add socket used by forward func net/colo-proxy: Add packet enqueue & handle func net/colo-proxy: Handle packet and connection net/colo-proxy: Compare pri pkt to sec pkt net/colo-proxy: Colo-proxy do checkpoint and clear include/qemu/jhash.h | 61 net/Makefile.objs| 1 + net/colo-proxy.c | 939 +++ net/colo-proxy.h | 24 ++ qemu-options.hx | 6 + trace-events | 8 + vl.c | 3 +- 7 files changed, 1041 insertions(+), 1 deletion(-) create mode 100644 include/qemu/jhash.h create mode 100644 net/colo-proxy.c create mode 100644 net/colo-proxy.h . -- Thanks zhangchen
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 12/22/2015 06:42 PM, Zhang Chen wrote: > From: zhangchen> > Hi,all > > This patch add an colo-proxy object, COLO-Proxy is a part of COLO, > based on qemu netfilter and it's a plugin for qemu netfilter. the function > keep Secondary VM connect normal to Primary VM and compare packets > sent by PVM to sent by SVM.if the packet difference,notify COLO do > checkpoint and send all primary packet has queued. Thanks for the work. I don't object this method but still not convinced that qemu is the best place for this. As been raised in the past discussion, it's almost impossible to cooperate with vhost backends. If we want this to be used in production environment, need to think of a solution for vhost. There's no such worry if we decouple this from qemu. > > You can also get the series from: > > https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 > > Usage: > > primary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port > > secondary: > -netdev tap,id=bn0 -device e1000,netdev=bn0 > -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port Have a quick glance at how secondary mode work. What it does is just forwarding packets between a nic and a socket, qemu socket backend did exact the same job. You could even use socket in primary node and let packet compare module talk to both primary and secondary node. > > NOTE: > queue must set "all". See enum NetFilterDirection for detail. > colo-proxy need queue all packets > colo-proxy V2 just can compare ip packet > > > ## Background > > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) > project is a high availability solution. Both Primary VM (PVM) and Secondary > VM > (SVM) run in parallel. They receive the same request from client, and generate > responses in parallel too. If the response packets from PVM and SVM are > identical, they are released immediately. Otherwise, a VM checkpoint (on > demand)is conducted. > > Paper: > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > > COLO on Xen: > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > > COLO on Qemu/KVM: > http://wiki.qemu.org/Features/COLO > > By the needs of capturing response packets from PVM and SVM and finding out > whether they are identical, we introduce a new module to qemu networking > called colo-proxy. > > V2: > rebase colo-proxy with qemu-colo-v2.2-periodic-mode > fix dave's comments > fix wency's comments > fix zhanghailiang's comments > > v1: > initial patch. > > > > zhangchen (10): > Init colo-proxy object based on netfilter > Jhash: add linux kernel jhashtable in qemu > Colo-proxy: add colo-proxy framework > Colo-proxy: add data structure and jhash func > net/colo-proxy: Add colo interface to use proxy > net/colo-proxy: add socket used by forward func > net/colo-proxy: Add packet enqueue & handle func > net/colo-proxy: Handle packet and connection > net/colo-proxy: Compare pri pkt to sec pkt > net/colo-proxy: Colo-proxy do checkpoint and clear > > include/qemu/jhash.h | 61 > net/Makefile.objs| 1 + > net/colo-proxy.c | 939 > +++ > net/colo-proxy.h | 24 ++ > qemu-options.hx | 6 + > trace-events | 8 + > vl.c | 3 +- > 7 files changed, 1041 insertions(+), 1 deletion(-) > create mode 100644 include/qemu/jhash.h > create mode 100644 net/colo-proxy.c > create mode 100644 net/colo-proxy.h >
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
Hi~ Just a small ping... No news for a week. Colo proxy is a part of COLO project, we need review and comments. Thanks zhangchen On 12/22/2015 06:42 PM, Zhang Chen wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port NOTE: queue must set "all". See enum NetFilterDirection for detail. colo-proxy need queue all packets colo-proxy V2 just can compare ip packet ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand)is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. V2: rebase colo-proxy with qemu-colo-v2.2-periodic-mode fix dave's comments fix wency's comments fix zhanghailiang's comments v1: initial patch. zhangchen (10): Init colo-proxy object based on netfilter Jhash: add linux kernel jhashtable in qemu Colo-proxy: add colo-proxy framework Colo-proxy: add data structure and jhash func net/colo-proxy: Add colo interface to use proxy net/colo-proxy: add socket used by forward func net/colo-proxy: Add packet enqueue & handle func net/colo-proxy: Handle packet and connection net/colo-proxy: Compare pri pkt to sec pkt net/colo-proxy: Colo-proxy do checkpoint and clear include/qemu/jhash.h | 61 net/Makefile.objs| 1 + net/colo-proxy.c | 939 +++ net/colo-proxy.h | 24 ++ qemu-options.hx | 6 + trace-events | 8 + vl.c | 3 +- 7 files changed, 1041 insertions(+), 1 deletion(-) create mode 100644 include/qemu/jhash.h create mode 100644 net/colo-proxy.c create mode 100644 net/colo-proxy.h -- Thanks zhangchen
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 12/29/2015 02:58 PM, Jason Wang wrote: On 12/29/2015 02:31 PM, Zhang Chen wrote: Hi~ Just a small ping... No news for a week. Colo proxy is a part of COLO project, we need review and comments. Thanks zhangchen Hi, will find sometime to review this this week. Thanks Thanks very much for your help~~ On 12/22/2015 06:42 PM, Zhang Chen wrote: From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port NOTE: queue must set "all". See enum NetFilterDirection for detail. colo-proxy need queue all packets colo-proxy V2 just can compare ip packet ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand)is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. V2: rebase colo-proxy with qemu-colo-v2.2-periodic-mode fix dave's comments fix wency's comments fix zhanghailiang's comments v1: initial patch. zhangchen (10): Init colo-proxy object based on netfilter Jhash: add linux kernel jhashtable in qemu Colo-proxy: add colo-proxy framework Colo-proxy: add data structure and jhash func net/colo-proxy: Add colo interface to use proxy net/colo-proxy: add socket used by forward func net/colo-proxy: Add packet enqueue & handle func net/colo-proxy: Handle packet and connection net/colo-proxy: Compare pri pkt to sec pkt net/colo-proxy: Colo-proxy do checkpoint and clear include/qemu/jhash.h | 61 net/Makefile.objs| 1 + net/colo-proxy.c | 939 +++ net/colo-proxy.h | 24 ++ qemu-options.hx | 6 + trace-events | 8 + vl.c | 3 +- 7 files changed, 1041 insertions(+), 1 deletion(-) create mode 100644 include/qemu/jhash.h create mode 100644 net/colo-proxy.c create mode 100644 net/colo-proxy.h . -- Thanks zhangchen
Re: [Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
On 12/29/2015 02:31 PM, Zhang Chen wrote: > Hi~ > Just a small ping... > No news for a week. > Colo proxy is a part of COLO project, we need review and comments. > > > Thanks > zhangchen Hi, will find sometime to review this this week. Thanks > > > On 12/22/2015 06:42 PM, Zhang Chen wrote: >> From: zhangchen>> >> Hi,all >> >> This patch add an colo-proxy object, COLO-Proxy is a part of COLO, >> based on qemu netfilter and it's a plugin for qemu netfilter. the >> function >> keep Secondary VM connect normal to Primary VM and compare packets >> sent by PVM to sent by SVM.if the packet difference,notify COLO do >> checkpoint and send all primary packet has queued. >> >> You can also get the series from: >> >> https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 >> >> >> Usage: >> >> primary: >> -netdev tap,id=bn0 -device e1000,netdev=bn0 >> -object >> colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port >> >> secondary: >> -netdev tap,id=bn0 -device e1000,netdev=bn0 >> -object >> colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port >> >> NOTE: >> queue must set "all". See enum NetFilterDirection for detail. >> colo-proxy need queue all packets >> colo-proxy V2 just can compare ip packet >> >> >> ## Background >> >> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >> Service) >> project is a high availability solution. Both Primary VM (PVM) and >> Secondary VM >> (SVM) run in parallel. They receive the same request from client, and >> generate >> responses in parallel too. If the response packets from PVM and SVM are >> identical, they are released immediately. Otherwise, a VM checkpoint (on >> demand)is conducted. >> >> Paper: >> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >> >> COLO on Xen: >> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >> >> COLO on Qemu/KVM: >> http://wiki.qemu.org/Features/COLO >> >> By the needs of capturing response packets from PVM and SVM and >> finding out >> whether they are identical, we introduce a new module to qemu networking >> called colo-proxy. >> >> V2: >>rebase colo-proxy with qemu-colo-v2.2-periodic-mode >>fix dave's comments >>fix wency's comments >>fix zhanghailiang's comments >> >> v1: >>initial patch. >> >> >> >> zhangchen (10): >>Init colo-proxy object based on netfilter >>Jhash: add linux kernel jhashtable in qemu >>Colo-proxy: add colo-proxy framework >>Colo-proxy: add data structure and jhash func >>net/colo-proxy: Add colo interface to use proxy >>net/colo-proxy: add socket used by forward func >>net/colo-proxy: Add packet enqueue & handle func >>net/colo-proxy: Handle packet and connection >>net/colo-proxy: Compare pri pkt to sec pkt >>net/colo-proxy: Colo-proxy do checkpoint and clear >> >> include/qemu/jhash.h | 61 >> net/Makefile.objs| 1 + >> net/colo-proxy.c | 939 >> +++ >> net/colo-proxy.h | 24 ++ >> qemu-options.hx | 6 + >> trace-events | 8 + >> vl.c | 3 +- >> 7 files changed, 1041 insertions(+), 1 deletion(-) >> create mode 100644 include/qemu/jhash.h >> create mode 100644 net/colo-proxy.c >> create mode 100644 net/colo-proxy.h >> >
[Qemu-devel] [RFC PATCH v2 00/10] Add colo-proxy based on netfilter
From: zhangchenHi,all This patch add an colo-proxy object, COLO-Proxy is a part of COLO, based on qemu netfilter and it's a plugin for qemu netfilter. the function keep Secondary VM connect normal to Primary VM and compare packets sent by PVM to sent by SVM.if the packet difference,notify COLO do checkpoint and send all primary packet has queued. You can also get the series from: https://github.com/zhangckid/qemu/tree/colo-v2.2-periodic-mode-with-colo-proxyV2 Usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=primary,addr=host:port secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,mode=secondary,addr=host:port NOTE: queue must set "all". See enum NetFilterDirection for detail. colo-proxy need queue all packets colo-proxy V2 just can compare ip packet ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand)is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. V2: rebase colo-proxy with qemu-colo-v2.2-periodic-mode fix dave's comments fix wency's comments fix zhanghailiang's comments v1: initial patch. zhangchen (10): Init colo-proxy object based on netfilter Jhash: add linux kernel jhashtable in qemu Colo-proxy: add colo-proxy framework Colo-proxy: add data structure and jhash func net/colo-proxy: Add colo interface to use proxy net/colo-proxy: add socket used by forward func net/colo-proxy: Add packet enqueue & handle func net/colo-proxy: Handle packet and connection net/colo-proxy: Compare pri pkt to sec pkt net/colo-proxy: Colo-proxy do checkpoint and clear include/qemu/jhash.h | 61 net/Makefile.objs| 1 + net/colo-proxy.c | 939 +++ net/colo-proxy.h | 24 ++ qemu-options.hx | 6 + trace-events | 8 + vl.c | 3 +- 7 files changed, 1041 insertions(+), 1 deletion(-) create mode 100644 include/qemu/jhash.h create mode 100644 net/colo-proxy.c create mode 100644 net/colo-proxy.h -- 1.9.1