On 11/14/2012 04:23 AM, Isaku Yamahata wrote: > On Tue, Nov 13, 2012 at 05:46:13PM +0000, Hudzia, Benoit wrote: >> Hi, >> >> One concept we have been playing around in the context of and hybrid and >> post copy and might make sense if you are orienting your effort toward RDMA >> / Post copy is to move most of the logic in the destination side. >> >> This is one thing you might want to consider as it can solve some of the >> issue you currently have and allow you to maintain almost a single API / >> Protocol once integrating with post copy approach. >> >> The idea is to drive the migration from the destination side. I.e. The page >> are pulled from the destination and not pushed from the source side. >> >> Ex: current pre-copy : >> >> *extract dirty bitmap ( dirty bitmap extraction can be scheduled or >> triggered by destination) >> * send it to the destination side >> * have the destination iterating over the bitmap ( can do page >> prioritization here) > > IIRC last year, you mentioned page prioritization, but didn't this year. > Is it still supported? > Where is it implemented? in qemu or kernel? I did a prototype and couldn't find workload that benefits from it so it was moved down in priority. Maybe I will return to it in the future.
Regards, Orit > > >> * depending of protocol : >> _ with standard socket ( or RDS) : >> . Destination : request page(s)<- can be batched >> . source receive request send back the page >> . destination process >> _ with RDMA : >> . Destination Read Page from source to local page ( the >> page have been mapped to RDMA at the bitmap extraction) ( RDMA support >> scatter gather) > > Although I'm not familiar with RDMA, RDMA requires the exchange of > DMA-address between > sender and receiver in advance and pinning down pages. > It it correct? > > >> _ with post copy >> . pretty much the same but the dirty bitmap reset is >> done in kernel during the post copy operation ( provide a better dirty bit >> tracking granularity) >> >> >> Disadvantage: >> * add a round trip that can be compensate with batch operation ( only >> with standard socket) >> >> Advantage : >> * most of the heavy lifting is done at the destination side leaving the >> source to respond to request in an event based format >> * resolve a lot of issue you have with your threading form the sender >> side ( accounting etc.. ) >> * extremely friendly to optimised solution >> * if the bitmap generation is expensive we can overlap their generation >> creating a semi continuous delivery of them guaranteeing an uninterrupted >> and optimised flow. => we decouple the bitmap generation from the send/ >> receive operation. >> >> >> >> Anyway , I will notify you as soon as I have the patch / library available >> for RDMA / postcopy. >> >> Note On the fault tolerance part: this require a lot more heavy code >> optimisation and poking around to guarantee efficient checkpointing. Most of >> the solution we tested so far ( Remus and an old version of kemari) scale >> poorly . Again, an RDMA / post copy solution is kind of necessary when you >> talk about check pointing enterprise class applications. > > IIRC Kemari guys evaluated IB case. I'm not sure that it was with RDMA or > IPoIB. > > thanks, >> >> >> Regards >> Benoit >> >> >> >> >> >>> -----Original Message----- >>> From: Juan Quintela [mailto:quint...@redhat.com] >>> Sent: 13 November 2012 16:19 >>> To: qemu-devel qemu-devel; Orit Wasserman; chegu_vi...@hp.com; >>> Hudzia, Benoit; Isaku Yamahata; Michael Roth >>> Subject: Migration ToDo list >>> >>> >>> Hi >>> >>> If you have anything else to put, please add. >>> >>> Migration Thread >>> * Plan is integrate it as one of first thing in December (me) >>> * Remove copies with buffered file (me) >>> >>> Bitmap Optimization >>> * Finish moving to individual bitmaps for migration/vga/code >>> * Make sure we don't copy things around >>> * Shared memory bitmap with kvm? >>> * Move to 2MB pages bitmap and then fine grain? >>> >>> QIDL >>> * Review the patches (me) >>> >>> PostCopy >>> * Review patches? >>> * See what we can already integrate? >>> I remember for last year that we could integrate the 1st third or so >>> >>> RDMA >>> * Send RDMA/tcp/.... library they already have (Benoit) >>> * This is required for postcopy >>> * This can be used for precopy >>> >>> General >>> * Change protocol to: >>> a) being always 16byte aligned (paolo said that is faster) >>> b) do scatter/gather of the pages? >>> >>> Fault Tolerance >>> * That is built on top of migration code, but I have nothing to add. >>> >>> Any more ideas? >>> >>> Later, Juan. >> >