On 05/29/2015 12:24 AM, Dr. David Alan Gilbert wrote: > * zhanghailiang (zhang.zhanghaili...@huawei.com) wrote: >> This is the 5th version of COLO, here is only COLO frame part, include: VM >> checkpoint, >> failover, proxy API, block replication API, not include block replication. >> The block part has been sent by wencongyang: >> "[Qemu-devel] [PATCH COLO-Block v5 00/15] Block replication for continuous >> checkpoints" >> >> we have finished some new features and optimization on COLO (As a >> development branch in github), >> but for easy of review, it is better to keep it simple now, so we will not >> add too much new >> codes into this frame patch set before it been totally reviewed. >> >> You can get the latest integrated qemu colo patches from github (Include >> Block part): >> https://github.com/coloft/qemu/commits/colo-v1.2-basic >> https://github.com/coloft/qemu/commits/colo-v1.2-developing (more features) >> >> Please NOTE the difference between these two branch. >> colo-v1.2-basic is exactly same with this patch series, which has basic >> features of COLO. >> Compared with colo-v1.2-basic, colo-v1.2-developing has some optimization in >> the >> process of checkpoint, including: >> 1) separate ram and device save/load process to reduce size of extra >> memory >> used during checkpoint >> 2) live migrate part of dirty pages to slave during sleep time. >> Besides, we add some statistic info in colo-v1.2-developing, which you can >> get these stat >> info by using command 'info migrate'. > > > Hi, > I have that running now. > > Some notes: > 1) The colo-proxy is working OK until qemu quits, and then it gets an RCU > problem; see below > 2) I've attached some minor tweaks that were needed to build with the 4.1rc > kernel I'm using; > they're very minor changes and I don't think related to (1). > 3) I've also included some minor fixups I needed to get the -developing > world > to build; my compiler is fussy about unused variables etc - but I think > the code > in ram_save_complete in your -developing patch is wrong because there > are two > 'pages' variables and the one in the inner loop is the only one changed. > 4) I've started trying simple benchmarks and tests now: > a) With a simple web server most requests have very little overhead, the > comparison > matches most of the time; I do get quite large spikes (0.04s->1.05s) > which I guess > corresponds to when a checkpoint happens, but I'm not sure why the > spike is so big, > since the downtime isn't that big. > b) I tried something with more dynamic pages - the front page of a simple > bugzilla > install; it failed the comparison every time; it took me a while to > figure out > why, but it generates a unique token in it's javascript each time (for > a password reset > link), and I guess the randomness used by that doesn't match on the > two hosts. > It surprised me, because I didn't expect this page to have much > randomness > in. > > 4a is really nice - it shows the benefit of COLO over the simple > checkpointing; > checkpoints happen very rarely. > > The colo-proxy rcu problem I hit shows as rcu-stalls in both primary and > secondary > after the qemu quits; the backtrace of the qemu stack is:
How to reproduce it? Use monitor command quit to quit qemu? Or kill the qemu? > > [<ffffffff810d8c0c>] wait_rcu_gp+0x5c/0x80 > [<ffffffff810ddb05>] synchronize_rcu+0x45/0xd0 > [<ffffffffa0a251e5>] colo_node_release+0x35/0x50 [nfnetlink_colo] > [<ffffffffa0a25795>] colonl_close_event+0xe5/0x160 [nfnetlink_colo] > [<ffffffff81090c96>] notifier_call_chain+0x66/0x90 > [<ffffffff8109154c>] atomic_notifier_call_chain+0x6c/0x110 > [<ffffffff815eee07>] netlink_release+0x5b7/0x7f0 > [<ffffffff815878bf>] sock_release+0x1f/0x90 > [<ffffffff81587942>] sock_close+0x12/0x20 > [<ffffffff812193c3>] __fput+0xd3/0x210 > [<ffffffff8121954e>] ____fput+0xe/0x10 > [<ffffffff8108d9f7>] task_work_run+0xb7/0xf0 > [<ffffffff81002d4d>] do_notify_resume+0x8d/0xa0 > [<ffffffff81722b66>] int_signal+0x12/0x17 > [<ffffffffffffffff>] 0xffffffffffffffff Thanks for your test. The backtrace is very useful, and we will fix it soon. > > that's with both the 423a8e268acbe3e644a16c15bc79603cfe9eb084 from yesterday > and > older e58e5152b74945871b00a88164901c0d46e6365e tags on colo-proxy. > I'm not sure of the right fix; perhaps it might be possible to replace the > synchronize_rcu in colo_node_release by a call_rcu that does the kfree later? I agree with it. Thanks Wen Congyang > > Thanks, > > Dave > >>
