TadiosAbebe commented on issue #11141: URL: https://github.com/apache/cloudstack/issues/11141#issuecomment-3569210929
@bradh352 , opening the console is also my first step when checking whether a compute host is experiencing issues. Over the past few weeks, I ran several tests across my five compute hosts (all using Ceph) to better understand the behavior. ### Initial Tests These tests were performed a few days after restarting both cloudstack-agent and libvirtd. | host | vm_create(in sec) | vm_stop(in sec) | vm_start(in sec) | vm_console(in sec) | |---|---|---|---|---| | host1 | 86 | 30 | 60 | 5| | host2 | 37 | 18 | 28 | 4 | | host3 | 54 | 9 | 33 | 1 | | host4 | 49 | 3 | 19 | 0 | | host5 | 12 | 3 | 19 | 0 | ### After Restarting libvirtd I restarted libvirtd on all hosts and repeated the same tests. | host | vm_create(in sec) | vm_stop(in sec) | vm_start(in sec) | vm_console(in sec) | |---|---|---|---|---| | host1 | 21 | 3 | 19 | 0| | host2 | 22 | 3 | 19 | 0 | | host3 | 25 | 3 | 19 | 0 | | host4 | 21 | 7 | 19 | 0 | | host5 | 22 | 3 | 18 | 0 | ### Same Tests Again (After ~2 Days) | host | vm_create(in sec) | vm_stop(in sec) | vm_start(in sec) | vm_console(in sec) | |---|---|---|---|---| | host1 | 31 | 9 | 21 | 1| | host2 | 28 | 3 | 18 | 0 | | host3 | 28 | 6 | 23 | 1 | | host4 | 30 | 3 | 19 | 0 | | host5 | 26 | 3 | 18 | 0 | These results may not be perfectly accurate due to different factors, but the performance differences are still notable especially the improvement immediately after restarting libvirtd. My main concern with scheduling an automated restart of cloudstack-agent was, if it restarts in the middle of an operation, it may leave behind orphaned resources. I believe this is how I ended up with volumes stuck in Destroy state and a virtual router stuck in Expunging, which I’m still unable to fully remove. It would be helpful to know whether restarting libvirtd is truly the reliable fix for you too. Even if that’s the case, I still don’t understand why this degradation happens over time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
