[Qemu-devel] [RFC] Migration convergence - a suggestion
Well the issue is not new, anyhow, following a conversation with Orit ... Since we want the migration to finish, I believe that the "migration speed" parameter alone cannot do the job. I suggest using two distinct parameters: 1. Migration speed - will be used to limit the network resources utilization 2. aggressionLevel - A number between 0.0 and 1.0, where low values imply minimal interruption to the guest, and 1.0 mean that the guest will be completely stalled. In any case the migration will have to do its work and finish given any actual migration-speed, so even low aggressionLevel values will sometimes imply that the guest will be throttled substantially. The algorithm: The aggressionLevel should determine the targetGuest%CPU (how much CPU time we want to allocate to the guest) With aggressionLevel = 1.0, the guest gets no CPU-resources (stalled). With aggressionLevel = 0.0, the guest gets minGuest%CPU, such that migrationRate == dirtyPagesRate. This minGuest%CPU is continuously updated based on the running average of the recent samples (more below). Note that the targetGuest%CPU allocation is continuously updated due to changes guest behavior, network congestion, and alike. Some more details - minGuest%CPU (i.e., for dirtyPagesRate == migrationRate) is easy to calculate as a running average of (migrationRate / dirtyPagesRate * guest%CPU) - There are several methods to calculate the running average, my favorite is IIR, where, roughly speaking, newVal = 0.99 * oldVal + 0.01 * newSample - I would use two measures to ensure that there are more migrated pages than "dirty" pages. 1. The running average (based on recent samples) of the migrated pages is larger than that of the new dirty pages 2. The total number of migrated pages so far is larger than the total number of new dirty pages. And yes, many details are still missing. Ronen.
Re: [Qemu-devel] [RFC] Migration convergence - a suggestion
On 12/20/2011 01:06 AM, Ronen Hod wrote: Well the issue is not new, anyhow, following a conversation with Orit ... Since we want the migration to finish, I believe that the "migration speed" parameter alone cannot do the job. I suggest using two distinct parameters: 1. Migration speed - will be used to limit the network resources utilization 2. aggressionLevel - A number between 0.0 and 1.0, where low values imply minimal interruption to the guest, and 1.0 mean that the guest will be completely stalled. In any case the migration will have to do its work and finish given any actual migration-speed, so even low aggressionLevel values will sometimes imply that the guest will be throttled substantially. The algorithm: The aggressionLevel should determine the targetGuest%CPU (how much CPU time we want to allocate to the guest) QEMU has no way to limit the guest CPU time. Regards, Anthony Liguori
Re: [Qemu-devel] [RFC] Migration convergence - a suggestion
On 12/20/2011 03:39 PM, Anthony Liguori wrote: On 12/20/2011 01:06 AM, Ronen Hod wrote: Well the issue is not new, anyhow, following a conversation with Orit ... Since we want the migration to finish, I believe that the "migration speed" parameter alone cannot do the job. I suggest using two distinct parameters: 1. Migration speed - will be used to limit the network resources utilization 2. aggressionLevel - A number between 0.0 and 1.0, where low values imply minimal interruption to the guest, and 1.0 mean that the guest will be completely stalled. In any case the migration will have to do its work and finish given any actual migration-speed, so even low aggressionLevel values will sometimes imply that the guest will be throttled substantially. The algorithm: The aggressionLevel should determine the targetGuest%CPU (how much CPU time we want to allocate to the guest) QEMU has no way to limit the guest CPU time. Wouldn't any "yield" (sleep / whatever) limit the guest's CPU time, be it in qemu or in KVM. My intention is to suggest an algorithm that is based on guest throttling. Looking at the relevant BZs, I do not see how we can avoid it. I certainly have no claims regarding the architecture. Avi and mst, believe that it is better to continuously control the guest's CPU from the outside (libvirt) using cgroups. Although less responsive to changes, it should still work. In the meantime, I also discovered that everybody has a different point of view regarding the requirements. Regardless, I believe that the same basic mechanics (once decided), can do the work Some relevant configuration "requirements" are: 1. Max bandwidth 2. Min CPU per guest 3. Max guest stall time 4. Max migration time These requirements will often conflict, and may imply changes in behavior over time. I would also suggest that the management GUI will let the user select the aggression-level (or whatever), and display the implication on all the other parameters (total-time, %CPU) based on the current behavior of the guest and network. Regards, Ronen Regards, Anthony Liguori
Re: [Qemu-devel] [RFC] Migration convergence - a suggestion
On 12/21/2011 03:47 AM, Ronen Hod wrote: On 12/20/2011 03:39 PM, Anthony Liguori wrote: On 12/20/2011 01:06 AM, Ronen Hod wrote: Well the issue is not new, anyhow, following a conversation with Orit ... Since we want the migration to finish, I believe that the "migration speed" parameter alone cannot do the job. I suggest using two distinct parameters: 1. Migration speed - will be used to limit the network resources utilization 2. aggressionLevel - A number between 0.0 and 1.0, where low values imply minimal interruption to the guest, and 1.0 mean that the guest will be completely stalled. In any case the migration will have to do its work and finish given any actual migration-speed, so even low aggressionLevel values will sometimes imply that the guest will be throttled substantially. The algorithm: The aggressionLevel should determine the targetGuest%CPU (how much CPU time we want to allocate to the guest) QEMU has no way to limit the guest CPU time. Wouldn't any "yield" (sleep / whatever) limit the guest's CPU time, be it in qemu or in KVM. Practically speaking, not really. My intention is to suggest an algorithm that is based on guest throttling. Looking at the relevant BZs, I do not see how we can avoid it. I certainly have no claims regarding the architecture. Avi and mst, believe that it is better to continuously control the guest's CPU from the outside (libvirt) using cgroups. Although less responsive to changes, it should still work. Yes, that's really the only way to do it I think. You'll need to use the CFS bandwidth controller. In the meantime, I also discovered that everybody has a different point of view regarding the requirements. Regardless, I believe that the same basic mechanics (once decided), can do the work Some relevant configuration "requirements" are: 1. Max bandwidth 2. Min CPU per guest 3. Max guest stall time 4. Max migration time These requirements will often conflict, and may imply changes in behavior over time. I would also suggest that the management GUI will let the user select the aggression-level (or whatever), and display the implication on all the other parameters (total-time, %CPU) based on the current behavior of the guest and network. It really depends on your management GUI. If you have a task oriented GUI, you are unlikely to have a "live migration" feature. Rather, you'll have a mechanism to enable load balancing (which may use live migration) or you'll have an "evacuate node" feature to allow system maintenance. Your downtime policy really depends on the feature. The best thing for QEMU to do is make sure we expose all of the necessary hooks to allow a wide variety of policies to be implemented. Regards, Anthony Liguori Regards, Ronen Regards, Anthony Liguori
