Re: [Xen-devel] [PATCH 0/4] fix freemem loop
On Fri, 2015-03-06 at 09:52 +, Stefano Stabellini wrote: On Fri, 6 Mar 2015, Ian Campbell wrote: On Thu, 2015-03-05 at 21:08 -0700, Mike Latimer wrote: On Thursday, March 05, 2015 05:49:35 PM Ian Campbell wrote: On Tue, 2015-03-03 at 11:08 +, Stefano Stabellini wrote: Hi all, this patch series fixes the freemem loop on machines with very large amount of memory, where the current wait time is not enough. In order to be able to handle arbitrarly large amount of ram, we implement in libxl_wait_for_memory_target a policy of waiting until dom0 is making progress. What is the impact of the libxl API change made here on other callers, in particular libvirt? I should have CCd Jim when I asked the question. Now done. This change will have one interesting effect on libvirt. Currently, libxlDomainFreeMem loops 3 times, then returns 0 (if no errors are encountered). This means that domain creation starts before dom0 finishes ballooning (unlike xl's previous behavior, which would fail). With this change, domain creation through virsh will wait (in libxl_wait_for_memory_target) until dom0 finishes ballooning. This should result in an increase in the speed of the domain starting, as there will not be memory contention between the two processes. The libvirt side calls libxl_wait_for_memory_target with a 1 second timeout - which doesn't leave a huge amount of room for slow memory allocation. This timeout, as well as the logic in general, should be changed to match the new xl behavior (IMO). I expect this to really only matter when dealing with large domains. For libvirt we also need to consider what happens when a libvirt which is modified along these lines uses an older libxl (I believe back to 4.2 is supported). In the libvirt case even if libvirt calls libxl_wait_for_memory_target with just 1 second timeout, it is likely going to end up waiting longer than that due to the change in behaviour of the function. What I am saying is that even if we don't modify libvirt, we are going to get a more reliable and more stable behaviour than what we have now. That said, I agree that it would be good to improve the freemem loop in libvirt in a similar way to what we are doing with xl. This is missing my point. We need to consider the case of a modified libvirt with the old, broken, libxl too. You've only considered an unmodified libvirt running with a fixed libxl. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 0/4] fix freemem loop
On Fri, 6 Mar 2015, Ian Campbell wrote: On Fri, 2015-03-06 at 09:52 +, Stefano Stabellini wrote: On Fri, 6 Mar 2015, Ian Campbell wrote: On Thu, 2015-03-05 at 21:08 -0700, Mike Latimer wrote: On Thursday, March 05, 2015 05:49:35 PM Ian Campbell wrote: On Tue, 2015-03-03 at 11:08 +, Stefano Stabellini wrote: Hi all, this patch series fixes the freemem loop on machines with very large amount of memory, where the current wait time is not enough. In order to be able to handle arbitrarly large amount of ram, we implement in libxl_wait_for_memory_target a policy of waiting until dom0 is making progress. What is the impact of the libxl API change made here on other callers, in particular libvirt? I should have CCd Jim when I asked the question. Now done. This change will have one interesting effect on libvirt. Currently, libxlDomainFreeMem loops 3 times, then returns 0 (if no errors are encountered). This means that domain creation starts before dom0 finishes ballooning (unlike xl's previous behavior, which would fail). With this change, domain creation through virsh will wait (in libxl_wait_for_memory_target) until dom0 finishes ballooning. This should result in an increase in the speed of the domain starting, as there will not be memory contention between the two processes. The libvirt side calls libxl_wait_for_memory_target with a 1 second timeout - which doesn't leave a huge amount of room for slow memory allocation. This timeout, as well as the logic in general, should be changed to match the new xl behavior (IMO). I expect this to really only matter when dealing with large domains. For libvirt we also need to consider what happens when a libvirt which is modified along these lines uses an older libxl (I believe back to 4.2 is supported). In the libvirt case even if libvirt calls libxl_wait_for_memory_target with just 1 second timeout, it is likely going to end up waiting longer than that due to the change in behaviour of the function. What I am saying is that even if we don't modify libvirt, we are going to get a more reliable and more stable behaviour than what we have now. That said, I agree that it would be good to improve the freemem loop in libvirt in a similar way to what we are doing with xl. This is missing my point. We need to consider the case of a modified libvirt with the old, broken, libxl too. For that case, we might want to make sure that the new timeout passed to libxl_wait_for_memory_target is not lower than the overall libvirt timeout today (libxl_wait_for_free_memory+libxl_wait_for_memory_target). ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 0/4] fix freemem loop
On Fri, 6 Mar 2015, Ian Campbell wrote: On Thu, 2015-03-05 at 21:08 -0700, Mike Latimer wrote: On Thursday, March 05, 2015 05:49:35 PM Ian Campbell wrote: On Tue, 2015-03-03 at 11:08 +, Stefano Stabellini wrote: Hi all, this patch series fixes the freemem loop on machines with very large amount of memory, where the current wait time is not enough. In order to be able to handle arbitrarly large amount of ram, we implement in libxl_wait_for_memory_target a policy of waiting until dom0 is making progress. What is the impact of the libxl API change made here on other callers, in particular libvirt? I should have CCd Jim when I asked the question. Now done. This change will have one interesting effect on libvirt. Currently, libxlDomainFreeMem loops 3 times, then returns 0 (if no errors are encountered). This means that domain creation starts before dom0 finishes ballooning (unlike xl's previous behavior, which would fail). With this change, domain creation through virsh will wait (in libxl_wait_for_memory_target) until dom0 finishes ballooning. This should result in an increase in the speed of the domain starting, as there will not be memory contention between the two processes. The libvirt side calls libxl_wait_for_memory_target with a 1 second timeout - which doesn't leave a huge amount of room for slow memory allocation. This timeout, as well as the logic in general, should be changed to match the new xl behavior (IMO). I expect this to really only matter when dealing with large domains. For libvirt we also need to consider what happens when a libvirt which is modified along these lines uses an older libxl (I believe back to 4.2 is supported). In the libvirt case even if libvirt calls libxl_wait_for_memory_target with just 1 second timeout, it is likely going to end up waiting longer than that due to the change in behaviour of the function. What I am saying is that even if we don't modify libvirt, we are going to get a more reliable and more stable behaviour than what we have now. That said, I agree that it would be good to improve the freemem loop in libvirt in a similar way to what we are doing with xl. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 0/4] fix freemem loop
On Thu, 2015-03-05 at 21:08 -0700, Mike Latimer wrote: On Thursday, March 05, 2015 05:49:35 PM Ian Campbell wrote: On Tue, 2015-03-03 at 11:08 +, Stefano Stabellini wrote: Hi all, this patch series fixes the freemem loop on machines with very large amount of memory, where the current wait time is not enough. In order to be able to handle arbitrarly large amount of ram, we implement in libxl_wait_for_memory_target a policy of waiting until dom0 is making progress. What is the impact of the libxl API change made here on other callers, in particular libvirt? I should have CCd Jim when I asked the question. Now done. This change will have one interesting effect on libvirt. Currently, libxlDomainFreeMem loops 3 times, then returns 0 (if no errors are encountered). This means that domain creation starts before dom0 finishes ballooning (unlike xl's previous behavior, which would fail). With this change, domain creation through virsh will wait (in libxl_wait_for_memory_target) until dom0 finishes ballooning. This should result in an increase in the speed of the domain starting, as there will not be memory contention between the two processes. The libvirt side calls libxl_wait_for_memory_target with a 1 second timeout - which doesn't leave a huge amount of room for slow memory allocation. This timeout, as well as the logic in general, should be changed to match the new xl behavior (IMO). I expect this to really only matter when dealing with large domains. For libvirt we also need to consider what happens when a libvirt which is modified along these lines uses an older libxl (I believe back to 4.2 is supported). Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 0/4] fix freemem loop
On Fri, 2015-03-06 at 10:22 +, Stefano Stabellini wrote: On Fri, 6 Mar 2015, Ian Campbell wrote: On Fri, 2015-03-06 at 09:52 +, Stefano Stabellini wrote: On Fri, 6 Mar 2015, Ian Campbell wrote: On Thu, 2015-03-05 at 21:08 -0700, Mike Latimer wrote: On Thursday, March 05, 2015 05:49:35 PM Ian Campbell wrote: On Tue, 2015-03-03 at 11:08 +, Stefano Stabellini wrote: Hi all, this patch series fixes the freemem loop on machines with very large amount of memory, where the current wait time is not enough. In order to be able to handle arbitrarly large amount of ram, we implement in libxl_wait_for_memory_target a policy of waiting until dom0 is making progress. What is the impact of the libxl API change made here on other callers, in particular libvirt? I should have CCd Jim when I asked the question. Now done. This change will have one interesting effect on libvirt. Currently, libxlDomainFreeMem loops 3 times, then returns 0 (if no errors are encountered). This means that domain creation starts before dom0 finishes ballooning (unlike xl's previous behavior, which would fail). With this change, domain creation through virsh will wait (in libxl_wait_for_memory_target) until dom0 finishes ballooning. This should result in an increase in the speed of the domain starting, as there will not be memory contention between the two processes. The libvirt side calls libxl_wait_for_memory_target with a 1 second timeout - which doesn't leave a huge amount of room for slow memory allocation. This timeout, as well as the logic in general, should be changed to match the new xl behavior (IMO). I expect this to really only matter when dealing with large domains. For libvirt we also need to consider what happens when a libvirt which is modified along these lines uses an older libxl (I believe back to 4.2 is supported). In the libvirt case even if libvirt calls libxl_wait_for_memory_target with just 1 second timeout, it is likely going to end up waiting longer than that due to the change in behaviour of the function. What I am saying is that even if we don't modify libvirt, we are going to get a more reliable and more stable behaviour than what we have now. That said, I agree that it would be good to improve the freemem loop in libvirt in a similar way to what we are doing with xl. This is missing my point. We need to consider the case of a modified libvirt with the old, broken, libxl too. For that case, we might want to make sure that the new timeout passed to libxl_wait_for_memory_target is not lower than the overall libvirt timeout today (libxl_wait_for_free_memory+libxl_wait_for_memory_target). OK, sounds like we may not need a #define then. But perhaps you could update the comment above the prototypes of these functions in libxl.h which explains how to use the interface such that it works well with new libxl and at least tolerably with old libxl and (if it is different) how to use it so it works super-well with new libxl if you don't care about old libxl. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 0/4] fix freemem loop
Stefano Stabellini wrote: On Fri, 6 Mar 2015, Ian Campbell wrote: This is missing my point. We need to consider the case of a modified libvirt with the old, broken, libxl too. For that case, we might want to make sure that the new timeout passed to libxl_wait_for_memory_target is not lower than the overall libvirt timeout today (libxl_wait_for_free_memory+libxl_wait_for_memory_target). Currently in libvirt, libxl_wait_for_free_memory+libxl_wait_for_memory_target = 11sec. But IMO it will be fine to replace that with 'libxl_wait_for_memory_target(ctx, 0, 10)', as you did for xl in 3/4. A modified libvirt with unmodified libxl would behave the same as a modified xl with unmodified libxl. Regards, Jim ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 0/4] fix freemem loop
On Tue, 2015-03-03 at 11:08 +, Stefano Stabellini wrote: Hi all, this patch series fixes the freemem loop on machines with very large amount of memory, where the current wait time is not enough. In order to be able to handle arbitrarly large amount of ram, we implement in libxl_wait_for_memory_target a policy of waiting until dom0 is making progress. What is the impact of the libxl API change made here on other callers, in particular libvirt? If it is expected that existing callers should continue to work as before (perhaps with a redundant call etc) then please state this in the relevant commit message(s). The patch series also reverts libxl: Wait for ballooning if free memory is increasing, that is not actually implemented correctly. Stefano Stabellini (4): Revert libxl: Wait for ballooning if free memory is increasing libxl_wait_for_memory_target: wait as long as dom0 is making progress freemem: remove call to libxl_wait_for_free_memory libxl_wait_for_memory_target: wait for 2 sec at a time tools/libxl/libxl.c | 31 +++ tools/libxl/xl_cmdimpl.c | 29 ++--- 2 files changed, 29 insertions(+), 31 deletions(-) Cheers, Stefano ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 0/4] fix freemem loop
On Thursday, March 05, 2015 05:49:35 PM Ian Campbell wrote: On Tue, 2015-03-03 at 11:08 +, Stefano Stabellini wrote: Hi all, this patch series fixes the freemem loop on machines with very large amount of memory, where the current wait time is not enough. In order to be able to handle arbitrarly large amount of ram, we implement in libxl_wait_for_memory_target a policy of waiting until dom0 is making progress. What is the impact of the libxl API change made here on other callers, in particular libvirt? This change will have one interesting effect on libvirt. Currently, libxlDomainFreeMem loops 3 times, then returns 0 (if no errors are encountered). This means that domain creation starts before dom0 finishes ballooning (unlike xl's previous behavior, which would fail). With this change, domain creation through virsh will wait (in libxl_wait_for_memory_target) until dom0 finishes ballooning. This should result in an increase in the speed of the domain starting, as there will not be memory contention between the two processes. The libvirt side calls libxl_wait_for_memory_target with a 1 second timeout - which doesn't leave a huge amount of room for slow memory allocation. This timeout, as well as the logic in general, should be changed to match the new xl behavior (IMO). I expect this to really only matter when dealing with large domains. -Mike ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 0/4] fix freemem loop
On Tuesday, March 03, 2015 02:54:50 PM Mike Latimer wrote: Thanks for all the help and patience as we've worked through this. Ack to the whole series: Acked-by: Mike Latimer mlati...@suse.com I guess the more correct response is: Reviewed-by: Mike Latimer mlati...@suse.com Tested-by: Mike Latimer mlati...@suse.com -Mike ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH 0/4] fix freemem loop
Hi all, this patch series fixes the freemem loop on machines with very large amount of memory, where the current wait time is not enough. In order to be able to handle arbitrarly large amount of ram, we implement in libxl_wait_for_memory_target a policy of waiting until dom0 is making progress. The patch series also reverts libxl: Wait for ballooning if free memory is increasing, that is not actually implemented correctly. Stefano Stabellini (4): Revert libxl: Wait for ballooning if free memory is increasing libxl_wait_for_memory_target: wait as long as dom0 is making progress freemem: remove call to libxl_wait_for_free_memory libxl_wait_for_memory_target: wait for 2 sec at a time tools/libxl/libxl.c | 31 +++ tools/libxl/xl_cmdimpl.c | 29 ++--- 2 files changed, 29 insertions(+), 31 deletions(-) Cheers, Stefano ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 0/4] fix freemem loop
On Tuesday, March 03, 2015 11:08:38 AM Stefano Stabellini wrote: Hi all, this patch series fixes the freemem loop on machines with very large amount of memory, where the current wait time is not enough. In order to be able to handle arbitrarly large amount of ram, we implement in libxl_wait_for_memory_target a policy of waiting until dom0 is making progress. The patch series also reverts libxl: Wait for ballooning if free memory is increasing, that is not actually implemented correctly. Stefano Stabellini (4): Revert libxl: Wait for ballooning if free memory is increasing libxl_wait_for_memory_target: wait as long as dom0 is making progress freemem: remove call to libxl_wait_for_free_memory libxl_wait_for_memory_target: wait for 2 sec at a time tools/libxl/libxl.c | 31 +++ tools/libxl/xl_cmdimpl.c | 29 ++--- 2 files changed, 29 insertions(+), 31 deletions(-) I just tested this whole series and it works well in my environment (64G - 512G guests). dom0 now balloons down just the required amount. Also, domU startup works the first time, as it correctly waits until memory is freed. (Using dom0_mem is still a preferred option, as the ballooning delay can be significant.) Thanks for all the help and patience as we've worked through this. Ack to the whole series: Acked-by: Mike Latimer mlati...@suse.com -Mike ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
