Re: [PATCH RFC V5 0/3] kvm: Improving directed yield in PLE handler
On 22/07/12 14:43, Avi Kivity wrote: On 07/22/2012 03:34 PM, Raghavendra K T wrote: Thanks Marcelo for the review. Avi, Rik, Christian, please let me know if this series looks good now. It looks fine to me. Christian, is this okay for s390? Tested-by: Christian Borntraeger borntrae...@de.ibm.com # on s390x -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC V5 0/3] kvm: Improving directed yield in PLE handler
On 07/18/2012 04:37 PM, Raghavendra K T wrote: Currently Pause Loop Exit (PLE) handler is doing directed yield to a random vcpu on pl-exit. We already have filtering while choosing the candidate to yield_to. This change adds more checks while choosing a candidate to yield_to. On a large vcpu guests, there is a high probability of yielding to the same vcpu who had recently done a pause-loop exit. Such a yield can lead to the vcpu spinning again. The patchset keeps track of the pause loop exit and gives chance to a vcpu which has: (a) Not done pause loop exit at all (probably he is preempted lock-holder) (b) vcpu skipped in last iteration because it did pause loop exit, and probably has become eligible now (next eligible lock holder) This concept also helps in cpu relax interception cases which use same handler. Thanks, applied to 'queue'. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC V5 0/3] kvm: Improving directed yield in PLE handler
On 07/20/2012 11:06 PM, Marcelo Tosatti wrote: On Wed, Jul 18, 2012 at 07:07:17PM +0530, Raghavendra K T wrote: Currently Pause Loop Exit (PLE) handler is doing directed yield to a random vcpu on pl-exit. We already have filtering while choosing the candidate to yield_to. This change adds more checks while choosing a candidate to yield_to. On a large vcpu guests, there is a high probability of yielding to the same vcpu who had recently done a pause-loop exit. Such a yield can lead to the vcpu spinning again. The patchset keeps track of the pause loop exit and gives chance to a vcpu which has: (a) Not done pause loop exit at all (probably he is preempted lock-holder) (b) vcpu skipped in last iteration because it did pause loop exit, and probably has become eligible now (next eligible lock holder) This concept also helps in cpu relax interception cases which use same handler. Changes since V4: - Naming Change (Avi): struct ple == struct spin_loop cpu_relax_intercepted == in_spin_loop vcpu_check_and_update_eligible == vcpu_eligible_for_directed_yield - mark vcpu in spinloop as not eligible to avoid influence of previous exit Changes since V3: - arch specific fix/changes (Christian) Changes since v2: - Move ple structure to common code (Avi) - rename pause_loop_exited to cpu_relax_intercepted (Avi) - add config HAVE_KVM_CPU_RELAX_INTERCEPT (Avi) - Drop superfluous curly braces (Ingo) Changes since v1: - Add more documentation for structure and algorithm and Rename plo == ple (Rik). - change dy_eligible initial value to false. (otherwise very first directed yield will not be skipped. (Nikunj) - fixup signoff/from issue Future enhancements: (1) Currently we have a boolean to decide on eligibility of vcpu. It would be nice if I get feedback on guest (32 vcpu) whether we can improve better with integer counter. (with counter = say f(log n )). (2) We have not considered system load during iteration of vcpu. With that information we can limit the scan and also decide whether schedule() is better. [ I am able to use #kicked vcpus to decide on this But may be there are better ideas like information from global loadavg.] (3) We can exploit this further with PV patches since it also knows about next eligible lock-holder. Summary: There is a very good improvement for kvm based guest on PLE machine. The V5 has huge improvement for kbench. +---+---+---++---+ base_rikstdev patched stdev %improve +---+---+---++---+ kernbench (time in sec lesser is better) +---+---+---++---+ 1x49.2300 1.017122.6842 0.3073117.0233 % 2x91.9358 1.776853.9608 1.015470.37516 % +---+---+---++---+ +---+---+---++---+ ebizzy (records/sec more is better) +---+---+---++---+ 1x 1129.250028.67932125.625032.823988.23334 % 2x 1892.375075.11122377.1250 181.682225.61596 % +---+---+---++---+ Note: The patches are tested on x86. Links V4: https://lkml.org/lkml/2012/7/16/80 V3: https://lkml.org/lkml/2012/7/12/437 V2: https://lkml.org/lkml/2012/7/10/392 V1: https://lkml.org/lkml/2012/7/9/32 Raghavendra K T (3): config: Add config to support ple or cpu relax optimzation kvm : Note down when cpu relax intercepted or pause loop exited kvm : Choose a better candidate for directed yield --- arch/s390/kvm/Kconfig|1 + arch/x86/kvm/Kconfig |1 + include/linux/kvm_host.h | 39 +++ virt/kvm/Kconfig |3 +++ virt/kvm/kvm_main.c | 41 + 5 files changed, 85 insertions(+), 0 deletions(-) Reviewed-by: Marcelo Tosattimtosa...@redhat.com Thanks Marcelo for the review. Avi, Rik, Christian, please let me know if this series looks good now. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC V5 0/3] kvm: Improving directed yield in PLE handler
On 07/22/2012 03:34 PM, Raghavendra K T wrote: Thanks Marcelo for the review. Avi, Rik, Christian, please let me know if this series looks good now. It looks fine to me. Christian, is this okay for s390? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC V5 0/3] kvm: Improving directed yield in PLE handler
On 07/22/2012 08:34 AM, Raghavendra K T wrote: On 07/20/2012 11:06 PM, Marcelo Tosatti wrote: On Wed, Jul 18, 2012 at 07:07:17PM +0530, Raghavendra K T wrote: Currently Pause Loop Exit (PLE) handler is doing directed yield to a random vcpu on pl-exit. We already have filtering while choosing the candidate to yield_to. This change adds more checks while choosing a candidate to yield_to. On a large vcpu guests, there is a high probability of yielding to the same vcpu who had recently done a pause-loop exit. Such a yield can lead to the vcpu spinning again. The patchset keeps track of the pause loop exit and gives chance to a vcpu which has: (a) Not done pause loop exit at all (probably he is preempted lock-holder) (b) vcpu skipped in last iteration because it did pause loop exit, and probably has become eligible now (next eligible lock holder) This concept also helps in cpu relax interception cases which use same handler. Changes since V4: - Naming Change (Avi): struct ple == struct spin_loop cpu_relax_intercepted == in_spin_loop vcpu_check_and_update_eligible == vcpu_eligible_for_directed_yield - mark vcpu in spinloop as not eligible to avoid influence of previous exit Changes since V3: - arch specific fix/changes (Christian) Changes since v2: - Move ple structure to common code (Avi) - rename pause_loop_exited to cpu_relax_intercepted (Avi) - add config HAVE_KVM_CPU_RELAX_INTERCEPT (Avi) - Drop superfluous curly braces (Ingo) Changes since v1: - Add more documentation for structure and algorithm and Rename plo == ple (Rik). - change dy_eligible initial value to false. (otherwise very first directed yield will not be skipped. (Nikunj) - fixup signoff/from issue Future enhancements: (1) Currently we have a boolean to decide on eligibility of vcpu. It would be nice if I get feedback on guest (32 vcpu) whether we can improve better with integer counter. (with counter = say f(log n )). (2) We have not considered system load during iteration of vcpu. With that information we can limit the scan and also decide whether schedule() is better. [ I am able to use #kicked vcpus to decide on this But may be there are better ideas like information from global loadavg.] (3) We can exploit this further with PV patches since it also knows about next eligible lock-holder. Summary: There is a very good improvement for kvm based guest on PLE machine. The V5 has huge improvement for kbench. +---+---+---++---+ base_rik stdev patched stdev %improve +---+---+---++---+ kernbench (time in sec lesser is better) +---+---+---++---+ 1x 49.2300 1.0171 22.6842 0.3073 117.0233 % 2x 91.9358 1.7768 53.9608 1.0154 70.37516 % +---+---+---++---+ +---+---+---++---+ ebizzy (records/sec more is better) +---+---+---++---+ 1x 1129.2500 28.6793 2125.6250 32.8239 88.23334 % 2x 1892.3750 75.1112 2377.1250 181.6822 25.61596 % +---+---+---++---+ Note: The patches are tested on x86. Links V4: https://lkml.org/lkml/2012/7/16/80 V3: https://lkml.org/lkml/2012/7/12/437 V2: https://lkml.org/lkml/2012/7/10/392 V1: https://lkml.org/lkml/2012/7/9/32 Raghavendra K T (3): config: Add config to support ple or cpu relax optimzation kvm : Note down when cpu relax intercepted or pause loop exited kvm : Choose a better candidate for directed yield --- arch/s390/kvm/Kconfig | 1 + arch/x86/kvm/Kconfig | 1 + include/linux/kvm_host.h | 39 +++ virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 41 + 5 files changed, 85 insertions(+), 0 deletions(-) Reviewed-by: Marcelo Tosattimtosa...@redhat.com Thanks Marcelo for the review. Avi, Rik, Christian, please let me know if this series looks good now. The series looks good to me. Reviewed-by: Rik van Riel r...@redhat.com -- All rights reversed -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC V5 0/3] kvm: Improving directed yield in PLE handler
On Wed, Jul 18, 2012 at 07:07:17PM +0530, Raghavendra K T wrote: Currently Pause Loop Exit (PLE) handler is doing directed yield to a random vcpu on pl-exit. We already have filtering while choosing the candidate to yield_to. This change adds more checks while choosing a candidate to yield_to. On a large vcpu guests, there is a high probability of yielding to the same vcpu who had recently done a pause-loop exit. Such a yield can lead to the vcpu spinning again. The patchset keeps track of the pause loop exit and gives chance to a vcpu which has: (a) Not done pause loop exit at all (probably he is preempted lock-holder) (b) vcpu skipped in last iteration because it did pause loop exit, and probably has become eligible now (next eligible lock holder) This concept also helps in cpu relax interception cases which use same handler. Changes since V4: - Naming Change (Avi): struct ple == struct spin_loop cpu_relax_intercepted == in_spin_loop vcpu_check_and_update_eligible == vcpu_eligible_for_directed_yield - mark vcpu in spinloop as not eligible to avoid influence of previous exit Changes since V3: - arch specific fix/changes (Christian) Changes since v2: - Move ple structure to common code (Avi) - rename pause_loop_exited to cpu_relax_intercepted (Avi) - add config HAVE_KVM_CPU_RELAX_INTERCEPT (Avi) - Drop superfluous curly braces (Ingo) Changes since v1: - Add more documentation for structure and algorithm and Rename plo == ple (Rik). - change dy_eligible initial value to false. (otherwise very first directed yield will not be skipped. (Nikunj) - fixup signoff/from issue Future enhancements: (1) Currently we have a boolean to decide on eligibility of vcpu. It would be nice if I get feedback on guest (32 vcpu) whether we can improve better with integer counter. (with counter = say f(log n )). (2) We have not considered system load during iteration of vcpu. With that information we can limit the scan and also decide whether schedule() is better. [ I am able to use #kicked vcpus to decide on this But may be there are better ideas like information from global loadavg.] (3) We can exploit this further with PV patches since it also knows about next eligible lock-holder. Summary: There is a very good improvement for kvm based guest on PLE machine. The V5 has huge improvement for kbench. +---+---+---++---+ base_rikstdev patched stdev %improve +---+---+---++---+ kernbench (time in sec lesser is better) +---+---+---++---+ 1x49.2300 1.017122.6842 0.3073117.0233 % 2x91.9358 1.776853.9608 1.015470.37516 % +---+---+---++---+ +---+---+---++---+ ebizzy (records/sec more is better) +---+---+---++---+ 1x 1129.250028.67932125.625032.823988.23334 % 2x 1892.375075.11122377.1250 181.682225.61596 % +---+---+---++---+ Note: The patches are tested on x86. Links V4: https://lkml.org/lkml/2012/7/16/80 V3: https://lkml.org/lkml/2012/7/12/437 V2: https://lkml.org/lkml/2012/7/10/392 V1: https://lkml.org/lkml/2012/7/9/32 Raghavendra K T (3): config: Add config to support ple or cpu relax optimzation kvm : Note down when cpu relax intercepted or pause loop exited kvm : Choose a better candidate for directed yield --- arch/s390/kvm/Kconfig|1 + arch/x86/kvm/Kconfig |1 + include/linux/kvm_host.h | 39 +++ virt/kvm/Kconfig |3 +++ virt/kvm/kvm_main.c | 41 + 5 files changed, 85 insertions(+), 0 deletions(-) Reviewed-by: Marcelo Tosatti mtosa...@redhat.com -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH RFC V5 0/3] kvm: Improving directed yield in PLE handler
Currently Pause Loop Exit (PLE) handler is doing directed yield to a random vcpu on pl-exit. We already have filtering while choosing the candidate to yield_to. This change adds more checks while choosing a candidate to yield_to. On a large vcpu guests, there is a high probability of yielding to the same vcpu who had recently done a pause-loop exit. Such a yield can lead to the vcpu spinning again. The patchset keeps track of the pause loop exit and gives chance to a vcpu which has: (a) Not done pause loop exit at all (probably he is preempted lock-holder) (b) vcpu skipped in last iteration because it did pause loop exit, and probably has become eligible now (next eligible lock holder) This concept also helps in cpu relax interception cases which use same handler. Changes since V4: - Naming Change (Avi): struct ple == struct spin_loop cpu_relax_intercepted == in_spin_loop vcpu_check_and_update_eligible == vcpu_eligible_for_directed_yield - mark vcpu in spinloop as not eligible to avoid influence of previous exit Changes since V3: - arch specific fix/changes (Christian) Changes since v2: - Move ple structure to common code (Avi) - rename pause_loop_exited to cpu_relax_intercepted (Avi) - add config HAVE_KVM_CPU_RELAX_INTERCEPT (Avi) - Drop superfluous curly braces (Ingo) Changes since v1: - Add more documentation for structure and algorithm and Rename plo == ple (Rik). - change dy_eligible initial value to false. (otherwise very first directed yield will not be skipped. (Nikunj) - fixup signoff/from issue Future enhancements: (1) Currently we have a boolean to decide on eligibility of vcpu. It would be nice if I get feedback on guest (32 vcpu) whether we can improve better with integer counter. (with counter = say f(log n )). (2) We have not considered system load during iteration of vcpu. With that information we can limit the scan and also decide whether schedule() is better. [ I am able to use #kicked vcpus to decide on this But may be there are better ideas like information from global loadavg.] (3) We can exploit this further with PV patches since it also knows about next eligible lock-holder. Summary: There is a very good improvement for kvm based guest on PLE machine. The V5 has huge improvement for kbench. +---+---+---++---+ base_rikstdev patched stdev %improve +---+---+---++---+ kernbench (time in sec lesser is better) +---+---+---++---+ 1x49.2300 1.017122.6842 0.3073117.0233 % 2x91.9358 1.776853.9608 1.015470.37516 % +---+---+---++---+ +---+---+---++---+ ebizzy (records/sec more is better) +---+---+---++---+ 1x 1129.250028.67932125.625032.823988.23334 % 2x 1892.375075.11122377.1250 181.682225.61596 % +---+---+---++---+ Note: The patches are tested on x86. Links V4: https://lkml.org/lkml/2012/7/16/80 V3: https://lkml.org/lkml/2012/7/12/437 V2: https://lkml.org/lkml/2012/7/10/392 V1: https://lkml.org/lkml/2012/7/9/32 Raghavendra K T (3): config: Add config to support ple or cpu relax optimzation kvm : Note down when cpu relax intercepted or pause loop exited kvm : Choose a better candidate for directed yield --- arch/s390/kvm/Kconfig|1 + arch/x86/kvm/Kconfig |1 + include/linux/kvm_host.h | 39 +++ virt/kvm/Kconfig |3 +++ virt/kvm/kvm_main.c | 41 + 5 files changed, 85 insertions(+), 0 deletions(-) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
