Re: Steal time accounting in KVM. Benchmark.
Hi Paolo, On 10/24/15 10:03 AM, Alexey Makhalov wrote: What I figured out. It happens in intersection of 3 features: *irq time accounting *stolen time accounting *linux guest with tickless idle only (not fully tickless) Looks like timer interrupts storm is happening during this benchmark (with 2:1 cpu overcommit). irq time accounting gets crazy. Even 'top' shows weird statistic: 50% hi, 50% st, ~0% user, spinning processes use ~0% cpu - that is not correct. If this is the desired behavior or something need to improve? Regards, Wanpeng Li -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Steal time accounting in KVM. Benchmark.
What I figured out. It happens in intersection of 3 features: *irq time accounting *stolen time accounting *linux guest with tickless idle only (not fully tickless) Looks like timer interrupts storm is happening during this benchmark (with 2:1 cpu overcommit). irq time accounting gets crazy. Even 'top' shows weird statistic: 50% hi, 50% st, ~0% user, spinning processes use ~0% cpu - that is not correct. Thanks. On Tue, Oct 20, 2015 at 5:24 PM, Alexey Makhalov wrote: > Yes, VM1 results are as before. > > Alexey > > On Tue, Oct 20, 2015 at 4:04 PM, Wanpeng Li wrote: >> On 10/21/15 4:05 AM, Alexey Makhalov wrote: >>> >>> 'echo NO_NONTASK_CAPACITY > /sys/kernel/debug/sched_features' in both >>> guests. >>> Results: >>> VM1: STA is disabled -- no changes, still little bit bellow expected 90% >>> VM2: STA is enabled -- result is changed, but still bad. Hard to say >>> better or worse. It prefers to stuck at quarters (100% 75% 50% 25%) >>> Output is attached. >> >> >> If the output in attachment is for VM2 only? >> >> Regards, >> Wanpeng Li -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Steal time accounting in KVM. Benchmark.
Yes, VM1 results are as before. Alexey On Tue, Oct 20, 2015 at 4:04 PM, Wanpeng Li wrote: > On 10/21/15 4:05 AM, Alexey Makhalov wrote: >> >> 'echo NO_NONTASK_CAPACITY > /sys/kernel/debug/sched_features' in both >> guests. >> Results: >> VM1: STA is disabled -- no changes, still little bit bellow expected 90% >> VM2: STA is enabled -- result is changed, but still bad. Hard to say >> better or worse. It prefers to stuck at quarters (100% 75% 50% 25%) >> Output is attached. > > > If the output in attachment is for VM2 only? > > Regards, > Wanpeng Li -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Steal time accounting in KVM. Benchmark.
On 10/21/15 4:05 AM, Alexey Makhalov wrote: 'echo NO_NONTASK_CAPACITY > /sys/kernel/debug/sched_features' in both guests. Results: VM1: STA is disabled -- no changes, still little bit bellow expected 90% VM2: STA is enabled -- result is changed, but still bad. Hard to say better or worse. It prefers to stuck at quarters (100% 75% 50% 25%) Output is attached. If the output in attachment is for VM2 only? Regards, Wanpeng Li -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Steal time accounting in KVM. Benchmark.
'echo NO_NONTASK_CAPACITY > /sys/kernel/debug/sched_features' in both guests. Results: VM1: STA is disabled -- no changes, still little bit bellow expected 90% VM2: STA is enabled -- result is changed, but still bad. Hard to say better or worse. It prefers to stuck at quarters (100% 75% 50% 25%) Output is attached. Thanks, Alexey On Tue, Oct 20, 2015 at 5:39 AM, Wanpeng Li wrote: > Cc Peterz, > 2015-10-20 5:58 GMT+08:00 Alexey Makhalov : >> >> Hi, >> >> I did benchmarking of scheduler fairness with enabled steal time >> accounting(STA) in KVM. >> And results are really interesting. >> >> Looks like STA provides worse scheduler fairness against disabled STA >> (no-steal-acc cmdline param) >> >> I created benchmark, main idea is: 2 cgroups with cpu.shares >> proportion like 1/9, run identical work in both groups, and expecting >> to get the same proportion of work done – 1/9. Condition – CPU >> overcommit. >> >> On bare metal it is fair +- some percentages of fluctation. >> On KVM with no STA it’s less fair. With STA enabled results are >> ugly! One again – in CPU overcommit situation. >> >> >> Host: ubuntu 14.04, Intel i5 – 2x2 CPUs >> 2 VMs (4 vCPU each) are working in parallel. 2:1 cpu overcommit. >> >> Each VM has running benchmark: >> cgroups cpu.shares proportion is 128/1152 (10%/90%), work – spinning >> in cycle, number of cycles are being counted. > > > Could you try if 'echo NO_NONTASK_CAPACITY > > /sys/kernel/debug/sched_features' in guests works? > > Regards, > Wanpeng Li 90% 28539 75% 31907 76% 30348 75% 43677 75% 29765 63% 30351 74% 28535 70% 29590 65% 31281 87% 29476 75% 29505 75% 29737 75% 29534 75% 29627 89% 30411 75% 29069 74% 29786 74% 29776 74% 29633 74% 29470 74% 29406 90% 30274 90% 29615 96% 30497 100% 29263 100% 29241 100% 29748 100% 29790 99% 29769 74% 66239 52% 118436 50% 117409 50% 118167 50% 118990 50% 38479 48% 29641 50% 29757 50% 29754 50% 29688 50% 29671 50% 29745 50% 29831 50% 29596 50% 29645 50% 29661 50% 29567 50% 29511 50% 29763 50% 29919 51% 28823 51% 29944 50% 29501 50% 29406 50% 29410 49% 29519 51% 30109 49% 29586 49% 29621 49% 29579 43% 80183 49% 118185 49% 113732 45% 29698 25% 29528 24% 30078 36% 29716 32% 28976 43% 77606 19% 131592 0% 39503 12% 29709 25% 29717 24% 29720 25% 29695 25% 29703 25% 29570 25% 29492 25% 29790 25% 29677 25% 29451 25% 29536 25% 29476 25% 29613 24% 30045 24% 30227 37% 52553 43% 29986 50% 29741 49% 30003 50% 29982 35% 29907 39% 30156 48% 30590 66% 58939 --- benchmark was stopped on the first VM. VM1 was in idle from here --- 90% 58977 89% 58952 90% 58945 89% 58945 90% 58948 89% 59001 89% 58966 90% 58966 90% 59005 89% 58998 90% 58998 90% 59010 90% 59028 90% 59070 90% 58558 90% 58948 89% 59080 89% 58601 90% 58385 90% 59003 89% 57695 90% 58843 89% 59104 90% 59006 90% 57110 --- to here --- 97% 30300 100% 29335 100% 29541 100% 29609 100% 29880 100% 29925 100% 29517 100% 29723 100% 28467 99% 29597 100% 29817 100% 29772 100% 29761 100% 29547 100% 29556 100% 29848 100% 29713 100% 29616 100% 29659 100% 29709 100% 29757 100% 29806 100% 30357 100% 29784 100% 29812 100% 29679 100% 29839 100% 29633 100% 29650 100% 29730 100% 29964 100% 29993 100% 29899 100% 29951 100% 29924 100% 29938 100% 29784 100% 30043 100% 29308 99% 30625 95% 29907 99% 29811 100% 29626 100% 30054 100% 29744 100% 29693 100% 30008 100% 29935 100% 29825 100% 29684 100% 30010 100% 30163 93% 51597 89% 58972 89% 58692 90% 57975 100% 27946 100% 29624 100% 30365 98% 30468 99% 30971 100% 30907 100% 28079 100% 30074 100% 29001 100% 30366 99% 29814 100% 30405 97% 29866 100% 29003 100% 28545 100% 29693 100% 30013 100% 28179 100% 28644 100% 30081 100% 28191 100% 31325 100% 30223 100% 29848 100% 30057 100% 29675 100% 29859 100% 28160 100% 29679 100% 29689 100% 29727 100% 29715 100% 30547 100% 29281 100% 29761 100% 29596 100% 29797 100% 29664 100% 29768 100% 29708 100% 29614 100% 29627 100% 30158 100% 29801 100% 28569 100% 29118 100% 30540 100% 29335 100% 29188 99% 30413 100% 29438 100% 28382 100% 29492 100% 28555 100% 29700 99% 30374 75% 29759 75% 29954 75% 29588 75% 29540 75% 29616 75% 29890 75% 29457 75% 29466 75% 29724 74% 29395 74% 29613 75% 29776 75% 29455 75% 29788 75% 29592 75% 29679 75% 29749 75% 29715 75% 29992 75% 29271 73% 30173 74% 29480 75% 29415 75% 28416 75% 29786 75% 29086 75% 29293 75% 29670 75% 29898 75% 29756 75% 29498 75% 29946 75% 29328 75% 29120 75% 29821 75% 29765 75% 29673 75% 29682 75% 29657 74% 30045 75% 29641 75% 29662 75% 29595 75% 29635 75% 29720 73% 30315 75% 29589 75% 29607 75% 29629 75% 29738 75% 30068 72% 28898 75% 26530 74% 30075 73% 29182 74% 28506 73% 28135 74% 29083 75% 29609 73% 27956 74% 28921 74% 29045 74% 29576 74% 29228 74% 29494 74% 29218 73% 27872 74% 29787 74% 29633 68% 32062 98% 31992 98% 32120 90% 32984 98% 32568 98% 32307 97% 32300 98% 32262 99% 30436 94% 30023 96% 32511 99% 31075 99% 30112 100% 29835 100% 30079 100% 29980 100% 29954 100% 29771 100% 29993 100% 29796 100% 29895 100% 29748 100% 29846 100% 29950 100% 29900 100% 29871 100% 29826 100
Steal time accounting in KVM. Benchmark.
Hi, I did benchmarking of scheduler fairness with enabled steal time accounting(STA) in KVM. And results are really interesting. Looks like STA provides worse scheduler fairness against disabled STA (no-steal-acc cmdline param) I created benchmark, main idea is: 2 cgroups with cpu.shares proportion like 1/9, run identical work in both groups, and expecting to get the same proportion of work done – 1/9. Condition – CPU overcommit. On bare metal it is fair +- some percentages of fluctation. On KVM with no STA it’s less fair. With STA enabled results are ugly! One again – in CPU overcommit situation. Host: ubuntu 14.04, Intel i5 – 2x2 CPUs 2 VMs (4 vCPU each) are working in parallel. 2:1 cpu overcommit. Each VM has running benchmark: cgroups cpu.shares proportion is 128/1152 (10%/90%), work – spinning in cycle, number of cycles are being counted. Results: First column – work proportion. Expected value 90% Second column – total number of loop cycles from all workers (x10). Interval between printed lines – 1 sec VM1: STA is disabled (no-steal-acc) 85% 29683 85% 29520 87% 29574 87% 31386 84% 29578 87% 29595 86% 29644 87% 29655 87% 30059 88% 3 90% 30037 90% 29720 90% 29951 86% 29958 88% 29971 90% 29829 82% 29841 82% 29998 82% 29752 80% 29722 88% 28825 85% 28283 85% 29746 77% 29657 82% 29674 84% 29764 87% 29567 79% 29529 84% 29633 VM2: STA is enabled 51% 30709 38% 29564 44% 29429 47% 29759 41% 29792 0% 29987 51% 52666 50% 29753 50% 29670 54% 29461 45% 29927 35% 29743 50% 29683 50% 29971 47% 30033 0% 121204 7% 116039 25% 119069 24% 40269 5% 37684 24% 30039 22% 30269 44% 31861 29% 29652 24% 29558 24% 29783 24% 29494 24% 29607 43% 29613 25% 29377 My benchmark sources: https://github.com/YustasSwamp/steal-time-benchmark Any ideas? Do you have any steal time specific benchmarks? Thanks, Alexey -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html