That’s very interesting. I wasn’t aware of this “stress” tool. So I’ve
downloaded it and run a couple tests with it.
If I run basic —cpu 1 test (-n according to help is a dry run), the KVM server
spins the CPU 100% in user time. So no stealing at all.
Could you run a couple of tests like this (I am providing my own results):
KVM server (2 CPU but it is one threaded task so doesn’t matter how many)
# time for i in {1..500}; do dd if=/dev/zero of=/dev/shm/test bs=1M count=10
;echo interation $i done; done
….
10485760 bytes (10 MB) copied, 0.00273469 s, 3.8 GB/s
interation 500 done
real 0m2.223s
user 0m0.171s
sys 0m2.002s
During the test (I’ve changed 1..500 to 1..5000 to have more time to catch top
output) top was showing on average:
%Cpu1 : 7.0 us, 83.7 sy, 0.0 ni, 7.0 id, 0.0 wa, 0.0 hi, 0.0 si, 2.3 st
KVM virtual machine(1 CPU adding CPUs will not make difference in this case):
# time for i in {1..500}; do dd if=/dev/zero of=/dev/shm/test bs=1M count=10
;echo interation $i done; done
10485760 bytes (10 MB) copied, 0.0524781 s, 200 MB/s
interation 500 done
real 0m38.556s
user 0m1.180s
sys 0m11.550s
During the test (no need to do more than 1..500 becasue there is enough time to
check top) the KVM host was showing:
%Cpu0 : 53.7 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.7 hi, 0.0 si, 45.6 st
And KVM virtual machine itself was showing
%Cpu(s): 3.8 us, 30.1 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 66.2 st
Now pure CPU stress tests
KVM Host:
# time for i in {1..10}; do dd if=/dev/urandom of=/dev/null bs=1M count=10
;echo interation $i done; done
0485760 bytes (10 MB) copied, 0.630299 s, 16.6 MB/s
interation 10 done
real 0m6.294s
user 0m0.004s
sys 0m6.273s
top output on KVM host
%Cpu1 : 0.0 us,100.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KVM virtual machine:
# # time for i in {1..10}; do dd if=/dev/urandom of=/dev/null bs=1M count=10
;echo interation $i done; done
10485760 bytes (10 MB) copied, 0.64323 s, 16.3 MB/s
interation 10 done
real 0m6.779s
user 0m0.014s
sys 0m6.345s
top output on KVM host
%Cpu1 : 96.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 4.0 st
top output on KVM virtual machine
0.6 us, 85.4 sy, 0.0 ni, 7.9 id, 0.0 wa, 0.0 hi, 0.0 si, 6.0 st
So it seems that when it is about only CPU the difference is very small or
none. I am getting similar results if I do complicated equations with big
numbers (virtual machines solves it almost in the same time as 1st level host).
But when memory is involved, everything slows down drastically. I wonder what
results you will get from the dd from /dev/zero to /dev/shm.
And no, my system has plenty of memory, paging in z/VM is ZERO. Hardly anything
runs on this LPAR.
KVM host has plenty of real memory - 8G and the virtual machine is set to 4G.
It still has few Gigs left. No swapping, nothing else runs here.
Another huge difference in times is …. starting yast. On 1st level host it
takes fractures of a second. On the KVM virtual machine it takes >1min
Best regards
Gregory P
> On Sep 23, 2015, at 6:42 PM, Viktor Mihajlovski <mihaj...@linux.vnet.ibm.com>
> wrote:
>
> On 23.09.2015 15:32, Grzegorz Powiedziuk wrote:
>> BTW, I was playing with KVM few days ago and it looks pretty awesome in
>> terms of maintaining the environment and deploying new VMs but the
>> performance for me was really bad.
>> And I mean extremely bad. I am not sure if it was because I made the KVM
>> host (sles12) run as a virtual machine in z/VM or I was doing something
>> else wrong. I know that having kvm virtual machines in a 3rd level (under
>> sles -> under z/VM) will impact performance but my case it was extremly bad.
>> It was like running linux in hercules s390 in 2006 on old x86 desktop.
>>
>> The installation of linux in kvm virtual machine took 3-4 hours. Every
>> operation that involves cpu and memory takes 3-10 time more time than on a
>> KVM host itself.
>> Whenever something is happening in kvm virtual machine, the performance
>> toolkit shows that KVM host is doing about 50% in supervisor mode and 50% in
>> emulation mode which makes the t/v ratio for this machine about 2 which is
>> pretty bad. I didn’t have time to do more investigation on this yet.
>> The KVM host (Server) sees about 50% cpu time as a “steal time”.
>>
>
> Although KVM should generally be run in LPAR a slowdown in an order of
> magnitude in z/VM seems a bit odd.
> I actually do run KVM under z/VM at my own risk (I am absolutely NOT
> suggesting to do that). To recreate your problems I started a CPU burning
> task in a KVM guest (stress -n 1) and see the following in the host:
>
> Tasks: 162 total, 1 running, 161 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 32,5 us, 0,7 sy, 0,0 ni, 63,8 id, 0,0 wa, 0,0 hi, 0,1 si, 2,9
> st
> KiB Mem: 812852 total, 801232 used, 11620 free, 3152 buffers
> KiB Swap: 7212140 total, 1789268 used, 5422872 free, 611460 cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 38763 qemu 20 0 2842280 353068 350952 S 94,8 43,4 58:46.16
> qemu-system-s39
>
> As you can see it's less than 3% steal time in the host with roughly 95%
> problem state CPU utilization for the guest process (which indicates that SIE
> isn't too bad even for a 3rd level guest).
>
> The high steal time you observe could be a hint for either z/VM swapping on
> behalf of the KVM host or KVM swapping itself. Could you observe a high page
> rate?
>
> --
>
> Mit freundlichen Grüßen/Kind Regards
> Viktor Mihajlovski
>
> IBM Deutschland Research & Development GmbH
> Vorsitzender des Aufsichtsrats: Martina Köderitz
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen
> Registergericht: Amtsgericht Stuttgart, HRB 243294
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> ----------------------------------------------------------------------
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
