Re: Documentation for Linux on z Systems and KVM - new
> I stay in shape running to catch up.. Good point :) WBR, Sergey Scott Rohling Sent by: Linux on 390 Port 01-10-15 18:05 Please respond to Linux on 390 Port To: LINUX-390@VM.MARIST.EDU cc: Subject: Re: Documentation for Linux on z Systems and KVM - new 'The speed of progress' ?Is it not moving fast enough for you, Sergey? Perhaps we need 'dinosaur crossing' signs stamped on the z... ? I am personally not worried about slowing down the world ... I have failed even when actively trying ;-) I stay in shape running to catch up.. Scott Rohling On Thu, Oct 1, 2015 at 12:40 AM, Sergey Korzhevsky wrote: > Hi Mark, > >Thank you for detailed answer, but, actually, your vision is not > contradicted with mine, you just confirmed that the huge jump was made in > the last decade because of Linux on mainframe (in z/VM particularly). > I hope that with people like you, the speed of progress won't be slow > down. > > > WBR, Sergey > > > > > Mark Post > Sent by: Linux on 390 Port > 28-09-15 23:39 > Please respond to Linux on 390 Port > > To: LINUX-390@VM.MARIST.EDU > cc: > Subject:Re: Documentation for Linux on z Systems and KVM - > new > > > >>> On 9/28/2015 at 02:25 AM, Sergey Korzhevsky > wrote: > > Alan Altmark wrote: > >>> What is it about z that makes virtualization work better? > >>50 years of work on it? > > > > That is interesting answer. One thing came to my mind is the live guest > > relocation. As far as i could find, VMware introduced that feature > > (vMotion) in 2003, z/VM - in 2011. The same regarding network > > (GuestLAN/VSwitch). > > So, looks like z/VM slept all that years and was wake up by x86 world > > recently. > > Having been an active participant and observer of the community for a > while now, I think I can contribute some perspective. (From what I can > tell, you have been also so I find your comment a little surprising.) > > When Linux for the mainframe was first introduced, a lot of facilities we > take for granted today didn't exist. Guest LANs, VSWITCHes, cooperative > memory management and so on. That started to change pretty quickly. > Things that actually helped running more than just a few instances of > Linux were introduced and made life much easier. Live Guest Relocation > wasn't needed then, because not many shops were running huge amounts of > guests. That pain came along later. Even then, it wasn't for the same > reason that the x86 world wanted it. > > Mainframe shops running Linux on z/VM didn't worry much about hardware > failures and migrating workload to relieve overloaded servers usually > wasn't an issue because of decades of performance and capacity management. > What "we" wanted it for was because z/VM was so reliable it could run for > years but sometimes various maintenance was important to put on the > system. Trying to get multiple customers of the service to agree on a > maintenance window was becoming nearly impossible, because although they > wanted High Availability, they weren't willing to actually invest in it, > so the workload couldn't be failed over to another server in a cluster. > > There was another factor, although not a technical one. Many customers > have become checklist driven. If your product doesn't allow them to put > check marks in all the boxes on the list, it's obviously not a good > product and not worthy of consideration. So, z/VM development was getting > reports from Sales that this function was needed, just to be "in the > game." And, being the group that they are, z/VM development wanted to > approach the development needed in a more "system of systems" oriented way > than just bolting on a feature. Thus, Single System Image was born, and > it took quite a while and a lot of people to bring to the market. Taking > into account the various diversions that were forced on them during the > same period of time, it's amazing they got it out as quickly as they did. > > I think most people that have been in the z/VM world for a long time would > agree that having Linux available on the mainframe has breathed new life > into z/VM. Since then, they've been working hard to introduce things that > make sense for the mainframe environment. What new items they work on, > and what priority they have, _can_ be influenced by current and potential > customers. I encourage anyone who has thoughts on what those new items > should be to speak up, whether here or in the IBMVM mailing list, or at > SHARE. There are people in the
Re: Documentation for Linux on z Systems and KVM - new
'The speed of progress' ?Is it not moving fast enough for you, Sergey? Perhaps we need 'dinosaur crossing' signs stamped on the z... ? I am personally not worried about slowing down the world ... I have failed even when actively trying ;-) I stay in shape running to catch up.. Scott Rohling On Thu, Oct 1, 2015 at 12:40 AM, Sergey Korzhevsky wrote: > Hi Mark, > >Thank you for detailed answer, but, actually, your vision is not > contradicted with mine, you just confirmed that the huge jump was made in > the last decade because of Linux on mainframe (in z/VM particularly). > I hope that with people like you, the speed of progress won't be slow > down. > > > WBR, Sergey > > > > > Mark Post > Sent by: Linux on 390 Port > 28-09-15 23:39 > Please respond to Linux on 390 Port > > To: LINUX-390@VM.MARIST.EDU > cc: > Subject:Re: Documentation for Linux on z Systems and KVM - > new > > > >>> On 9/28/2015 at 02:25 AM, Sergey Korzhevsky > wrote: > > Alan Altmark wrote: > >>> What is it about z that makes virtualization work better? > >>50 years of work on it? > > > > That is interesting answer. One thing came to my mind is the live guest > > relocation. As far as i could find, VMware introduced that feature > > (vMotion) in 2003, z/VM - in 2011. The same regarding network > > (GuestLAN/VSwitch). > > So, looks like z/VM slept all that years and was wake up by x86 world > > recently. > > Having been an active participant and observer of the community for a > while now, I think I can contribute some perspective. (From what I can > tell, you have been also so I find your comment a little surprising.) > > When Linux for the mainframe was first introduced, a lot of facilities we > take for granted today didn't exist. Guest LANs, VSWITCHes, cooperative > memory management and so on. That started to change pretty quickly. > Things that actually helped running more than just a few instances of > Linux were introduced and made life much easier. Live Guest Relocation > wasn't needed then, because not many shops were running huge amounts of > guests. That pain came along later. Even then, it wasn't for the same > reason that the x86 world wanted it. > > Mainframe shops running Linux on z/VM didn't worry much about hardware > failures and migrating workload to relieve overloaded servers usually > wasn't an issue because of decades of performance and capacity management. > What "we" wanted it for was because z/VM was so reliable it could run for > years but sometimes various maintenance was important to put on the > system. Trying to get multiple customers of the service to agree on a > maintenance window was becoming nearly impossible, because although they > wanted High Availability, they weren't willing to actually invest in it, > so the workload couldn't be failed over to another server in a cluster. > > There was another factor, although not a technical one. Many customers > have become checklist driven. If your product doesn't allow them to put > check marks in all the boxes on the list, it's obviously not a good > product and not worthy of consideration. So, z/VM development was getting > reports from Sales that this function was needed, just to be "in the > game." And, being the group that they are, z/VM development wanted to > approach the development needed in a more "system of systems" oriented way > than just bolting on a feature. Thus, Single System Image was born, and > it took quite a while and a lot of people to bring to the market. Taking > into account the various diversions that were forced on them during the > same period of time, it's amazing they got it out as quickly as they did. > > I think most people that have been in the z/VM world for a long time would > agree that having Linux available on the mainframe has breathed new life > into z/VM. Since then, they've been working hard to introduce things that > make sense for the mainframe environment. What new items they work on, > and what priority they have, _can_ be influenced by current and potential > customers. I encourage anyone who has thoughts on what those new items > should be to speak up, whether here or in the IBMVM mailing list, or at > SHARE. There are people in these mailing lists and at SHARE that have a > direct line into the z/VM and Linux development groups at IBM. Take > advantage of that. > > > Mark Post > > -- > For LINUX-390 subscribe / signoff / archive access i
Re: Documentation for Linux on z Systems and KVM - new
Hi Mark, Thank you for detailed answer, but, actually, your vision is not contradicted with mine, you just confirmed that the huge jump was made in the last decade because of Linux on mainframe (in z/VM particularly). I hope that with people like you, the speed of progress won't be slow down. WBR, Sergey Mark Post Sent by: Linux on 390 Port 28-09-15 23:39 Please respond to Linux on 390 Port To: LINUX-390@VM.MARIST.EDU cc: Subject: Re: Documentation for Linux on z Systems and KVM - new >>> On 9/28/2015 at 02:25 AM, Sergey Korzhevsky wrote: > Alan Altmark wrote: >>> What is it about z that makes virtualization work better? >>50 years of work on it? > > That is interesting answer. One thing came to my mind is the live guest > relocation. As far as i could find, VMware introduced that feature > (vMotion) in 2003, z/VM - in 2011. The same regarding network > (GuestLAN/VSwitch). > So, looks like z/VM slept all that years and was wake up by x86 world > recently. Having been an active participant and observer of the community for a while now, I think I can contribute some perspective. (From what I can tell, you have been also so I find your comment a little surprising.) When Linux for the mainframe was first introduced, a lot of facilities we take for granted today didn't exist. Guest LANs, VSWITCHes, cooperative memory management and so on. That started to change pretty quickly. Things that actually helped running more than just a few instances of Linux were introduced and made life much easier. Live Guest Relocation wasn't needed then, because not many shops were running huge amounts of guests. That pain came along later. Even then, it wasn't for the same reason that the x86 world wanted it. Mainframe shops running Linux on z/VM didn't worry much about hardware failures and migrating workload to relieve overloaded servers usually wasn't an issue because of decades of performance and capacity management. What "we" wanted it for was because z/VM was so reliable it could run for years but sometimes various maintenance was important to put on the system. Trying to get multiple customers of the service to agree on a maintenance window was becoming nearly impossible, because although they wanted High Availability, they weren't willing to actually invest in it, so the workload couldn't be failed over to another server in a cluster. There was another factor, although not a technical one. Many customers have become checklist driven. If your product doesn't allow them to put check marks in all the boxes on the list, it's obviously not a good product and not worthy of consideration. So, z/VM development was getting reports from Sales that this function was needed, just to be "in the game." And, being the group that they are, z/VM development wanted to approach the development needed in a more "system of systems" oriented way than just bolting on a feature. Thus, Single System Image was born, and it took quite a while and a lot of people to bring to the market. Taking into account the various diversions that were forced on them during the same period of time, it's amazing they got it out as quickly as they did. I think most people that have been in the z/VM world for a long time would agree that having Linux available on the mainframe has breathed new life into z/VM. Since then, they've been working hard to introduce things that make sense for the mainframe environment. What new items they work on, and what priority they have, _can_ be influenced by current and potential customers. I encourage anyone who has thoughts on what those new items should be to speak up, whether here or in the IBMVM mailing list, or at SHARE. There are people in these mailing lists and at SHARE that have a direct line into the z/VM and Linux development groups at IBM. Take advantage of that. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
Thank you, Mark DJ On 09/30/2015 11:08 AM, Mark Post wrote: On 9/30/2015 at 11:58 AM, Dave Jones wrote: >> Hello, all. >> >> I now have the zKVM DVD 1.1 here. Where can I find the >> installation/configuration doc for it? > > wget -N http://public.dhe.ibm.com/software/dw/linux390/docu/l159vq00.pdf > wget -N http://public.dhe.ibm.com/software/dw/linux390/docu/l159va00.pdf > wget -N http://public.dhe.ibm.com/software/dw/linux390/docu/l4n2vg00.pdf > wget -N http://public.dhe.ibm.com/software/dw/linux390/docu/lhs0vi00.pdf > > > Mark Post > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > -- Dave Jones Houston, TX 281.578.7544 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
>>> On 9/30/2015 at 11:58 AM, Dave Jones wrote: > Hello, all. > > I now have the zKVM DVD 1.1 here. Where can I find the > installation/configuration doc for it? wget -N http://public.dhe.ibm.com/software/dw/linux390/docu/l159vq00.pdf wget -N http://public.dhe.ibm.com/software/dw/linux390/docu/l159va00.pdf wget -N http://public.dhe.ibm.com/software/dw/linux390/docu/l4n2vg00.pdf wget -N http://public.dhe.ibm.com/software/dw/linux390/docu/lhs0vi00.pdf Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
Hello, all. I now have the zKVM DVD 1.1 here. Where can I find the installation/configuration doc for it? Thanks and have a good one, too. DJ On 09/24/2015 02:51 AM, Christian Borntraeger wrote: > > z/VM does call SIE on behalf of the guest hypervisor. So for CPU bound > workload, which causes > almost no SIE exits things are fine. It is the sweet spot for 2nd level. As > soon as the KVM > guest will have many exits (e.g. some I/O, memory fault-in, reschedule) or > more than one CPU > in the KVM host things can get really slow as z/VM then has to interpret lots > of things. > In addition z/VM 2nd level support was in no way optimized to speed up a KVM > hypervisor, so > I assume that some of the optimizations for z/VM under z/VM have to fall back > to the slow path. > > Christian > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > -- Dave Jones Houston, TX 281.578.7544 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
Mark, Very nice, informative, post. Harley Linker Jr. -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Mark Post Sent: Monday, September 28, 2015 3:39 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: Documentation for Linux on z Systems and KVM - new >>> On 9/28/2015 at 02:25 AM, Sergey Korzhevsky wrote: > Alan Altmark wrote: >>> What is it about z that makes virtualization work better? >>50 years of work on it? > > That is interesting answer. One thing came to my mind is the live > guest relocation. As far as i could find, VMware introduced that > feature > (vMotion) in 2003, z/VM - in 2011. The same regarding network > (GuestLAN/VSwitch). > So, looks like z/VM slept all that years and was wake up by x86 world > recently. Having been an active participant and observer of the community for a while now, I think I can contribute some perspective. (From what I can tell, you have been also so I find your comment a little surprising.) When Linux for the mainframe was first introduced, a lot of facilities we take for granted today didn't exist. Guest LANs, VSWITCHes, cooperative memory management and so on. That started to change pretty quickly. Things that actually helped running more than just a few instances of Linux were introduced and made life much easier. Live Guest Relocation wasn't needed then, because not many shops were running huge amounts of guests. That pain came along later. Even then, it wasn't for the same reason that the x86 world wanted it. Mainframe shops running Linux on z/VM didn't worry much about hardware failures and migrating workload to relieve overloaded servers usually wasn't an issue because of decades of performance and capacity management. What "we" wanted it for was because z/VM was so reliable it could run for years but sometimes various maintenance was important to put on the system. Trying to get multiple customers of the service to agree on a maintenance window was becoming nearly impossible, because although they wanted High Availability, they weren't willing to actually invest in it, so the workload couldn't be failed over to another server in a cluster. There was another factor, although not a technical one. Many customers have become checklist driven. If your product doesn't allow them to put check marks in all the boxes on the list, it's obviously not a good product and not worthy of consideration. So, z/VM development was getting reports from Sales that this function was needed, just to be "in the game." And, being the group that they are, z/VM development wanted to approach the development needed in a more "system of systems" oriented way than just bolting on a feature. Thus, Single System Image was born, and it took quite a while and a lot of people to bring to the market. Taking into account the various diversions that were forced on them during the same period of time, it's amazing they got it out as quickly as they did. I think most people that have been in the z/VM world for a long time would agree that having Linux available on the mainframe has breathed new life into z/VM. Since then, they've been working hard to introduce things that make sense for the mainframe environment. What new items they work on, and what priority they have, _can_ be influenced by current and potential customers. I encourage anyone who has thoughts on what those new items should be to speak up, whether here or in the IBMVM mailing list, or at SHARE. There are people in these mailing lists and at SHARE that have a direct line into the z/VM and Linux development groups at IBM. Take advantage of that. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ *** The information contained in this communication is confidential, is intended only for the use of the recipient named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please resend this communication to the sender and delete the original message or any copy of it from your computer system. Thank You. -
Re: Documentation for Linux on z Systems and KVM - new
Good post - well said, Mark - Scott Rohling On Mon, Sep 28, 2015 at 1:39 PM, Mark Post wrote: > a good post > > -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
>>> On 9/28/2015 at 02:25 AM, Sergey Korzhevsky wrote: > Alan Altmark wrote: >>> What is it about z that makes virtualization work better? >>50 years of work on it? > > That is interesting answer. One thing came to my mind is the live guest > relocation. As far as i could find, VMware introduced that feature > (vMotion) in 2003, z/VM - in 2011. The same regarding network > (GuestLAN/VSwitch). > So, looks like z/VM slept all that years and was wake up by x86 world > recently. Having been an active participant and observer of the community for a while now, I think I can contribute some perspective. (From what I can tell, you have been also so I find your comment a little surprising.) When Linux for the mainframe was first introduced, a lot of facilities we take for granted today didn't exist. Guest LANs, VSWITCHes, cooperative memory management and so on. That started to change pretty quickly. Things that actually helped running more than just a few instances of Linux were introduced and made life much easier. Live Guest Relocation wasn't needed then, because not many shops were running huge amounts of guests. That pain came along later. Even then, it wasn't for the same reason that the x86 world wanted it. Mainframe shops running Linux on z/VM didn't worry much about hardware failures and migrating workload to relieve overloaded servers usually wasn't an issue because of decades of performance and capacity management. What "we" wanted it for was because z/VM was so reliable it could run for years but sometimes various maintenance was important to put on the system. Trying to get multiple customers of the service to agree on a maintenance window was becoming nearly impossible, because although they wanted High Availability, they weren't willing to actually invest in it, so the workload couldn't be failed over to another server in a cluster. There was another factor, although not a technical one. Many customers have become checklist driven. If your product doesn't allow them to put check marks in all the boxes on the list, it's obviously not a good product and not worthy of consideration. So, z/VM development was getting reports from Sales that this function was needed, just to be "in the game." And, being the group that they are, z/VM development wanted to approach the development needed in a more "system of systems" oriented way than just bolting on a feature. Thus, Single System Image was born, and it took quite a while and a lot of people to bring to the market. Taking into account the various diversions that were forced on them during the same period of time, it's amazing they got it out as quickly as they did. I think most people that have been in the z/VM world for a long time would agree that having Linux available on the mainframe has breathed new life into z/VM. Since then, they've been working hard to introduce things that make sense for the mainframe environment. What new items they work on, and what priority they have, _can_ be influenced by current and potential customers. I encourage anyone who has thoughts on what those new items should be to speak up, whether here or in the IBMVM mailing list, or at SHARE. There are people in these mailing lists and at SHARE that have a direct line into the z/VM and Linux development groups at IBM. Take advantage of that. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
I read these test figures from KVM run within z/VM.
In my world you run a hypervisor on the iron, not under another hypervisor, to
get performance.
Hypervisor under hypervisor is for test only.
So test KVN in an LPAR instead, you will miss a lot of z/VM goodies, but you
get the performance.
/Tore
Tore Agblad
zOpen, IT Services
Volvo Group Headquarters
Corporate Process & IT
SE-405 08, Gothenburg Sweden
E-mail: tore.agb...@volvo.com
http://www.volvo.com/volvoit/global/en-gb/
-Original Message-
From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Christian
Borntraeger
Sent: den 24 september 2015 11:08
To: LINUX-390@VM.MARIST.EDU
Subject: Re: Documentation for Linux on z Systems and KVM - new
Am 24.09.2015 um 03:34 schrieb Grzegorz Powiedziuk:
> That’s very interesting. I wasn’t aware of this “stress” tool. So I’ve
> downloaded it and run a couple tests with it.
> If I run basic —cpu 1 test (-n according to help is a dry run), the KVM
> server spins the CPU 100% in user time. So no stealing at all.
>
> Could you run a couple of tests like this (I am providing my own results):
>
> KVM server (2 CPU but it is one threaded task so doesn’t matter how many)
Does changing the KVM server to 1 CPU makes things better?
Having more than one CPU makes it harder for z/VM to virtualize
KVMs usage of SIE.
> # time for i in {1..500}; do dd if=/dev/zero of=/dev/shm/test bs=1M count=10
> ;echo interation $i done; done
> ….
> 10485760 bytes (10 MB) copied, 0.00273469 s, 3.8 GB/s
> interation 500 done
>
> real 0m2.223s
> user 0m0.171s
> sys 0m2.002s
>
> During the test (I’ve changed 1..500 to 1..5000 to have more time to catch
> top output) top was showing on average:
> %Cpu1 : 7.0 us, 83.7 sy, 0.0 ni, 7.0 id, 0.0 wa, 0.0 hi, 0.0 si, 2.3
> st
>
>
> KVM virtual machine(1 CPU adding CPUs will not make difference in this case):
> # time for i in {1..500}; do dd if=/dev/zero of=/dev/shm/test bs=1M count=10
> ;echo interation $i done; done
> 10485760 bytes (10 MB) copied, 0.0524781 s, 200 MB/s
> interation 500 done
[]
> So it seems that when it is about only CPU the difference is very small or
> none. I am getting similar results if I do complicated equations with big
> numbers (virtual machines solves it almost in the same time as 1st level
> host).
>
> But when memory is involved, everything slows down drastically. I wonder what
> results you will get from the dd from /dev/zero to /dev/shm.
>
> And no, my system has plenty of memory, paging in z/VM is ZERO. Hardly
> anything runs on this LPAR.
> KVM host has plenty of real memory - 8G and the virtual machine is set to 4G.
> It still has few Gigs left. No swapping, nothing else runs here.
That certainly makes sense. Shared memory does use page protection for change
bit tracking. KVM does use page protection as well for dirty tracking of guests.
And if I remember correctly z/VM 6.3 also uses page table magic for
change/reference
tracking. This is all fine. Unless: z/VM provides shadow page tables for the
3rd
level guest backing of a 2nd level hypervisor. It therefore needs to
emulate/trap
most page table operations and page faults, which can be pretty expensive.
Christian
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
You have picked some details that has not been needed in z/VM environments.
Since you normally have one or perhaps two systems, where should you move it
and why ?
Recently we switched some network hardware here and one router failed
connections due to some bit's
Wrongly set, making network gone for 70 seconds.
VMware server went down, z/VM just notified with a message that we have a not
responding network here and I will try the failover card. And later just
notified, ahh you are back again, welcome.
So not really sleeping :-)
Tore Agblad
zOpen, IT Services
Volvo Group Headquarters
Corporate Process & IT
SE-405 08, Gothenburg Sweden
E-mail: tore.agb...@volvo.com
http://www.volvo.com/volvoit/global/en-gb/
-Original Message-
From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Sergey
Korzhevsky
Sent: den 28 september 2015 8:25
To: LINUX-390@VM.MARIST.EDU
Subject: Re: Documentation for Linux on z Systems and KVM - new
Alan Altmark wrote:
>> What is it about z that makes virtualization work better?
>50 years of work on it?
That is interesting answer. One thing came to my mind is the live guest
relocation. As far as i could find, VMware introduced that feature
(vMotion) in 2003, z/VM - in 2011. The same regarding network
(GuestLAN/VSwitch).
So, looks like z/VM slept all that years and was wake up by x86 world
recently.
WBR, Sergey
Alan Altmark
Sent by: Linux on 390 Port
24-09-15 20:04
Please respond to Linux on 390 Port
To: LINUX-390@VM.MARIST.EDU
cc:
Subject: Re: Documentation for Linux on z Systems and KVM -
new
On Thursday, 09/24/2015 at 10:38 EDT, Rick Troth wrote:
> In the broader Linux market, z is alien; z/VM more-so. But now there's
> zKVM. But "we" know the value of z/VM. Perhaps zKVM beats xKVM (does
> it), but z/VM beats zKVM with one arm tied and one leg hamstrung. How
> the [expletive deleted] are we supposed to communicate the strengths of
> z/VM now that (some) customers have zKVM?
>
Sometimes it's not about how many or how fast, but whether you have enough
and they are fast enough to make it financially worthwhile. (This is all
about money, right?)
For those who find z/VM too alien, but who want what z can bring to the
table, zKVM is there. Dislike or fear of z/VM is no longer an inhibitor
to those who want Linux on z using a hypervisor that they already know.
Now, once they have what they want, it becomes the usual case of
run-evaluate-run-evaluate-repeat. Nothing new here in that respect. If
they find that there are things z/VM can do that zKVM cannot, then there
are decisions to be made. Emotional and technical. Is what you give up
from zKVM going to be worth what you get with z/VM?
Conversely, that same is question might be asked by existing z/VM
customers who are near that line between the two. If you manage your
system via QUERY, INDICATE, and DIRECTXA, and the Linux people won't sit
with you at lunch or slash your tires, then maybe, just maybe, zKVM should
be on your radar.
> What is it about z that makes virtualization work better?
50 years of work on it?
> * zKVM obviously means "KVM for IBM z Systems" and is presumed not a
> brand IBM blesses in any way.
zKVM is the product "short name" IBM uses for KVM for IBM z Systems.
Alan Altmark
Senior Managing z/VM and Linux Consultant
Lab Services System z Delivery Practice
IBM Systems & Technology Group
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
Alan Altmark wrote: >> What is it about z that makes virtualization work better? >50 years of work on it? That is interesting answer. One thing came to my mind is the live guest relocation. As far as i could find, VMware introduced that feature (vMotion) in 2003, z/VM - in 2011. The same regarding network (GuestLAN/VSwitch). So, looks like z/VM slept all that years and was wake up by x86 world recently. WBR, Sergey Alan Altmark Sent by: Linux on 390 Port 24-09-15 20:04 Please respond to Linux on 390 Port To: LINUX-390@VM.MARIST.EDU cc: Subject: Re: Documentation for Linux on z Systems and KVM - new On Thursday, 09/24/2015 at 10:38 EDT, Rick Troth wrote: > In the broader Linux market, z is alien; z/VM more-so. But now there's > zKVM. But "we" know the value of z/VM. Perhaps zKVM beats xKVM (does > it), but z/VM beats zKVM with one arm tied and one leg hamstrung. How > the [expletive deleted] are we supposed to communicate the strengths of > z/VM now that (some) customers have zKVM? > Sometimes it's not about how many or how fast, but whether you have enough and they are fast enough to make it financially worthwhile. (This is all about money, right?) For those who find z/VM too alien, but who want what z can bring to the table, zKVM is there. Dislike or fear of z/VM is no longer an inhibitor to those who want Linux on z using a hypervisor that they already know. Now, once they have what they want, it becomes the usual case of run-evaluate-run-evaluate-repeat. Nothing new here in that respect. If they find that there are things z/VM can do that zKVM cannot, then there are decisions to be made. Emotional and technical. Is what you give up from zKVM going to be worth what you get with z/VM? Conversely, that same is question might be asked by existing z/VM customers who are near that line between the two. If you manage your system via QUERY, INDICATE, and DIRECTXA, and the Linux people won't sit with you at lunch or slash your tires, then maybe, just maybe, zKVM should be on your radar. > What is it about z that makes virtualization work better? 50 years of work on it? > * zKVM obviously means "KVM for IBM z Systems" and is presumed not a > brand IBM blesses in any way. zKVM is the product "short name" IBM uses for KVM for IBM z Systems. Alan Altmark Senior Managing z/VM and Linux Consultant Lab Services System z Delivery Practice IBM Systems & Technology Group ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
Hi, I am not sure if I understand the offer for KVM from IBM. I know that it is available as a preview in SLES12 But what is: - 5648-KVM KVM for IBM z Systems V1.1 As far as I can tell, it is something that can be ordered from IBM catalog. But what is it? Did IBM came out with their own linux distro with KVM preinstalled on it (is it kind of like xcat?). Does it come on DVD or can be downloaded? How much different is it from KVM on SLES? Thank you Gregory > On Sep 22, 2015, at 8:22 AM, Dorothea Matthaeus wrote: > > Check out the new Linux on z Systems base technology publications on the > IBM Knowledge Center and on developerWorks: > > KVM Virtual Server Quick Start > KVM Virtual Server Management > Device Drivers, Features, and Commands for Linux as a KVM Guest > Installing SUSE Linux Enterprise Server 12 as a KVM Guest > > See: > > > http://www.ibm.com/support/knowledgecenter/linuxonibm/liaaf/lnz_r_kvm_base.html > > > http://www.ibm.com/developerworks/linux/linux390/documentation_dev.html > > > Dorothea Matthaeus > Linux on z Systems Information Development > IBM Deutschland Research and Development GmbH > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
On Thursday, 09/24/2015 at 10:38 EDT, Rick Troth wrote: > In the broader Linux market, z is alien; z/VM more-so. But now there's > zKVM. But "we" know the value of z/VM. Perhaps zKVM beats xKVM (does > it), but z/VM beats zKVM with one arm tied and one leg hamstrung. How > the [expletive deleted] are we supposed to communicate the strengths of > z/VM now that (some) customers have zKVM? > Sometimes it's not about how many or how fast, but whether you have enough and they are fast enough to make it financially worthwhile. (This is all about money, right?) For those who find z/VM too alien, but who want what z can bring to the table, zKVM is there. Dislike or fear of z/VM is no longer an inhibitor to those who want Linux on z using a hypervisor that they already know. Now, once they have what they want, it becomes the usual case of run-evaluate-run-evaluate-repeat. Nothing new here in that respect. If they find that there are things z/VM can do that zKVM cannot, then there are decisions to be made. Emotional and technical. Is what you give up from zKVM going to be worth what you get with z/VM? Conversely, that same is question might be asked by existing z/VM customers who are near that line between the two. If you manage your system via QUERY, INDICATE, and DIRECTXA, and the Linux people won't sit with you at lunch or slash your tires, then maybe, just maybe, zKVM should be on your radar. > What is it about z that makes virtualization work better? 50 years of work on it? > * zKVM obviously means "KVM for IBM z Systems" and is presumed not a > brand IBM blesses in any way. zKVM is the product "short name" IBM uses for KVM for IBM z Systems. Alan Altmark Senior Managing z/VM and Linux Consultant Lab Services System z Delivery Practice IBM Systems & Technology Group ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
>>> On 9/24/2015 at 10:36 AM, Rick Troth wrote: > * zKVM obviously means "KVM for IBM z Systems" and is presumed not a > brand IBM blesses in any way. I note a few instances of "zKVM" in the System Administration Guide that IBM published. (SC27-8237-00) Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
>>> On 9/24/2015 at 04:56 AM, Christian Borntraeger >>> wrote: > Am 24.09.2015 um 01:08 schrieb Mark Post: -snip- >> Those "8,000 virtual machines > on a z13" quotes I keep seeing from IBM are > all talking >> about z/VM, even though they never come right out and say it. I would > expect far less >> capacity from KVM at this point in its development. > > We have improved KVM since SLES12 was released. Some interesting > reading/watching on I have no doubt that KVM has been improved. > the progress of KVM and things we fixed in the past 2 years can be found > here. > > slides: > https://drive.google.com/file/d/0B6HTUUWSPdd-QlBta2ZEOWhQRlU/view?usp=sharing > presentation: > https://www.youtube.com/watch?v=cj-HLi1q6ZI So, a presentation on getting 1,000 KVM guests to run on a z13. I stand by my statement that you should expect far less capacity from KVM _at_this_point_in_its_development. I'm not saying it can't be made better. I believe it can, and will be. I'm just saying that IBM Marketing's failure to note that z/VM is required (today) to hit that 8,000 guest limit is irresponsible, if not outright intentionally deceptive. After all, they've been told enough times that they should. I did it myself at the SHARE General Session in Seattle. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
On 09/24/2015 04:56 AM, Christian Borntraeger wrote: > If you already have KVM on x86 and no z/VM in the house, > KVM on z might be the right tool to integrate the mainframe > in your open environment because it might integrate better > into the operational model that is already deployed. This brings a vital question closer to the surface: why z? As it stands, zKVM (see below) might actually do more than xKVM. But if you can't measure it some of us are hard pressed to see a business case. z is expensive with a capital x. Since Linux was ported to S/390, the VM/ESA community backed it heartily ... and had solid numbers to prove that mainframe Linux is truly cost effective. The hard part has been getting "Unix people" past the foreignness factor. Even when they don't have the emotional baggage of green screen imagery, there are still operational challenges: there is no standard protocol for driving hypervisors. In the broader Linux market, z is alien; z/VM more-so. But now there's zKVM. But "we" know the value of z/VM. Perhaps zKVM beats xKVM (does it), but z/VM beats zKVM with one arm tied and one leg hamstrung. How the [expletive deleted] are we supposed to communicate the strengths of z/VM now that (some) customers have zKVM? What is it about z that makes virtualization work better? If KVM on z does not wow customers compared to KVM on x, they're just not interested. On the performance front, z/VM wins over zKVM. How and for how long? There are also several crucial features of z/VM which KVM lacks. I won't enumerate them just now. We're talking about things which could be taken up by KVM (even if some remain specific to KVM on z). * zKVM obviously means "KVM for IBM z Systems" and is presumed not a brand IBM blesses in any way. If there are no other claims, I donate it to the public domain for purpose of shortening "KVM for IBM z Systems" to a more conversable term. Edit a bit. -- R; <>< -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
On Thursday, 09/24/2015 at 03:53 EDT, Christian Borntraeger
wrote:
> z/VM does call SIE on behalf of the guest hypervisor. So for CPU bound
> workload, which causes almost no SIE exits things are fine. It is the
sweet spot for 2nd level. As
> soon as the KVM guest will have many exits (e.g. some I/O, memory
fault-in, reschedule) or more
> than one CPU > in the KVM host things can get really slow as z/VM then
has to interpret lots
> of things.
>
> In addition z/VM 2nd level support was in no way optimized to speed up a
KVM
> hypervisor, so
> I assume that some of the optimizations for z/VM under z/VM have to fall
back
> to the slow path.
To the best of my knowledge, there are no optimizations for for 2nd level
z/VM, since "SIE is SIE". (SIE is the name of the instruction that
implements the Interpretive Execution Facility.)
I call this the "SIE Instruction Pancake Effect". Each SIE that issued
in a vertical software stack will "pancake" down until there is hardware
to run it. But as it pancakes, the "sphere" of memory and CPU resources
around the SIE instruction gets smaller and smaller. If the host reaches
outside the sphere, the instruction ends ("SIE exit").
If the sphere was small to begin with, you will see the least effect. If
the sphere is large, then the Pancake Effect is very noticeable as SIE
repeatedly exits, taking many more virtual SIE instructions to accomplish
the same thing as would a single real SIE instruction.
Alan Altmark
Senior Managing z/VM and Linux Consultant
Lab Services System z Delivery Practice
IBM Systems & Technology Group
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
Am 24.09.2015 um 03:34 schrieb Grzegorz Powiedziuk:
> That’s very interesting. I wasn’t aware of this “stress” tool. So I’ve
> downloaded it and run a couple tests with it.
> If I run basic —cpu 1 test (-n according to help is a dry run), the KVM
> server spins the CPU 100% in user time. So no stealing at all.
>
> Could you run a couple of tests like this (I am providing my own results):
>
> KVM server (2 CPU but it is one threaded task so doesn’t matter how many)
Does changing the KVM server to 1 CPU makes things better?
Having more than one CPU makes it harder for z/VM to virtualize
KVMs usage of SIE.
> # time for i in {1..500}; do dd if=/dev/zero of=/dev/shm/test bs=1M count=10
> ;echo interation $i done; done
> ….
> 10485760 bytes (10 MB) copied, 0.00273469 s, 3.8 GB/s
> interation 500 done
>
> real 0m2.223s
> user 0m0.171s
> sys 0m2.002s
>
> During the test (I’ve changed 1..500 to 1..5000 to have more time to catch
> top output) top was showing on average:
> %Cpu1 : 7.0 us, 83.7 sy, 0.0 ni, 7.0 id, 0.0 wa, 0.0 hi, 0.0 si, 2.3
> st
>
>
> KVM virtual machine(1 CPU adding CPUs will not make difference in this case):
> # time for i in {1..500}; do dd if=/dev/zero of=/dev/shm/test bs=1M count=10
> ;echo interation $i done; done
> 10485760 bytes (10 MB) copied, 0.0524781 s, 200 MB/s
> interation 500 done
[]
> So it seems that when it is about only CPU the difference is very small or
> none. I am getting similar results if I do complicated equations with big
> numbers (virtual machines solves it almost in the same time as 1st level
> host).
>
> But when memory is involved, everything slows down drastically. I wonder what
> results you will get from the dd from /dev/zero to /dev/shm.
>
> And no, my system has plenty of memory, paging in z/VM is ZERO. Hardly
> anything runs on this LPAR.
> KVM host has plenty of real memory - 8G and the virtual machine is set to 4G.
> It still has few Gigs left. No swapping, nothing else runs here.
That certainly makes sense. Shared memory does use page protection for change
bit tracking. KVM does use page protection as well for dirty tracking of guests.
And if I remember correctly z/VM 6.3 also uses page table magic for
change/reference
tracking. This is all fine. Unless: z/VM provides shadow page tables for the
3rd
level guest backing of a 2nd level hypervisor. It therefore needs to
emulate/trap
most page table operations and page faults, which can be pretty expensive.
Christian
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
Am 24.09.2015 um 01:08 schrieb Mark Post: On 9/23/2015 at 05:57 PM, Grzegorz Powiedziuk wrote: >> As long as KVM can get close to z/vm performance then I >> see a great potential in it. > > I don't think you're going to see that for quite a while. I think we need to add some additional information. All the announcement, GA, support for SLES12 as guest talking from IBM and SUSE is not about the tech preview in SLES12. It is about the IBM product http://www-03.ibm.com/systems/z/solutions/virtualization/kvm/ together with the upcoming SLES12SP1 as a guest. How much this differs from LPAR or z/VM or the KVM version in SLES12 certainly depends on your workload. As always, pre-planning and testing is always a good idea. As of today the sweet spot of KVM on z is certainly a different one: KVM provides a different scheme of management and has different ways of doing things. If you already have z/VM then you very likely want to stay there as your operational model obviously works fine and its proven and reliable in your shop. If you already have KVM on x86 and no z/VM in the house, KVM on z might be the right tool to integrate the mainframe in your open environment because it might integrate better into the operational model that is already deployed. > Those "8,000 virtual machines > on a z13" quotes I keep seeing from IBM are > all talking > about z/VM, even though they never come right out and say it. I would expect > far less > capacity from KVM at this point in its development. We have improved KVM since SLES12 was released. Some interesting reading/watching on the progress of KVM and things we fixed in the past 2 years can be found here. slides: https://drive.google.com/file/d/0B6HTUUWSPdd-QlBta2ZEOWhQRlU/view?usp=sharing presentation: https://www.youtube.com/watch?v=cj-HLi1q6ZI Christian -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
Am 24.09.2015 um 00:42 schrieb Viktor Mihajlovski: > On 23.09.2015 15:32, Grzegorz Powiedziuk wrote: >> BTW, I was playing with KVM few days ago and it looks pretty awesome in >> terms of maintaining the environment and deploying new VMs but the >> performance for me was really bad. >> And I mean extremely bad. I am not sure if it was because I made the KVM >> host (sles12) run as a virtual machine in z/VM or I was doing something >> else wrong. I know that having kvm virtual machines in a 3rd level (under >> sles -> under z/VM) will impact performance but my case it was extremly bad. >> It was like running linux in hercules s390 in 2006 on old x86 desktop. >> >> The installation of linux in kvm virtual machine took 3-4 hours. Every >> operation that involves cpu and memory takes 3-10 time more time than on a >> KVM host itself. >> Whenever something is happening in kvm virtual machine, the performance >> toolkit shows that KVM host is doing about 50% in supervisor mode and 50% in >> emulation mode which makes the t/v ratio for this machine about 2 which is >> pretty bad. I didn’t have time to do more investigation on this yet. >> The KVM host (Server) sees about 50% cpu time as a “steal time”. >> > > Although KVM should generally be run in LPAR a slowdown in an order of > magnitude in z/VM seems a bit odd. > I actually do run KVM under z/VM at my own risk (I am absolutely NOT > suggesting to do that). To recreate your problems I started a CPU burning > task in a KVM guest (stress -n 1) and see the following in the host: > > Tasks: 162 total, 1 running, 161 sleeping, 0 stopped, 0 zombie > %Cpu(s): 32,5 us, 0,7 sy, 0,0 ni, 63,8 id, 0,0 wa, 0,0 hi, 0,1 si, 2,9 > st > KiB Mem:812852 total, 801232 used,11620 free, 3152 buffers > KiB Swap: 7212140 total, 1789268 used, 5422872 free, 611460 cached > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 38763 qemu 20 0 2842280 353068 350952 S 94,8 43,4 58:46.16 > qemu-system-s39 > > As you can see it's less than 3% steal time in the host with roughly 95% > problem state CPU utilization for the guest process (which indicates that SIE > isn't too bad even for a 3rd level guest). > > The high steal time you observe could be a hint for either z/VM swapping on > behalf of the KVM host or KVM swapping itself. Could you observe a high page > rate? > z/VM does call SIE on behalf of the guest hypervisor. So for CPU bound workload, which causes almost no SIE exits things are fine. It is the sweet spot for 2nd level. As soon as the KVM guest will have many exits (e.g. some I/O, memory fault-in, reschedule) or more than one CPU in the KVM host things can get really slow as z/VM then has to interpret lots of things. In addition z/VM 2nd level support was in no way optimized to speed up a KVM hypervisor, so I assume that some of the optimizations for z/VM under z/VM have to fall back to the slow path. Christian -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
That’s very interesting. I wasn’t aware of this “stress” tool. So I’ve
downloaded it and run a couple tests with it.
If I run basic —cpu 1 test (-n according to help is a dry run), the KVM server
spins the CPU 100% in user time. So no stealing at all.
Could you run a couple of tests like this (I am providing my own results):
KVM server (2 CPU but it is one threaded task so doesn’t matter how many)
# time for i in {1..500}; do dd if=/dev/zero of=/dev/shm/test bs=1M count=10
;echo interation $i done; done
….
10485760 bytes (10 MB) copied, 0.00273469 s, 3.8 GB/s
interation 500 done
real0m2.223s
user0m0.171s
sys 0m2.002s
During the test (I’ve changed 1..500 to 1..5000 to have more time to catch top
output) top was showing on average:
%Cpu1 : 7.0 us, 83.7 sy, 0.0 ni, 7.0 id, 0.0 wa, 0.0 hi, 0.0 si, 2.3 st
KVM virtual machine(1 CPU adding CPUs will not make difference in this case):
# time for i in {1..500}; do dd if=/dev/zero of=/dev/shm/test bs=1M count=10
;echo interation $i done; done
10485760 bytes (10 MB) copied, 0.0524781 s, 200 MB/s
interation 500 done
real0m38.556s
user0m1.180s
sys 0m11.550s
During the test (no need to do more than 1..500 becasue there is enough time to
check top) the KVM host was showing:
%Cpu0 : 53.7 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.7 hi, 0.0 si, 45.6 st
And KVM virtual machine itself was showing
%Cpu(s): 3.8 us, 30.1 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 66.2 st
Now pure CPU stress tests
KVM Host:
# time for i in {1..10}; do dd if=/dev/urandom of=/dev/null bs=1M count=10
;echo interation $i done; done
0485760 bytes (10 MB) copied, 0.630299 s, 16.6 MB/s
interation 10 done
real0m6.294s
user0m0.004s
sys 0m6.273s
top output on KVM host
%Cpu1 : 0.0 us,100.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KVM virtual machine:
# # time for i in {1..10}; do dd if=/dev/urandom of=/dev/null bs=1M count=10
;echo interation $i done; done
10485760 bytes (10 MB) copied, 0.64323 s, 16.3 MB/s
interation 10 done
real0m6.779s
user0m0.014s
sys 0m6.345s
top output on KVM host
%Cpu1 : 96.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 4.0 st
top output on KVM virtual machine
0.6 us, 85.4 sy, 0.0 ni, 7.9 id, 0.0 wa, 0.0 hi, 0.0 si, 6.0 st
So it seems that when it is about only CPU the difference is very small or
none. I am getting similar results if I do complicated equations with big
numbers (virtual machines solves it almost in the same time as 1st level host).
But when memory is involved, everything slows down drastically. I wonder what
results you will get from the dd from /dev/zero to /dev/shm.
And no, my system has plenty of memory, paging in z/VM is ZERO. Hardly anything
runs on this LPAR.
KVM host has plenty of real memory - 8G and the virtual machine is set to 4G.
It still has few Gigs left. No swapping, nothing else runs here.
Another huge difference in times is …. starting yast. On 1st level host it
takes fractures of a second. On the KVM virtual machine it takes >1min
Best regards
Gregory P
> On Sep 23, 2015, at 6:42 PM, Viktor Mihajlovski
> wrote:
>
> On 23.09.2015 15:32, Grzegorz Powiedziuk wrote:
>> BTW, I was playing with KVM few days ago and it looks pretty awesome in
>> terms of maintaining the environment and deploying new VMs but the
>> performance for me was really bad.
>> And I mean extremely bad. I am not sure if it was because I made the KVM
>> host (sles12) run as a virtual machine in z/VM or I was doing something
>> else wrong. I know that having kvm virtual machines in a 3rd level (under
>> sles -> under z/VM) will impact performance but my case it was extremly bad.
>> It was like running linux in hercules s390 in 2006 on old x86 desktop.
>>
>> The installation of linux in kvm virtual machine took 3-4 hours. Every
>> operation that involves cpu and memory takes 3-10 time more time than on a
>> KVM host itself.
>> Whenever something is happening in kvm virtual machine, the performance
>> toolkit shows that KVM host is doing about 50% in supervisor mode and 50% in
>> emulation mode which makes the t/v ratio for this machine about 2 which is
>> pretty bad. I didn’t have time to do more investigation on this yet.
>> The KVM host (Server) sees about 50% cpu time as a “steal time”.
>>
>
> Although KVM should generally be run in LPAR a slowdown in an order of
> magnitude in z/VM seems a bit odd.
> I actually do run KVM under z/VM at my own risk (I am absolutely NOT
> suggesting to do that). To recreate your problems I started a CPU burning
> task in a KVM guest (stress -n 1) and see the following in the host:
>
> Tasks: 162 total, 1 running, 161 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 32,5 us, 0,7 sy, 0,0 ni, 63,8 id, 0,0 wa, 0,0 hi, 0,1 si, 2,9
> st
> KiB Mem:812852 total, 801232 used,11620 free, 3152 buffers
> KiB Swap: 7212140 total, 1789268 used, 5422872 free, 611460
Re: Documentation for Linux on z Systems and KVM - new
>>> On 9/23/2015 at 05:57 PM, Grzegorz Powiedziuk >>> wrote: > As long as KVM can get close to z/vm performance then I > see a great potential in it. I don't think you're going to see that for quite a while. Those "8,000 virtual machines on a z13" quotes I keep seeing from IBM are all talking about z/VM, even though they never come right out and say it. I would expect far less capacity from KVM at this point in its development. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
On 23.09.2015 15:32, Grzegorz Powiedziuk wrote: BTW, I was playing with KVM few days ago and it looks pretty awesome in terms of maintaining the environment and deploying new VMs but the performance for me was really bad. And I mean extremely bad. I am not sure if it was because I made the KVM host (sles12) run as a virtual machine in z/VM or I was doing something else wrong. I know that having kvm virtual machines in a 3rd level (under sles -> under z/VM) will impact performance but my case it was extremly bad. It was like running linux in hercules s390 in 2006 on old x86 desktop. The installation of linux in kvm virtual machine took 3-4 hours. Every operation that involves cpu and memory takes 3-10 time more time than on a KVM host itself. Whenever something is happening in kvm virtual machine, the performance toolkit shows that KVM host is doing about 50% in supervisor mode and 50% in emulation mode which makes the t/v ratio for this machine about 2 which is pretty bad. I didn’t have time to do more investigation on this yet. The KVM host (Server) sees about 50% cpu time as a “steal time”. Although KVM should generally be run in LPAR a slowdown in an order of magnitude in z/VM seems a bit odd. I actually do run KVM under z/VM at my own risk (I am absolutely NOT suggesting to do that). To recreate your problems I started a CPU burning task in a KVM guest (stress -n 1) and see the following in the host: Tasks: 162 total, 1 running, 161 sleeping, 0 stopped, 0 zombie %Cpu(s): 32,5 us, 0,7 sy, 0,0 ni, 63,8 id, 0,0 wa, 0,0 hi, 0,1 si, 2,9 st KiB Mem:812852 total, 801232 used,11620 free, 3152 buffers KiB Swap: 7212140 total, 1789268 used, 5422872 free, 611460 cached PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 38763 qemu 20 0 2842280 353068 350952 S 94,8 43,4 58:46.16 qemu-system-s39 As you can see it's less than 3% steal time in the host with roughly 95% problem state CPU utilization for the guest process (which indicates that SIE isn't too bad even for a 3rd level guest). The high steal time you observe could be a hint for either z/VM swapping on behalf of the KVM host or KVM swapping itself. Could you observe a high page rate? -- Mit freundlichen Grüßen/Kind Regards Viktor Mihajlovski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Köderitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
Ok, thanks for the explanation. I knew about missing SIE from one of the SHARE presentations but I didn't know that the performance downgrade will be so huge. I will try to get a free LPAR just for KVM at some point but for now I will keep learning and exploring its features in z/vm. So far I really like it. Installation was a breeze ... besides the fact that I needed to get and install demo of SLES because Red Hat is again was one step behind. As long as KVM can get close to z/vm performance then I see a great potential in it. Gregory P 2015-09-23 15:43 GMT-04:00 Neale Ferguson : > KVM under z/VM will suck because the hardware only supports two levels of > “SIE”. SIE is whats used to allow an LPAR and a virtual machine to operate > at hardware speeds. A lot of the stuff that used to be done by VM/SP and > predecessors when running virtual machines is done by the hardware (well > the microcode/millicode/…). > > A virtual machine that tries to dispatch a guest of its own on SIE (like > KVM running on z/VM) has to get all those operations performed by the > hypervisor and not the hardware. Thus you get an enormous overhead and the > performance you are experiencing. > > On 9/23/15, 3:32 PM, "Linux on 390 Port on behalf of Grzegorz Powiedziuk" > wrote: > > >BTW, I was playing with KVM few days ago and it looks pretty awesome in > >terms of maintaining the environment and deploying new VMs but the > >performance for me was really bad. > >And I mean extremely bad. I am not sure if it was because I made the KVM > >host (sles12) run as a virtual machine in z/VM or I was doing something > >else wrong. I know that having kvm virtual machines in a 3rd level (under > >sles -> under z/VM) will impact performance but my case it was extremly > >bad. It was like running linux in hercules s390 in 2006 on old x86 > >desktop. > > > >The installation of linux in kvm virtual machine took 3-4 hours. Every > >operation that involves cpu and memory takes 3-10 time more time than on > >a KVM host itself. > >Whenever something is happening in kvm virtual machine, the performance > >toolkit shows that KVM host is doing about 50% in supervisor mode and 50% > >in emulation mode which makes the t/v ratio for this machine about 2 > >which is pretty bad. I didn’t have time to do more investigation on this > >yet. > >The KVM host (Server) sees about 50% cpu time as a “steal time”. > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
KVM under z/VM will suck because the hardware only supports two levels of “SIE”. SIE is whats used to allow an LPAR and a virtual machine to operate at hardware speeds. A lot of the stuff that used to be done by VM/SP and predecessors when running virtual machines is done by the hardware (well the microcode/millicode/…). A virtual machine that tries to dispatch a guest of its own on SIE (like KVM running on z/VM) has to get all those operations performed by the hypervisor and not the hardware. Thus you get an enormous overhead and the performance you are experiencing. On 9/23/15, 3:32 PM, "Linux on 390 Port on behalf of Grzegorz Powiedziuk" wrote: >BTW, I was playing with KVM few days ago and it looks pretty awesome in >terms of maintaining the environment and deploying new VMs but the >performance for me was really bad. >And I mean extremely bad. I am not sure if it was because I made the KVM >host (sles12) run as a virtual machine in z/VM or I was doing something >else wrong. I know that having kvm virtual machines in a 3rd level (under >sles -> under z/VM) will impact performance but my case it was extremly >bad. It was like running linux in hercules s390 in 2006 on old x86 >desktop. > >The installation of linux in kvm virtual machine took 3-4 hours. Every >operation that involves cpu and memory takes 3-10 time more time than on >a KVM host itself. >Whenever something is happening in kvm virtual machine, the performance >toolkit shows that KVM host is doing about 50% in supervisor mode and 50% >in emulation mode which makes the t/v ratio for this machine about 2 >which is pretty bad. I didn’t have time to do more investigation on this >yet. >The KVM host (Server) sees about 50% cpu time as a “steal time”. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
>>> On 9/23/2015 at 03:32 PM, Grzegorz Powiedziuk >>> wrote: > BTW, I was playing with KVM few days ago and it looks pretty awesome in terms > of maintaining the environment and deploying new VMs but the performance for > me was really bad. > And I mean extremely bad. I am not sure if it was because I made the KVM > host (sles12) run as a virtual machine in z/VM or I was doing something else > wrong. Almost certainly the fact that you installed the KVM host in a z/VM guest, and not in an LPAR. Doing that means that the KVM host doesn't have access to SIE to accelerate things. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Documentation for Linux on z Systems and KVM - new
BTW, I was playing with KVM few days ago and it looks pretty awesome in terms of maintaining the environment and deploying new VMs but the performance for me was really bad. And I mean extremely bad. I am not sure if it was because I made the KVM host (sles12) run as a virtual machine in z/VM or I was doing something else wrong. I know that having kvm virtual machines in a 3rd level (under sles -> under z/VM) will impact performance but my case it was extremly bad. It was like running linux in hercules s390 in 2006 on old x86 desktop. The installation of linux in kvm virtual machine took 3-4 hours. Every operation that involves cpu and memory takes 3-10 time more time than on a KVM host itself. Whenever something is happening in kvm virtual machine, the performance toolkit shows that KVM host is doing about 50% in supervisor mode and 50% in emulation mode which makes the t/v ratio for this machine about 2 which is pretty bad. I didn’t have time to do more investigation on this yet. The KVM host (Server) sees about 50% cpu time as a “steal time”. Anyone else played with it? Thanks Gregory Powiedziuk > On Sep 22, 2015, at 8:22 AM, Dorothea Matthaeus wrote: > > Check out the new Linux on z Systems base technology publications on the > IBM Knowledge Center and on developerWorks: > > KVM Virtual Server Quick Start > KVM Virtual Server Management > Device Drivers, Features, and Commands for Linux as a KVM Guest > Installing SUSE Linux Enterprise Server 12 as a KVM Guest > > See: > > > http://www.ibm.com/support/knowledgecenter/linuxonibm/liaaf/lnz_r_kvm_base.html > > > http://www.ibm.com/developerworks/linux/linux390/documentation_dev.html > > > Dorothea Matthaeus > Linux on z Systems Information Development > IBM Deutschland Research and Development GmbH > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
