Re: HugePage support with RHEL
On Fri, 2 Sep 2011 09:54:31 -0500 Stephen Frazier ste...@doc.state.ok.us wrote: On 9/2/2011 7:37 AM, Martin Schwidefsky wrote: There is no documentation for z/VM as the edat facility is not supported. The emulation for large pages takes place in the Linux guest. Looking at IBM past support for new hardware in guests of z/VM, I would suggest that it will be supported in z/VM 7.1 or before. Look for edat facility support to be added in a as yet unannounced point release of z/VM 6.?. Also, IBM probably will not talk about it until they are ready to announce it. This is harder than it looks. With z/VM you have two complete sets of page tables to worry about, the host page table and the guest page table. The host page table differs from the guest page table, each entry in the lowest level of the page table (the pte) has an extension (the pgste). Among other things you have the guest/host reference-backup-bit and the guest/host change-backup-bit in the pgste which are required to get paging right over the two levels of page tables. Now what happens if z/VM uses large pages in the host page table for the guest? The pgste table is tied to the pte table but that table does not exists if the segment table entry points to a 1MB frame. You loose access to the backup bits and your normal paging does not work anymore. The virtualization of the edat architecture is hard. -- blue skies, Martin. Reality continues to ruin my life. - Calvin. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: HugePage support with RHEL
On Fri, 2 Sep 2011 09:54:31 -0500 Stephen Frazier wrote: Looking at IBM past support for new hardware in guests of z/VM, I would suggest that it will be supported in z/VM 7.1 or before. Look for edat facility support to be added in a as yet unannounced point release of z/VM 6.?. Also, IBM probably will not talk about it until they are ready to announce it. Maybe it's my jaundiced eye, but I have a different perspective on this. IBM are the proprietary (monopoly ?) supplier of both the hardware and the software (in the case of z/VM). So you can't tell me they don't know what's in the future product. Why isn't support there day one ?. Look how long it took for CP 32-bit storage constraint relief to make it out the door - how about STP support ?. z/OS knows all about (non-virtualized) big page support - and Martin and his mates were able to fudge it *in the guest*. This is hipervisor responsibility. Maybe if/when live guest relocation makes it out the door, such features will suddenly become worthwhile investing developer time in. Shane ... -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: HugePage support with RHEL
On Thu, 1 Sep 2011 15:01:52 -0400 Brad Hinson bhin...@redhat.com wrote: Hi James, It appears that in order to use hardware large page support, Linux must be running in LPAR mode. I can't find anything that says this is supported in z/VM. Hopefully someone can correct me if I'm wrong. I can confirm that on a z10 under z/VM 6.1 I also do not see 'edat' in /proc/cpuinfo, so hugepage support is emulated in software. You can use large pages in LPAR and under z/VM. In LPAR we have real large pages if we have the edat facility. If there is no edat facility or if we are running under z/VM we use large page emulation. There are two benefits to using hugepages: 1) The TLB pressure in reduced by using 1MB frames. To get this benefit the edat facility is required since this needs the large page segment table entries. No love here for z/VM. 2) The memory savings due to the reduced number of page tables. There are two cases: 2a) under LPAR with edat the 1MB frames are directly referenced by the segment table entry, the lowest page table level is not allocated at all. 2b) under z/VM there is no edat facility and no large page segment table entries. Here a single page table for the 1MB frame is allocated which is shared by all users of the large page. The page table overhead to map 2GB of memory: i) without large pages: 1 segment table, 2048 page tables ii) with large page emulation: 1 segment table, 1 page table iii) with edat large pages: 1 segment table In numbers i) 4112 KB, ii) 18 KB, and iii) 16 KB. This number is per process. If your database uses processes for its transactions and maps large share memory areas the memory savings quickly add up. If you have e.g. 128 processes mapping 2GB you'll need for case i) 514 MB, ii) 2.25 MB, and iii) 2 MB. -- blue skies, Martin. Reality continues to ruin my life. - Calvin. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: HugePage support with RHEL
Thanks Martin, I almost gave up hope on HughPages due to all our zLinux Guests run under zVM, and we like using zVM to control the swapping as best as possible. I do have two follow-up questions: 1) Can you point me to any reference material dealing with HugePages with zVM (v6.1) where I can start my homework on the topic? I did a search online of IBM zVM 6.1 doc on HugePages, Large Pages and came up with nothing. 2) What is required or needs to be done to enable edat in zLinux from the zVM side? Again, would you be able to point me to any doc in the zVM side? How do we get the edat facility, is it hardware or a setting in either the SE or HMC with the LPAR definition, or in the IODF? James Chaplin Systems Programmer, MVS, zVM zLinux -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Martin Schwidefsky Sent: Friday, September 02, 2011 4:11 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: HugePage support with RHEL On Thu, 1 Sep 2011 15:01:52 -0400 Brad Hinson bhin...@redhat.com wrote: Hi James, It appears that in order to use hardware large page support, Linux must be running in LPAR mode. I can't find anything that says this is supported in z/VM. Hopefully someone can correct me if I'm wrong. I can confirm that on a z10 under z/VM 6.1 I also do not see 'edat' in /proc/cpuinfo, so hugepage support is emulated in software. You can use large pages in LPAR and under z/VM. In LPAR we have real large pages if we have the edat facility. If there is no edat facility or if we are running under z/VM we use large page emulation. There are two benefits to using hugepages: 1) The TLB pressure in reduced by using 1MB frames. To get this benefit the edat facility is required since this needs the large page segment table entries. No love here for z/VM. 2) The memory savings due to the reduced number of page tables. There are two cases: 2a) under LPAR with edat the 1MB frames are directly referenced by the segment table entry, the lowest page table level is not allocated at all. 2b) under z/VM there is no edat facility and no large page segment table entries. Here a single page table for the 1MB frame is allocated which is shared by all users of the large page. The page table overhead to map 2GB of memory: i) without large pages: 1 segment table, 2048 page tables ii) with large page emulation: 1 segment table, 1 page table iii) with edat large pages: 1 segment table In numbers i) 4112 KB, ii) 18 KB, and iii) 16 KB. This number is per process. If your database uses processes for its transactions and maps large share memory areas the memory savings quickly add up. If you have e.g. 128 processes mapping 2GB you'll need for case i) 514 MB, ii) 2.25 MB, and iii) 2 MB. -- blue skies, Martin. Reality continues to ruin my life. - Calvin. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: HugePage support with RHEL
On Fri, 2 Sep 2011 07:35:56 -0400 CHAPLIN, JAMES (CTR) james.chap...@associates.dhs.gov wrote: I almost gave up hope on HughPages due to all our zLinux Guests run under zVM, and we like using zVM to control the swapping as best as possible. I do have two follow-up questions: 1) Can you point me to any reference material dealing with HugePages with zVM (v6.1) where I can start my homework on the topic? I did a search online of IBM zVM 6.1 doc on HugePages, Large Pages and came up with nothing. There is no documentation for z/VM as the edat facility is not supported. The emulation for large pages takes place in the Linux guest. 2) What is required or needs to be done to enable edat in zLinux from the zVM side? Again, would you be able to point me to any doc in the zVM side? How do we get the edat facility, is it hardware or a setting in either the SE or HMC with the LPAR definition, or in the IODF? You cannot enable the edat facility under z/VM for a Linux guest (and you won't see edat in the cpu features) but you can still use large pages in the Linux guest. Just specify the hugepages=nr kernel parameter. The Linux kernel will automatically use the large page emulation if the edat facility is missing. You may want to look at the Large Page Support chapter in Linux on System z Device Drivers, Features and Commands: http://public.dhe.ibm.com/software/dw/linux390/docu/lk39dd11.pdf -- blue skies, Martin. Reality continues to ruin my life. - Calvin. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: HugePage support with RHEL
On 9/2/2011 7:37 AM, Martin Schwidefsky wrote: There is no documentation for z/VM as the edat facility is not supported. The emulation for large pages takes place in the Linux guest. Looking at IBM past support for new hardware in guests of z/VM, I would suggest that it will be supported in z/VM 7.1 or before. Look for edat facility support to be added in a as yet unannounced point release of z/VM 6.?. Also, IBM probably will not talk about it until they are ready to announce it. -- Stephen Frazier Information Technology Unit Oklahoma Department of Corrections 3400 Martin Luther King Oklahoma City, Ok, 73111-4298 Tel.: (405) 425-2549 Fax: (405) 425-2554 Mobile: (405) 464-7818 email: stevef%doc.state.ok.us -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
HugePage support with RHEL
I am currently looking at the HugePage support feature and whether it can help us at our site. But I hit a roadblock that someone out there may be able to help me with. In the doc I have found (both from RH and a collection of other presentations on the web), I have found how to set the /proc/sys/vm/nr_hugepages value with sysctl with no problem. But my concern is with the next bit of information concerning whether the hardware has large page support and if not, how do I enable that support. When I issue cat /proc/cpuinfo, we do not have the edat value set in features (page 26 in http://zjournal.tcipubs.com/issues/zJ.DEC-JAN09.pdf). We have two z196s with zVM 6.1 running zLinux guests using RHEL 5.6 (2.6.18-238.9.1.el5). Can anyone point me to how to get the feature edat turned on and where documentation on this may be located? James Chaplin Systems Programmer, MVS, zVM zLinux -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: HugePage support with RHEL
Hi James, It appears that in order to use hardware large page support, Linux must be running in LPAR mode. I can't find anything that says this is supported in z/VM. Hopefully someone can correct me if I'm wrong. I can confirm that on a z10 under z/VM 6.1 I also do not see 'edat' in /proc/cpuinfo, so hugepage support is emulated in software. See here: http://linuxvm.org/present/SHARE111/S9262ms.pdf (see slide 11) -- Brad Hinson bhin...@redhat.com Worldwide System z Sales, Strategy, Marketing Red Hat, Inc. +1 (919) 360-0443 http://www.redhat.com/z On Sep 1, 2011, at 11:31 AM, CHAPLIN, JAMES (CTR) wrote: I am currently looking at the HugePage support feature and whether it can help us at our site. But I hit a roadblock that someone out there may be able to help me with. In the doc I have found (both from RH and a collection of other presentations on the web), I have found how to set the /proc/sys/vm/nr_hugepages value with sysctl with no problem. But my concern is with the next bit of information concerning whether the hardware has large page support and if not, how do I enable that support. When I issue cat /proc/cpuinfo, we do not have the edat value set in features (page 26 in http://zjournal.tcipubs.com/issues/zJ.DEC-JAN09.pdf). We have two z196s with zVM 6.1 running zLinux guests using RHEL 5.6 (2.6.18-238.9.1.el5). Can anyone point me to how to get the feature edat turned on and where documentation on this may be located? James Chaplin Systems Programmer, MVS, zVM zLinux -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: HugePage support with RHEL
On Thu, Sep 01, 2011 at 03:01:52PM -0400, Brad Hinson wrote: It appears that in order to use hardware large page support, Linux must be running in LPAR mode. I can't find anything that says this is supported in z/VM. Hopefully someone can correct me if I'm wrong. I can confirm that on a z10 under z/VM 6.1 I also do not see 'edat' in /proc/cpuinfo, so hugepage support is emulated in software. And I can confirm that it's there on a z10 in LPAR mode. :) Kind regards, Philipp Kern -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
