Re: Memory limits for Scientific Linux kernels
On Fri, Jan 29, 2010 at 08:33:27AM +0100, Tim Edwards wrote: > > Unfortunately we have some internally-developed software which relies > strictly on a particular 'certified' OS version, architecture, package > versions etc. It takes a long time to get that re-certified for a newer > platform, especially an architecture change. Does the certification survive running the certified system on virtual hardware? (I would expect replacing real hardware with virtual hardware to to be equivalent to replacing an Intel NIC with a 3COM NIC, or an Intel mobo with an ASUS mobo, etc. If your certification is also tied to specific hardware, I so feel for you...) -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: Memory limits for Scientific Linux kernels
On Fri, Jan 29, 2010 at 01:59:13PM -0800, Akemi Yagi wrote: > > [from red hat bugzilla] ... Additionally, the Red > Hat Enterprise Linux 5 PAE variant does not allow 4G of addressable memory > per-process like the Red Hat Enterprise Linux 4 kernel-hugemem variant does. If I remember right, this is what happened: For a 32-bit OS, a user process can only address 4 GiBytes of memory. But some of this memory has to be reserved by the OS itself for sundry OS needs. Originally, Linux had it divided as 2 GiBi user + 2 GiBi system, and it did not matter for machines with only 2 GB of physical memory or less (right?). Then, as machines with more memory became more affordable and common, users wanted to use this memory, so the user part was pushed up to 3+1 split, then to 3.5+0.5 (I think). Then Red Hat had a special patch for Linux 2.4 kernels that permitted 4+0 split, but it was not accepted into the mainstream kernel (I guess 0 bytes for kernel use was too little) and when the OS switched to the 2.6 kernels with RHEL5, this custom modification disappeared. -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: Memory limits for Scientific Linux kernels
On Fri, Jan 29, 2010 at 9:25 AM, Connie Sieh wrote: > Yes Oracle gets the same rules as everyone else. Note that there is the > concept of HUGEPAGES which a application can use which may help with this , > not sure if Oracle uses that. I think there are some Oracle tech notes > that cover this topic. While we are still talking about the memory stuff ... there is an interesting issue about "4G of addressable memory per process" in the upstream bugzilla[1]. Here's an excerpt: "Physical Address Extension allows x86 processors to address up to 64GB of physical RAM, but due to differences between the Red Hat Enterprise Linux 4 and 5 kernels, only Red Hat Enterprise Linux 4 (with kernel-hugemem package) is able to reliably address all 64GB of memory. Additionally, the Red Hat Enterprise Linux 5 PAE variant does not allow 4G of addressable memory per-process like the Red Hat Enterprise Linux 4 kernel-hugemem variant does. However, the x86_64 kernel does not suffer from any of these limitations, and is the suggested Red Hat Enterprise Linux 5 architecture to use with large-memory systems." [1] https://bugzilla.redhat.com/show_bug.cgi?id=241314 Akemi
Re: Memory limits for Scientific Linux kernels
On Fri, 29 Jan 2010, Howard, Chris wrote: =20 Do these memory limitations also include shared memory segments? For example, we have Oracle database servers with quite a bit of memory tied up in shared memory segments for the databases. 4 Gig won't get me very far with a large database. Yes Oracle gets the same rules as everyone else. Note that there is the concept of HUGEPAGES which a application can use which may help with this , not sure if Oracle uses that. I think there are some Oracle tech notes that cover this topic. > =20 Chris Howard -Connie Sieh
RE: Memory limits for Scientific Linux kernels
Do these memory limitations also include shared memory segments? For example, we have Oracle database servers with quite a bit of memory tied up in shared memory segments for the databases. 4 Gig won't get me very far with a large database. Chris Howard
Re: Memory limits for Scientific Linux kernels
On 28/01/10 18:00, Stephan Wiesand wrote: > > On Jan 28, 2010, at 16:24 , Alan Bartlett wrote: > >> >> Perhaps I may be permitted to mention the CentOS product page -- >> >> http://www.centos.org/product.html >> >> As the objective of the CentOS Project is to be 100% binary compatible >> with TUV's product, the information on that page should have some >> relevance to SL users. :-) > > But then, it's not completely accurate either ;-) I don't see it mention the > largesmp and PAE kernels, for instance. > > Tim, I think the SL5/32 limit is 4 GB (3 GB usable) with the regular kernel, > 16 GB with the PAE one. I guess the PAE kernel doesn't get as much testing in > the field as x86_64. Just curious: why would you want to run a 32-bit OS on a > machine with that much RAM? > > > Regards, > Stephan > Unfortunately we have some internally-developed software which relies strictly on a particular 'certified' OS version, architecture, package versions etc. It takes a long time to get that re-certified for a newer platform, especially an architecture change. In the meantime we're seeing free reporting 32GB of RAM in an SL5.3 i386 Xen virtual machine that's running the normal SMP kernel. This is why I was a bit confused as I can see no evidence of a HUGEMEM kernel even existing for RHEL or SL 5.x. Here's the output: [r...@localhost ~]# lsb_release -a LSB Version: :core-3.0-ia32:core-3.0-noarch:graphics-3.0-ia32:graphics-3.0-noarch Distributor ID: ScientificSL Description:Scientific Linux SL release 4.3 (Beryllium) Release:4.3 Codename: Beryllium [r...@localhost ~]# free -m total used free sharedbuffers cached Mem: 32503 1405 31097 0230241 -/+ buffers/cache:933 31570 Swap: 1992 0 1992 [r...@localhost ~]# uname -a Linux localhost 2.6.9-89.0.19.ELsmp #1 SMP Fri Jan 8 04:31:36 CST 2010 i686 athlon i386 GNU/Linux Tim Edwards
Re: Memory limits for Scientific Linux kernels
> On Thu, Jan 28, 2010 at 9:00 AM, Stephan Wiesand > wrote: > > >> http://www.centos.org/product.html > > > But then, it's not completely accurate either ;-) I don't see it mention the largesmp and PAE kernels, for instance. > > Well, PAE is missing, but largesmp is mentioned in note (5) :-D > But, yes, that page needs updating. (7) The x86 "Hugemem" kernel is not provided in CentOS 5. I believe that covered the larger memory installations in releases prior to 5. Regards, Michael. > Akemi --- End of Original Message ---
Re: Memory limits for Scientific Linux kernels
Hi Konstantin, -- Original Message --- From: Konstantin Olchanski To: Tim Edwards Cc: "scientific-linux-us...@fnal.gov" Sent: Thu, 28 Jan 2010 13:38:59 -0800 Subject: Re: Memory limits for Scientific Linux kernels > On Thu, Jan 28, 2010 at 02:56:31PM +0100, Tim Edwards wrote: > > We're trying to work out memory limits for 32-bit versions of SL4 and 5. > > AFAIC, the effective hard limits for 32-bit Linux are 3 GBytes per user > process (they may have squeezed it up to 3.5 GBytes) and 4 GBytes > of physical RAM for the machine. > > If your machine has more than 4 Gbytes of physical RAM, you should > be running a 64-bit OS. In this case, 32-bit processes are still > limited to 3-3.5 Gbytes of memory, there is no way around that. > > The "by the specs" "paper" limits may say that 32-bit Linux kernels > can use more memory, and it might even work in practice, but be aware > that it requires that the Linux kernel use some page table magic > and relies on funny Intel CPU extenstions (PAE & co). As I > understand, this leads to noticable drop in performance. Just asking for some advice based on your comments above. I have several HP ProLiant DL360 G3 servers, their CPU's are 32 bit only (G4 and above are 64bit), as such I run 32 bit Linux on them. However, many of them have more than 4Gb of RAM (2x2Gb modules plus 2x512Mb is typical) and I use the kernel-PAE to access that extra RAM. The servers typically don't use more than 4Gb of RAM, actually, I hardly ever see them do, they typically access 2-3Gb for normal daily operations. Performance isn't an issue either however, are you saying in my situation I would be better off, performance wise, to just keep the 2x 2Gb DIMMs in them, yank the 2x 512Mb and revert back to the standard "kernel" RPM's without PAE to get a noticeable increase in performance? I wonder if there's some weblinks anywhere which could show performance graphs of with and without PAE's? Thanks. Michael. > I am now curious why are you interested in running 32-bit Linux > when 64-bit Linux was "invented" specifically to "fix" the > memory limits. > > K.O. > > > Redhat's page (http://www.redhat.com/rhel/compare/) says that the > > maximums are 64GB or RHEL4 and 16GB for RHEL5 (I guess because they > > dropped the HUGEMEM kernel RPM in RHEL5). > > > > This page (http://www.scientificlinux.org/documentation/misc/limits) > > says that it's 64GB in SL4 but gives no information for SL5. So two > > questions: > > > > Does SL4 i386 have a 'HUGEMEM' kernel build or do you just build those > > features into the normal -smp kernel build in order to support 64GB RAM? > > > > What is the memory limit on SL5 i386? > > > > Thanks > > > > Tim Edwards > > -- > Konstantin Olchanski > Data Acquisition Systems: The Bytes Must Flow! > Email: olchansk-at-triumf-dot-ca > Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada --- End of Original Message ---
Re: Memory limits for Scientific Linux kernels
On Thu, Jan 28, 2010 at 02:56:31PM +0100, Tim Edwards wrote: > We're trying to work out memory limits for 32-bit versions of SL4 and 5. AFAIC, the effective hard limits for 32-bit Linux are 3 GBytes per user process (they may have squeezed it up to 3.5 GBytes) and 4 GBytes of physical RAM for the machine. If your machine has more than 4 Gbytes of physical RAM, you should be running a 64-bit OS. In this case, 32-bit processes are still limited to 3-3.5 Gbytes of memory, there is no way around that. The "by the specs" "paper" limits may say that 32-bit Linux kernels can use more memory, and it might even work in practice, but be aware that it requires that the Linux kernel use some page table magic and relies on funny Intel CPU extenstions (PAE & co). As I understand, this leads to noticable drop in performance. I am now curious why are you interested in running 32-bit Linux when 64-bit Linux was "invented" specifically to "fix" the memory limits. K.O. > Redhat's page (http://www.redhat.com/rhel/compare/) says that the > maximums are 64GB or RHEL4 and 16GB for RHEL5 (I guess because they > dropped the HUGEMEM kernel RPM in RHEL5). > > This page (http://www.scientificlinux.org/documentation/misc/limits) > says that it's 64GB in SL4 but gives no information for SL5. So two > questions: > > Does SL4 i386 have a 'HUGEMEM' kernel build or do you just build those > features into the normal -smp kernel build in order to support 64GB RAM? > > What is the memory limit on SL5 i386? > > Thanks > > Tim Edwards -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: Memory limits for Scientific Linux kernels
On Thu, Jan 28, 2010 at 9:00 AM, Stephan Wiesand wrote: >> http://www.centos.org/product.html > But then, it's not completely accurate either ;-) I don't see it mention the > largesmp and PAE kernels, for instance. Well, PAE is missing, but largesmp is mentioned in note (5) :-D But, yes, that page needs updating. Akemi
Re: Memory limits for Scientific Linux kernels
Stephan Wiesand writes: > Just curious: why would you want to run a 32-bit OS on a machine with > that much RAM? For me, binary compatability with 32bit machines in the same computing cluster. Users keeping different programs compiled differently based on which box their job happens to run on is chaotic at best, although being careful with condor can help. -- Alec Habig, University of Minnesota Duluth Physics Dept. ha...@neutrino.d.umn.edu http://neutrino.d.umn.edu/~habig/
Re: Memory limits for Scientific Linux kernels
On Jan 28, 2010, at 16:24 , Alan Bartlett wrote: > On 28 January 2010 15:18, Troy Dawson wrote: >> Tim Edwards wrote: >>> >>> Does SL4 i386 have a 'HUGEMEM' kernel build or do you just build those >>> features into the normal -smp kernel build in order to support 64GB RAM? >>> >>> What is the memory limit on SL5 i386? > >> Wow, that SL page is old. Three years old. I need to update it and put >> caveat's on it. The problem is that I can't put a link to RedHat's page >> even though that is where I get the information. > > Perhaps I may be permitted to mention the CentOS product page -- > > http://www.centos.org/product.html > > As the objective of the CentOS Project is to be 100% binary compatible > with TUV's product, the information on that page should have some > relevance to SL users. :-) But then, it's not completely accurate either ;-) I don't see it mention the largesmp and PAE kernels, for instance. Tim, I think the SL5/32 limit is 4 GB (3 GB usable) with the regular kernel, 16 GB with the PAE one. I guess the PAE kernel doesn't get as much testing in the field as x86_64. Just curious: why would you want to run a 32-bit OS on a machine with that much RAM? Regards, Stephan -- Stephan Wiesand DESY -DV- Platanenenallee 6 15738 Zeuthen, Germany smime.p7s Description: S/MIME cryptographic signature
Re: Memory limits for Scientific Linux kernels
On Thu, Jan 28, 2010 at 7:18 AM, Troy Dawson wrote: > Tim Edwards wrote: >> This page (http://www.scientificlinux.org/documentation/misc/limits) >> says that it's 64GB in SL4 but gives no information for SL5. So two >> questions: >> >> Does SL4 i386 have a 'HUGEMEM' kernel build or do you just build those >> features into the normal -smp kernel build in order to support 64GB RAM? >> >> What is the memory limit on SL5 i386? > Wow, that SL page is old. Three years old. I need to update it and put > caveat's on it. The problem is that I can't put a link to RedHat's page > even though that is where I get the information. > > We use RedHat's kernel with no changes, compiling it to be compatible with > their kernel. So whatever RedHat has listed for limits is what we have for > limits. Same is true for CentOS kernels. So, this page may be useful: https://www.centos.org/product.html Akemi
Re: Memory limits for Scientific Linux kernels
On 28 January 2010 15:18, Troy Dawson wrote: > Tim Edwards wrote: >> This page (http://www.scientificlinux.org/documentation/misc/limits) >> says that it's 64GB in SL4 but gives no information for SL5. So two >> questions: >> >> Does SL4 i386 have a 'HUGEMEM' kernel build or do you just build those >> features into the normal -smp kernel build in order to support 64GB RAM? >> >> What is the memory limit on SL5 i386? > Wow, that SL page is old. Three years old. I need to update it and put > caveat's on it. The problem is that I can't put a link to RedHat's page > even though that is where I get the information. Perhaps I may be permitted to mention the CentOS product page -- http://www.centos.org/product.html As the objective of the CentOS Project is to be 100% binary compatible with TUV's product, the information on that page should have some relevance to SL users. :-) Alan.
Re: Memory limits for Scientific Linux kernels
Tim Edwards wrote: We're trying to work out memory limits for 32-bit versions of SL4 and 5. Redhat's page (http://www.redhat.com/rhel/compare/) says that the maximums are 64GB or RHEL4 and 16GB for RHEL5 (I guess because they dropped the HUGEMEM kernel RPM in RHEL5). This page (http://www.scientificlinux.org/documentation/misc/limits) says that it's 64GB in SL4 but gives no information for SL5. So two questions: Does SL4 i386 have a 'HUGEMEM' kernel build or do you just build those features into the normal -smp kernel build in order to support 64GB RAM? What is the memory limit on SL5 i386? Thanks Tim Edwards Wow, that SL page is old. Three years old. I need to update it and put caveat's on it. The problem is that I can't put a link to RedHat's page even though that is where I get the information. We use RedHat's kernel with no changes, compiling it to be compatible with their kernel. So whatever RedHat has listed for limits is what we have for limits. Troy -- __ Troy Dawson daw...@fnal.gov (630)840-6468 Fermilab ComputingDivision/LSCS/CSI/USS Group __
Memory limits for Scientific Linux kernels
We're trying to work out memory limits for 32-bit versions of SL4 and 5. Redhat's page (http://www.redhat.com/rhel/compare/) says that the maximums are 64GB or RHEL4 and 16GB for RHEL5 (I guess because they dropped the HUGEMEM kernel RPM in RHEL5). This page (http://www.scientificlinux.org/documentation/misc/limits) says that it's 64GB in SL4 but gives no information for SL5. So two questions: Does SL4 i386 have a 'HUGEMEM' kernel build or do you just build those features into the normal -smp kernel build in order to support 64GB RAM? What is the memory limit on SL5 i386? Thanks Tim Edwards
