Re: Linux 2GB Memory Limit
On Tue, 13 Jul 2004 23:26:48 +0100, Marvin Wright [EMAIL PROTECTED] wrote: Hi, I'm now running redhat AS 3.0 with kernel version 2.4 and have 8GB of RAM. If I set my innodb_buffer_pool to 2048M, it just will not start, I get this error. 040713 22:10:24 mysqld started 040713 22:10:24 Warning: Asked for 196608 thread stack, but got 126976 InnoDB: Fatal error: cannot allocate 2147500032 bytes of InnoDB: memory with malloc! Total allocated memory Now I remember what I tracked down the limit to be ... 2147500032 is just above 2 gigabytes of memory. From what I have seen, glibc (not sure if this is fixed in recent versions) just refuses to allocate chunks of memory larger than 2 gigs in a single call. This seems a little odd given the library the malloc code is based on, but I haven't dug deeper. You can probably get around this if you do both of: 1. replace the call to malloc() in the innodb source with one that does a mmap() 2. run a kernel that has the 4G/4G patch, and possibly also moves where mmap()ed regions start to be a bit lower than 1 gig (not sure what the 4G/4G patch does with that). A pain in the ass. I strongly encourage people wanting larger innodb buffers to consider 64-bit Opterons or, less desirably, Intel's xeons w/64-bit support when they become generally available fairly soon. -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
RE: Linux 2GB Memory Limit
Hi, I'm now running redhat AS 3.0 with kernel version 2.4 and have 8GB of RAM. If I set my innodb_buffer_pool to 2048M, it just will not start, I get this error. 040713 22:10:24 mysqld started 040713 22:10:24 Warning: Asked for 196608 thread stack, but got 126976 InnoDB: Fatal error: cannot allocate 2147500032 bytes of InnoDB: memory with malloc! Total allocated memory InnoDB: by InnoDB 37066436 bytes. Operating system errno: 12 InnoDB: Cannot continue operation! InnoDB: Check if you should increase the swap file or InnoDB: ulimits of your operating system. InnoDB: On FreeBSD check you have compiled the OS with InnoDB: a big enough maximum process size. InnoDB: We now intentionally generate a seg fault so that InnoDB: on Linux we get a stack trace. mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=1044480 max_used_connections=0 max_connections=800 threads_connected=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1643385 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x83f4800 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Bogus stack limit or frame pointer, fp=0xbfffe368, stack_bottom=0x69726575, thread_stack=126976, aborting backtrace. Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd-query at 0x712d776f is invalid pointer thd-thread_id=925983092 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. 040713 22:10:24 mysqld ended My ulimit is ulimit -a core file size(blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) 4 max memory size (kbytes, -m) unlimited open files(-n) 1024 pipe size (512 bytes, -p) 8 stack size(kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes(-u) 7168 virtual memory(kbytes, -v) unlimited What do I have to do to the OS so that it will let me have an innodb buffer pool of 2GB ?? Regards, Marvin. -Original Message- From: Marc Slemko [mailto:[EMAIL PROTECTED] Sent: 09 July 2004 20:15 To: Marvin Wright Cc: [EMAIL PROTECTED] Subject: Re: Linux 2GB Memory Limit On Fri, 9 Jul 2004 15:46:37 +0100 , Marvin Wright [EMAIL PROTECTED] wrote: Hi, Current Platform RH version is 7.3 IBM Blade Server - 2 x Intel(R) Xeon(TM) CPU 3.20GHz 32 GB SCSI 4 GB Ram This is the platform we are moving to in a week or so RH Enterprise AS 2.1 or 3.0 4 x Intel(R) Xeon(TM) MP CPU 2.70GHz 128 GB SCSI Raid 16 GB Ram So with the new platform I'll be able to have a much bigger InnoDB buffer Note it will still be limited to something that is definitely no bigger than 4 gigs, and may be smaller... I haven't had any luck with ~2 gig innodb buffer sizes even on systems with 3 or 3.5 gigs of addess space available per process, but I never looked into that too deeply so it may work fine with the right setup. This is probably a bit late, but I would have definitely recommended running 64-bit opterons in your configuration since then you could have a larger innodb buffer. This e-mail has been scanned for all viruses by Star Internet. The service is powered by MessageLabs. For more information on a proactive anti-virus service working around the clock, around the globe, visit: http://www.star.net.uk ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ** This e-mail has been scanned for all viruses by Star Internet. The service is powered by MessageLabs. For more information on a proactive anti-virus service working around the clock, around the globe, visit: http
RE: Linux 2GB Memory Limit
What version are you using? What platform are you on? How old is your hardware? The 2Gb limit has long been addressed. RH9, Fedora, RHES all support more than 2Gb Ram (assuming Ram) out of the box... but its dependent on the kernel. Newer 2.4 uses a 3G/1G split to address the 4Gb it could handle. (3Gb user, 1Gb kernel). 2.6 uses 4G/4G split allowing a lot more to be used (64Gb with the hugemem kernels). This has been backported to the RHES 2.4.21 kernels. If you are running some stuff on larger amounts of memory in a production environment I'd start to look at the Enterprise distributions such as RHEL (or their F/OSS rebuild equivalents like TAO Linux and White box Linux). Kev -Original Message- From: Marvin Wright [mailto:[EMAIL PROTECTED] Sent: 09 July 2004 14:07 To: [EMAIL PROTECTED] Subject: Linux 2GB Memory Limit Hi, Is there any work around for this yet where a process can not allocate more than 2GB. Can I upgrade my Redhat OS to any particular version ? Many Thanks. Marvin Wright Flights Developer Lastminute.com [EMAIL PROTECTED] +44 (0) 207 802 4543 This e-mail has been scanned for all viruses by Star Internet. The service is powered by MessageLabs. For more information on a proactive anti-virus service working around the clock, around the globe, visit: http://www.star.net.uk -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED] ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ** -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
RE: Linux 2GB Memory Limit
Hi, Current Platform RH version is 7.3 IBM Blade Server - 2 x Intel(R) Xeon(TM) CPU 3.20GHz 32 GB SCSI 4 GB Ram This is the platform we are moving to in a week or so RH Enterprise AS 2.1 or 3.0 4 x Intel(R) Xeon(TM) MP CPU 2.70GHz 128 GB SCSI Raid 16 GB Ram So with the new platform I'll be able to have a much bigger InnoDB buffer pool. Thats good news. Cheers. Marvin. -Original Message- From: Kevin Jackson [mailto:[EMAIL PROTECTED] Sent: 09 July 2004 15:28 To: [EMAIL PROTECTED] Subject: RE: Linux 2GB Memory Limit What version are you using? What platform are you on? How old is your hardware? The 2Gb limit has long been addressed. RH9, Fedora, RHES all support more than 2Gb Ram (assuming Ram) out of the box... but its dependent on the kernel. Newer 2.4 uses a 3G/1G split to address the 4Gb it could handle. (3Gb user, 1Gb kernel). 2.6 uses 4G/4G split allowing a lot more to be used (64Gb with the hugemem kernels). This has been backported to the RHES 2.4.21 kernels. If you are running some stuff on larger amounts of memory in a production environment I'd start to look at the Enterprise distributions such as RHEL (or their F/OSS rebuild equivalents like TAO Linux and White box Linux). Kev -Original Message- From: Marvin Wright [mailto:[EMAIL PROTECTED] Sent: 09 July 2004 14:07 To: [EMAIL PROTECTED] Subject: Linux 2GB Memory Limit Hi, Is there any work around for this yet where a process can not allocate more than 2GB. Can I upgrade my Redhat OS to any particular version ? Many Thanks. Marvin Wright Flights Developer Lastminute.com [EMAIL PROTECTED] +44 (0) 207 802 4543 This e-mail has been scanned for all viruses by Star Internet. The service is powered by MessageLabs. For more information on a proactive anti-virus service working around the clock, around the globe, visit: http://www.star.net.uk -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED] ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ** -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED] This e-mail has been scanned for all viruses by Star Internet. The service is powered by MessageLabs. For more information on a proactive anti-virus service working around the clock, around the globe, visit: http://www.star.net.uk This e-mail has been scanned for all viruses by Star Internet. The service is powered by MessageLabs. For more information on a proactive anti-virus service working around the clock, around the globe, visit: http://www.star.net.uk -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
RE: Linux 2GB Memory Limit
Again, this isn't an issue in latest kernels for well over 3 years (2.4.0 Test7+) and uses LFS, though the filesystem implementation has been more recent but still a couple of years old). Ext3 supports this if you are this if you are looking at the Enterprise Linux kernels and ReiserFS also support this as do the other journaling filesystems. http://www.suse.de/~aj/linux_lfs.html for more info. Again, all modern distros support this feature - but they support it because the Linux kernel supports it, which you can make work with your existing distro version. Kev -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: 09 July 2004 15:38 To: Kevin Jackson Subject: RE: Linux 2GB Memory Limit He is talking about file size. --ja On Fri, 9 Jul 2004, Kevin Jackson wrote: What version are you using? What platform are you on? How old is your hardware? The 2Gb limit has long been addressed. RH9, Fedora, RHES all support more than 2Gb Ram (assuming Ram) out of the box... but its dependent on the kernel. Newer 2.4 uses a 3G/1G split to address the 4Gb it could handle. (3Gb user, 1Gb kernel). 2.6 uses 4G/4G split allowing a lot more to be used (64Gb with the hugemem kernels). This has been backported to the RHES 2.4.21 kernels. If you are running some stuff on larger amounts of memory in a production environment I'd start to look at the Enterprise distributions such as RHEL (or their F/OSS rebuild equivalents like TAO Linux and White box Linux). Kev -Original Message- From: Marvin Wright [mailto:[EMAIL PROTECTED] Sent: 09 July 2004 14:07 To: [EMAIL PROTECTED] Subject: Linux 2GB Memory Limit Hi, Is there any work around for this yet where a process can not allocate more than 2GB. Can I upgrade my Redhat OS to any particular version ? Many Thanks. Marvin Wright Flights Developer Lastminute.com [EMAIL PROTECTED] +44 (0) 207 802 4543 This e-mail has been scanned for all viruses by Star Internet. The service is powered by MessageLabs. For more information on a proactive anti-virus service working around the clock, around the globe, visit: http://www.star.net.uk -- ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ** -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Linux 2GB Memory Limit
On Fri, 9 Jul 2004 15:46:37 +0100 , Marvin Wright [EMAIL PROTECTED] wrote: Hi, Current Platform RH version is 7.3 IBM Blade Server - 2 x Intel(R) Xeon(TM) CPU 3.20GHz 32 GB SCSI 4 GB Ram This is the platform we are moving to in a week or so RH Enterprise AS 2.1 or 3.0 4 x Intel(R) Xeon(TM) MP CPU 2.70GHz 128 GB SCSI Raid 16 GB Ram So with the new platform I'll be able to have a much bigger InnoDB buffer Note it will still be limited to something that is definitely no bigger than 4 gigs, and may be smaller... I haven't had any luck with ~2 gig innodb buffer sizes even on systems with 3 or 3.5 gigs of addess space available per process, but I never looked into that too deeply so it may work fine with the right setup. This is probably a bit late, but I would have definitely recommended running 64-bit opterons in your configuration since then you could have a larger innodb buffer. -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
