Dismal performance on a 16G memory/8 core server - my.cnf settings?
Hello, We recently purchased a Dell PowerEdge 6650 thinking it would be a real fast server. Specs are: OS: Linux Debian 4.0/Etch RAID 5 on 4x U320 15k rpm drives (uses a perc-raid 3/DC hardware raid controller) 16GB of RAM 4 3.0 Ghz Xeon processors - I think they're dual core, in /proc/cpuinfo it shows up as 8 processors - maybe it's only HT I first made the mistake of using the default kernel, which provides SMP support but not large memory support. I have the output of a mysql sql-bench run from mysql on a Mac Mini to compare performance with. The server was only 0.35 (relative) the speed of the Mac mini - that means an 8 core 3.0 Ghz Xeon server with 16GB of RAM was only about 3x as fast as a as a single-core 1.25 Ghz G4 with 1GB of RAM (and a mini uses those little laptop hard drives, too). Needless to say my employer was shocked at the terrible performance and decided to sell the 6650 right away. But I can't help but wonder if there's not something terribly wrong with the settings - either the OS or mysql settings. I changed the kernel to the -bigmem kernel. It now sees all the RAM, but the sql-bench output on this try was _exactly_ the same: 0.35 I copied the my-huge.cnf from the examples directory and changed the thread_concurrency setting to 8 (because it said to set it to No. of CPUs*2). I also set the tmpdir, basedir, datadir and language, which were set in the original my.cnf I ran sql-bench again and the performance was even worse this time: 0.36 Someone suggested I try the -amd64 kernels which provide 64 bit but when I try to boot it I get various errors about this CPU does not support long (something) please use a 32-bit OS - the 64 bit install CD says the same message. So I assume these are not 64 bit CPUs. Any idea how I can configure this server to maximize performace? I think the multiple CPUs are a waste: I'm not looking for lots of concurrency, I want 1 query done really fast. Thanks. JW -- -- System Administrator - Cedar Creek Software http://www.cedarcreeksoftware.com -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Dismal performance on a 16G memory/8 core server - my.cnf settings?
Hi, On Wed, Apr 23, 2008 at 11:07 PM, JW [EMAIL PROTECTED] wrote: Hello, We recently purchased a Dell PowerEdge 6650 thinking it would be a real fast server. Specs are: OS: Linux Debian 4.0/Etch RAID 5 on 4x U320 15k rpm drives (uses a perc-raid 3/DC hardware raid controller) 16GB of RAM 4 3.0 Ghz Xeon processors - I think they're dual core, in /proc/cpuinfo it shows up as 8 processors - maybe it's only HT I first made the mistake of using the default kernel, which provides SMP support but not large memory support. I have the output of a mysql sql-bench run from mysql on a Mac Mini to compare performance with. The server was only 0.35 (relative) the speed of the Mac mini - that means an 8 core 3.0 Ghz Xeon server with 16GB of RAM was only about 3x as fast as a as a single-core 1.25 Ghz G4 with 1GB of RAM (and a mini uses those little laptop hard drives, too). Needless to say my employer was shocked at the terrible performance and decided to sell the 6650 right away. But I can't help but wonder if there's not something terribly wrong with the settings - either the OS or mysql settings. I changed the kernel to the -bigmem kernel. It now sees all the RAM, but the sql-bench output on this try was _exactly_ the same: 0.35 I copied the my-huge.cnf from the examples directory and changed the thread_concurrency setting to 8 (because it said to set it to No. of CPUs*2). I also set the tmpdir, basedir, datadir and language, which were set in the original my.cnf I ran sql-bench again and the performance was even worse this time: 0.36 Someone suggested I try the -amd64 kernels which provide 64 bit but when I try to boot it I get various errors about this CPU does not support long (something) please use a 32-bit OS - the 64 bit install CD says the same message. So I assume these are not 64 bit CPUs. They almost certainly are. Look at the contents of /proc/cpuinfo. You are probably using a 32-bit OS. You can't use a lot of memory efficiently unless you install a 64-bit OS, regardless of whether it has big memory support. But that's an x86_64 OS, not an AMD64 OS. These are not the same architecture. Any idea how I can configure this server to maximize performace? I think the multiple CPUs are a waste: I'm not looking for lots of concurrency, I want 1 query done really fast. You will be bound by CPU performance on any given single query, yes. But properly tuned, you may get a lot more performance out of this machine. Have you tuned MySQL (key_buffer_size and/or innodb_buffer_pool_size) to use the added memory, for starters? How much data do you even have? If your data all fits in the mac mini's memory and it has a comparable CPU and bus, I wouldn't be surprised to see it keeping up with the Dell fairly well on this benchmark. More to the point: does the benchmark reflect your real-life workload? Baron -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Dismal performance on a 16G memory/8 core server - my.cnf settings?
Hi, Someone suggested I try the -amd64 kernels which provide 64 bit but when I try to boot it I get various errors about this CPU does not support long (something) please use a 32-bit OS - the 64 bit install CD says the same message. So I assume these are not 64 bit CPUs. They almost certainly are. Look at the contents of /proc/cpuinfo. You are probably using a 32-bit OS. You can't use a lot of memory efficiently unless you install a 64-bit OS, regardless of whether it has big memory support. But that's an x86_64 OS, not an AMD64 OS. These are not the same architecture. Er, since he's talking about a 6650, a 6th generation Dell machine, it very likely *does* have 32-bit CPUs. And he's said it came with the PERC 3/DC card, which is a very old RAID card. I would hope this machine didn't cost much, as it's quite old. Besides that, though, x86_64 is exactly the amd64 architecture. AMD came up with it, Linux called it amd64, and then when Intel copied it and called it EM64T, it was renamed in Linux to x86_64 to be more generic. The above message is the exact one you get when you try to boot an x86_64 kernel on a 32-bit CPU. Regards, Jeremy -- high performance mysql consulting www.provenscaling.com -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Dismal performance on a 16G memory/8 core server - my.cnf settings?
Hi, On Thu, Apr 24, 2008 at 12:20 PM, Jeremy Cole [EMAIL PROTECTED] wrote: Hi, Someone suggested I try the -amd64 kernels which provide 64 bit but when I try to boot it I get various errors about this CPU does not support long (something) please use a 32-bit OS - the 64 bit install CD says the same message. So I assume these are not 64 bit CPUs. They almost certainly are. Look at the contents of /proc/cpuinfo. You are probably using a 32-bit OS. You can't use a lot of memory efficiently unless you install a 64-bit OS, regardless of whether it has big memory support. But that's an x86_64 OS, not an AMD64 OS. These are not the same architecture. Er, since he's talking about a 6650, a 6th generation Dell machine, it very likely *does* have 32-bit CPUs. And he's said it came with the PERC 3/DC card, which is a very old RAID card. I would hope this machine didn't cost much, as it's quite old. When you get old like me you won't remember every machine model :-) I didn't think they'd made 32-bit Xeons for a long time. (But I guess that's your point). Besides that, though, x86_64 is exactly the amd64 architecture. AMD came up with it, Linux called it amd64, and then when Intel copied it and called it EM64T, it was renamed in Linux to x86_64 to be more generic. And this one gets me every time. OK, sorry for the wrong advice JW! You can probably sort all this out on your own now. Like I said, /proc/cpuinfo. Baron -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Dismal performance on a 16G memory/8 core server - my.cnf settings?
I think what we need to know is more stuff about the database itself. How big is it? I assume if you were able to serve it from a Mac Mini it can't be all that big. 16GB should be big enough to contain all the data and serve it up quickly. And while 4-disk RAID 5 isn't all that great, it's certainly better than a single ATA disk in Mac Mini. If your database is huge, then 16GB is barely enough to hold innodb buffer pool (if innodb) and maybe the indexes. Then your RAID config will come into play (and maybe filesystem type, tuning). Then whether it's a Dell PE1750 or a 6650 would not make a huge difference since it's I/O bound. Were you testing with something like production data or just some test data? Have you modified my.cnf to reflect the new hardware config? Like Baron Schwartz asked, does your test reflect real-life workload? And yes, the 3.0Ghz Xeon processors you mentioned are 32-bit. You get that error message if your processor doesn't have EM64T capability. -Paul JW wrote: Hello, We recently purchased a Dell PowerEdge 6650 thinking it would be a real fast server. Specs are: OS: Linux Debian 4.0/Etch RAID 5 on 4x U320 15k rpm drives (uses a perc-raid 3/DC hardware raid controller) 16GB of RAM 4 3.0 Ghz Xeon processors - I think they're dual core, in /proc/cpuinfo it shows up as 8 processors - maybe it's only HT I first made the mistake of using the default kernel, which provides SMP support but not large memory support. I have the output of a mysql sql-bench run from mysql on a Mac Mini to compare performance with. The server was only 0.35 (relative) the speed of the Mac mini - that means an 8 core 3.0 Ghz Xeon server with 16GB of RAM was only about 3x as fast as a as a single-core 1.25 Ghz G4 with 1GB of RAM (and a mini uses those little laptop hard drives, too). Needless to say my employer was shocked at the terrible performance and decided to sell the 6650 right away. But I can't help but wonder if there's not something terribly wrong with the settings - either the OS or mysql settings. I changed the kernel to the -bigmem kernel. It now sees all the RAM, but the sql-bench output on this try was _exactly_ the same: 0.35 I copied the my-huge.cnf from the examples directory and changed the thread_concurrency setting to 8 (because it said to set it to No. of CPUs*2). I also set the tmpdir, basedir, datadir and language, which were set in the original my.cnf I ran sql-bench again and the performance was even worse this time: 0.36 Someone suggested I try the -amd64 kernels which provide 64 bit but when I try to boot it I get various errors about this CPU does not support long (something) please use a 32-bit OS - the 64 bit install CD says the same message. So I assume these are not 64 bit CPUs. Any idea how I can configure this server to maximize performace? I think the multiple CPUs are a waste: I'm not looking for lots of concurrency, I want 1 query done really fast. Thanks. JW -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
Re: Dismal performance on a 16G memory/8 core server - my.cnf settings?
JW wrote: Hello, We recently purchased a Dell PowerEdge 6650 thinking it would be a real fast server. Specs are: OS: Linux Debian 4.0/Etch RAID 5 on 4x U320 15k rpm drives (uses a perc-raid 3/DC hardware raid controller) 16GB of RAM 4 3.0 Ghz Xeon processors - I think they're dual core, in /proc/cpuinfo it shows up as 8 processors - maybe it's only HT snip This machine is not what is seems to be unfortunately. 1. Dell until their latest series with the Woodcrest based CPUs was a notoriously bad performer period. 2. Raid 5 + Database == run faster over serial cable. Use RAID 1+0. 3. Dell hasn't made a re-branded a decent RAID controller that is anything more than 18 months old. 4. 16GB doesn't help you because you are limited by your 32bitness. Get a real machine with real processors. 5. The CPU is hyper threaded, not dual core. I doubt you would ever get decent performance out of this for any database let alone MySQL. Tell your boss to suck it up, spend 5k. You can get a new, decent Dell for that. Or better yet, a HP. Sincerely, Joshua D. Drake -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/[EMAIL PROTECTED]
