Re: [PERFORM] Performance increase with elevator=deadline
Hi, Il giorno 11/apr/08, alle ore 20:03, Craig Ringer ha scritto: Speaking of I/O performance with PostgreSQL, has anybody here done any testing to compare results with LVM to results with the same filesystem on a conventionally partitioned or raw volume? I'd probably use LVM even at a performance cost because of its admin benefits, but I'd be curious if there is any known cost for use with Pg. I've never been able to measure one with other workloads. I performed some tests some time ago. LVM is significantly slower. The disk subsystem is a HP P400/512MB battery-backed controller with 4 disks in raid 10. See the tests: ext3 tests: bonnie++ -s 16000 -u 0 -f -b = = = = = = = Version 1.03 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP 16000M 153637 50 78895 17 204124 17 700.6 1 --Sequential Create-- Random Create -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2233 10 + +++ 2606 8 2255 10 + ++ + 2584 7 16000M,,,153637,50,78895,17,,,204124,17,700.6,1,16,2233,10,+,+++, 2606,8,2255,10,+,+++,2584,7 bonnie++ -s 16000 -u 0 -f = = = = = = = Version 1.03 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP 16000M 162223 51 77277 17 207055 17 765.3 1 --Sequential Create-- Random Create -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 + +++ + +++ + +++ + +++ + +++ + +++ 16000M,,,162223,51,77277,17,,,207055,17,765.3,1,16,+,+++,+,+++, +,+++,+,+++,+,+++,+,+++ = = = = = = = LVM tests: bonnie++ -u 0 -f -s 16000 -b Version 1.03 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP 16000M 153768 52 53420 13 177414 15 699.8 1 --Sequential Create-- Random Create -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2158 9 + +++ 2490 7 2177 9 + ++ + 2460 7 16000M,,,153768,52,53420,13,,,177414,15,699.8,1,16,2158,9,+,+++, 2490,7,2177,9,+,+++,2460,7 bonnie++ -u 0 -f -s 16000 Version 1.03 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP 16000M 161476 53 54904 13 171693 14 774.3 1 --Sequential Create-- Random Create -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 + +++ + +++ + +++ + +++ + +++ + +++ 16000M,,,161476,53,54904,13,,,171693,14,774.3,1,16,+,+++,+,+++, +,+++,+,+++,+,+++,+,+++ Bye, e. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Performance increase with elevator=deadline
Gregory Stark wrote: >> After some time of trial and error we found that changing the I/O scheduling >> algorithm to "deadline" improved I/O performance by a factor 4 (!) for >> this specific load test. > > What was the algorithm before? The default algorithm, CFQ I think it is. Yours, Laurenz Albe -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Performance increase with elevator=deadline
On Tue, 15 Apr 2008, Florian Weimer wrote: * Jeff: Using 4 of these with a dataset of about 30GB across a few files (Machine has 8GB mem) I went from around 100 io/sec to 330 changing to noop. Quite an improvement. If you have a decent controller CFQ is not what you want. I tried deadline as well and it was a touch slower. The controller is a 3ware 9550sx with 4 disks in a raid10. I'll be trying this out on the big array later today. I found it suprising this info wasn't more widespread (the use of CFQ on a good controller). 3ware might be a bit special because the controller has got very deep queues on its own, so many assumptions of the kernel I/O schedulers do not seem to apply. Toying with the kernel/controller queue depths might help, but I haven't done real benchmarks to see if it's actually a difference. A few days ago, I experienced this: On a machine with a 3ware controller, a simple getxattr call on a file in an uncontended directory took several minutes because a PostgreSQL dump process was running in the background (and some other activity of a legacy database which caused frequent fdatasync calls). Clearly, this is unacceptable, and I've since switched to the deadline scheduler, too. So far, this particular behavior hasn't occurred again. one other thing to watch out for. up until very recent kernels (2.6.23 or 2.6.24) it was possible for one very busy block device to starve other block devices. they added isolation of queues for different block devices, but I've seen reports that the isolation can end up throttling high performance devices as a result. I haven't had time to really dig into this, but there are tuning knobs available to adjust the que space available to different devices and the reports are significantly better activity on a tuned system. David Lang -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Performance increase with elevator=deadline
* Jeff: > Using 4 of these with a dataset of about 30GB across a few files > (Machine has 8GB mem) I went from around 100 io/sec to 330 changing to > noop. Quite an improvement. If you have a decent controller CFQ is > not what you want. I tried deadline as well and it was a touch > slower. The controller is a 3ware 9550sx with 4 disks in a raid10. > > I'll be trying this out on the big array later today. I found it > suprising this info wasn't more widespread (the use of CFQ on a good > controller). 3ware might be a bit special because the controller has got very deep queues on its own, so many assumptions of the kernel I/O schedulers do not seem to apply. Toying with the kernel/controller queue depths might help, but I haven't done real benchmarks to see if it's actually a difference. A few days ago, I experienced this: On a machine with a 3ware controller, a simple getxattr call on a file in an uncontended directory took several minutes because a PostgreSQL dump process was running in the background (and some other activity of a legacy database which caused frequent fdatasync calls). Clearly, this is unacceptable, and I've since switched to the deadline scheduler, too. So far, this particular behavior hasn't occurred again. -- Florian Weimer<[EMAIL PROTECTED]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Performance increase with elevator=deadline
On Sat, 12 Apr 2008, Craig Ringer wrote: Speaking of I/O performance with PostgreSQL, has anybody here done any testing to compare results with LVM to results with the same filesystem on a conventionally partitioned or raw volume? There was some chatter on this topic last year; a quick search finds http://archives.postgresql.org/pgsql-performance/2007-06/msg5.php which is a fair statement of the situation. I don't recall any specific benchmarks. -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Performance increase with elevator=deadline
Matthew wrote: On Fri, 11 Apr 2008, Jeff wrote: Using 4 of these with a dataset of about 30GB across a few files (Machine has 8GB mem) I went from around 100 io/sec to 330 changing to noop. Quite an improvement. If you have a decent controller CFQ is not what you want. I tried deadline as well and it was a touch slower. The controller is a 3ware 9550sx with 4 disks in a raid10. I ran Greg's fadvise test program a while back on a 12-disc array. The three schedulers (deadline, noop, anticipatory) all performed pretty-much the same, with the fourth (cfq, the default) being consistently slower. I use CFQ on some of my servers, despite the fact that it's often slower in total throughput terms, because it delivers much more predictable I/O latencies that help prevent light I/O processes being starved by heavy I/O processes. In particular, an Linux terminal server used at work has taken a lot of I/O tuning before it delivers even faintly acceptable I/O latencies under any sort of load. Bounded I/O latency at the expense of throughput is not what you usually want on a DB server, where throughput is king, so I'm not at all surprised that CFQ performs poorly for PostgreSQL. I've done no testing on that myself, though, because with my DB size and the nature of my queries most of them are CPU bound anyway. Speaking of I/O performance with PostgreSQL, has anybody here done any testing to compare results with LVM to results with the same filesystem on a conventionally partitioned or raw volume? I'd probably use LVM even at a performance cost because of its admin benefits, but I'd be curious if there is any known cost for use with Pg. I've never been able to measure one with other workloads. -- Craig Ringer -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Performance increase with elevator=deadline
On Fri, 11 Apr 2008, Jeff wrote: Using 4 of these with a dataset of about 30GB across a few files (Machine has 8GB mem) I went from around 100 io/sec to 330 changing to noop. Quite an improvement. If you have a decent controller CFQ is not what you want. I tried deadline as well and it was a touch slower. The controller is a 3ware 9550sx with 4 disks in a raid10. I ran Greg's fadvise test program a while back on a 12-disc array. The three schedulers (deadline, noop, anticipatory) all performed pretty-much the same, with the fourth (cfq, the default) being consistently slower. it also seems changing elevators on the fly works fine (echo schedulername > /sys/block/.../queue/scheduler I admit I sat there flipping back and forth going "disk go fast.. disk go slow.. disk go fast... " :) Oh Homer Simpson, your legacy lives on. Matthew -- I suppose some of you have done a Continuous Maths course. Yes? Continuous Maths? Whoah, it was like that, was it! -- Computer Science Lecturer -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Performance increase with elevator=deadline
On Apr 11, 2008, at 7:22 AM, Albe Laurenz wrote: After some time of trial and error we found that changing the I/O scheduling algorithm to "deadline" improved I/O performance by a factor 4 (!) for this specific load test. I was inspired once again to look into this - as I'm recently hitting some glass ceilings with my machines. I have a little app I wrote called pgiosim (its on pgfoundry - http://pgfoundry.org/projects/pgiosim) that basically just opens some files, and does random seeks and 8kB reads, much like what our beloved PG does. Using 4 of these with a dataset of about 30GB across a few files (Machine has 8GB mem) I went from around 100 io/sec to 330 changing to noop. Quite an improvement. If you have a decent controller CFQ is not what you want. I tried deadline as well and it was a touch slower. The controller is a 3ware 9550sx with 4 disks in a raid10. I'll be trying this out on the big array later today. I found it suprising this info wasn't more widespread (the use of CFQ on a good controller). it also seems changing elevators on the fly works fine (echo schedulername > /sys/block/.../queue/scheduler I admit I sat there flipping back and forth going "disk go fast.. disk go slow.. disk go fast... " :) -- Jeff Trout <[EMAIL PROTECTED]> http://www.stuarthamm.net/ http://www.dellsmartexitin.com/ -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Performance increase with elevator=deadline
"Albe Laurenz" <[EMAIL PROTECTED]> writes: > This refers to the performance problem reported in > http://archives.postgresql.org/pgsql-performance/2008-04/msg00052.php > > After some time of trial and error we found that changing the I/O scheduling > algorithm to "deadline" improved I/O performance by a factor 4 (!) for > this specific load test. What was the algorithm before? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
[PERFORM] Performance increase with elevator=deadline
This refers to the performance problem reported in http://archives.postgresql.org/pgsql-performance/2008-04/msg00052.php After some time of trial and error we found that changing the I/O scheduling algorithm to "deadline" improved I/O performance by a factor 4 (!) for this specific load test. It seems that the bottleneck in this case was actually in the Linux kernel. Since performance statements are useless without a description of the system and the type of load, I'll send a few details to make this report more useful for the archives: The machine is a PC with 8 AMD Opteron 885 CPUs and 32 GB RAM, attached to a HP EVA 8100 storage system with 72 disks. We are running 64-bit Linux 2.6.18-53.1.6.el5 from RedHat Enterprise 5.1. The I/O queue depth is set to 64. Our benchmark tools show a possible I/O performance of about 11000 transactions per second for random scattered reads of 8k blocks. PostgreSQL version is 8.2.4. The database system is a cluster with 6 databases containing tables up to a couple of GB in size. The whole database cluster takes about 200 GB of storage. The database load is a set of read-only statements, several of which have miserable execution plans and perform table and index scans. With the default I/O scheduler we observe a performance of about 600 I/O transactions or 7 MB per second. After booting with elevator=deadline both values increase by a factor of up to 4 and the query response times sink drastically. Yours, Laurenz Albe -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
