Re: [PERFORM] raid10 write performance
On 06/22/10 16:40, Greg Smith wrote: Grzegorz Jaśkiewicz wrote: raid: serveRAID M5014 SAS/SATA controller Do the performant servers have a different RAID card? This one has terrible performance, and could alone be the source of your issue. The ServeRAID cards are slow in general, and certainly slow running RAID10. What are some good RAID10 cards nowadays? On the other hand, RAID10 is simple enough that soft-RAID implementations should be more than adequate - any ideas why a dedicated card has it slow? -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
* Ivan Voras: On the other hand, RAID10 is simple enough that soft-RAID implementations should be more than adequate - any ideas why a dedicated card has it slow? Barrier support on RAID10 seems to require some smallish amount of non-volatile storage which supports a high number of write operations per second, so a software-only solution might not be available. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
On 06/23/10 14:00, Florian Weimer wrote: * Ivan Voras: On the other hand, RAID10 is simple enough that soft-RAID implementations should be more than adequate - any ideas why a dedicated card has it slow? Barrier support on RAID10 seems to require some smallish amount of non-volatile storage which supports a high number of write operations per second, so a software-only solution might not be available. If I understand you correctly, this can be said in general for all spinning-disk usage and is not specific to RAID10. (And in the case of high, constant TPS, no amount of NVRAM will help you). -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
On Wed, 23 Jun 2010, Ivan Voras wrote: On 06/23/10 14:00, Florian Weimer wrote: Barrier support on RAID10 seems to require some smallish amount of non-volatile storage which supports a high number of write operations per second, so a software-only solution might not be available. If I understand you correctly, this can be said in general for all spinning-disk usage and is not specific to RAID10. (And in the case of high, constant TPS, no amount of NVRAM will help you). No. Write barriers work fine with a single disc, assuming it is set up correctly. The barrier is a command telling the disc to make sure that one piece of data is safe before starting to write another piece of data. However, as soon as you have multiple discs, the individual discs do not have a way of communicating with each other to make sure that the first piece of data is written before the other. That's why you need a little bit of non-volatile storage to mediate that to properly support barriers. Of course, from a performance point of view, yes, you need some NVRAM on any kind of spinning storage to maintain high commit rates. Matthew -- I wouldn't be so paranoid if you weren't all out to get me!! -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
On Wed, Jun 23, 2010 at 8:25 AM, Ivan Voras ivo...@freebsd.org wrote: On 06/23/10 14:00, Florian Weimer wrote: * Ivan Voras: On the other hand, RAID10 is simple enough that soft-RAID implementations should be more than adequate - any ideas why a dedicated card has it slow? Barrier support on RAID10 seems to require some smallish amount of non-volatile storage which supports a high number of write operations per second, so a software-only solution might not be available. If I understand you correctly, this can be said in general for all spinning-disk usage and is not specific to RAID10. (And in the case of high, constant TPS, no amount of NVRAM will help you). Not entirely true. Let's say you have enough battery backed cache to hold 10,000 transaction writes in memory at once. The RAID controller can now re-order those writes so that they go from one side of the disk to the other, instead of randomly all over the place. That will most certainly help improve your throughput. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
On Wed, Jun 23, 2010 at 6:06 AM, Ivan Voras ivo...@freebsd.org wrote: On 06/22/10 16:40, Greg Smith wrote: Grzegorz Jaśkiewicz wrote: raid: serveRAID M5014 SAS/SATA controller Do the performant servers have a different RAID card? This one has terrible performance, and could alone be the source of your issue. The ServeRAID cards are slow in general, and certainly slow running RAID10. What are some good RAID10 cards nowadays? LSI, Areca, 3Ware (now LSI I believe) On the other hand, RAID10 is simple enough that soft-RAID implementations should be more than adequate - any ideas why a dedicated card has it slow? This is mostly a problem with some older cards that focused on RAID-5 performance, and RAID-10 was an afterthought. On many of these cards (older PERCs for instance) it was faster to either use a bunch of RAID-1 pairs in hardware with RAID-0 in software on top, or put the thing into JBOD mode and do it all in software. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
[PERFORM] raid10 write performance
Hi folks, is there a general problem with raid10 performance postgresql on it? We see very low performance on writes (2-3x slower than on less performant servers). I wonder if it is solely problem of raid10 configuration, or if it is postgresql's thing. Would moving WAL dir to separate disk help potentially ? We're running centos 5.4, and server config is: x3550 M2, xeon 4c e5530 2.4ghz , 6GB of ram disks: ibm 300gb 2.5 SAS raid: serveRAID M5014 SAS/SATA controller strip size is the default 128k thanks. -- GJ -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote: Hi folks, is there a general problem with raid10 performance postgresql on it? We see very low performance on writes (2-3x slower than on less performant servers). I wonder if it is solely problem of raid10 configuration, or if it is postgresql's thing. RAID 10 is the commonly suggested layout for DB's as its performance to redundancy is good. The question that begs to be ask is what is the IO layout on the other servers your comparing against. Would moving WAL dir to separate disk help potentially ? Yes it can have a big impact. http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm http://wiki.postgresql.org/wiki/Performance_Analysis_Tools http://wiki.postgresql.org/wiki/Performance_Optimization We're running centos 5.4, and server config is: x3550 M2, xeon 4c e5530 2.4ghz , 6GB of ram disks: ibm 300gb 2.5 SAS raid: serveRAID M5014 SAS/SATA controller strip size is the default 128k thanks. All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by our proprietary quotation system. Quotations received via any other form of communication will not be honored. CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other information proprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it addresses. If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified that any unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this e-mail immediately. Thank you. attachment: justin.vcf -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
Grzegorz Jaśkiewicz wrote: raid: serveRAID M5014 SAS/SATA controller Do the performant servers have a different RAID card? This one has terrible performance, and could alone be the source of your issue. The ServeRAID cards are slow in general, and certainly slow running RAID10. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
Justin Graf wrote: On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote: Would moving WAL dir to separate disk help potentially ? Yes it can have a big impact. WAL on a separate spindle will make a HUGE difference in performance. TPS rates frequently double OR BETTER with WAL on a dedicated spindle. Strongly recommended. Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk. Snapshotting the data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in an non-restoreable backup. The manual discusses this but it's easy to miss don't or you'll get a NASTY surprise if something goes wrong. -- Karl attachment: karl.vcf -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
Of course, no backup strategy is complete without testing a full restore onto bare hardware :-) On Tue, Jun 22, 2010 at 9:29 AM, Karl Denninger k...@denninger.net wrote: Justin Graf wrote: On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote: Would moving WAL dir to separate disk help potentially ? Yes it can have a big impact. WAL on a separate spindle will make a HUGE difference in performance. TPS rates frequently double OR BETTER with WAL on a dedicated spindle. Strongly recommended. Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk. Snapshotting the data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in an non-restoreable backup. The manual discusses this but it's easy to miss don't or you'll get a NASTY surprise if something goes wrong. -- Karl -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] raid10 write performance
On Jun 22, 2010, at 7:29 AM, Karl Denninger wrote: Justin Graf wrote: On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote: Would moving WAL dir to separate disk help potentially ? Yes it can have a big impact. WAL on a separate spindle will make a HUGE difference in performance. TPS rates frequently double OR BETTER with WAL on a dedicated spindle. Strongly recommended. Most of the performance increase on Linux, if your RAID card has a data-safe write-back cache (battery or solid state cache persistence in case of power failure), is a separate file system, not separate spindle. Especially if ext3 is used with the default ordered journal, it is absolutely caustic to performance to have WAL and data on the same file system. The whole 'separate spindle' thing is important if you don't have a good caching raid card, otherwise it doesn't matter that much. Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk. Snapshotting the data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in an non-restoreable backup. The manual discusses this but it's easy to miss don't or you'll get a NASTY surprise if something goes wrong. -- Karl karl.vcfATT1..txt -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
