Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-18 Thread Jignesh Shah
>              9.2 + DW patch >              --- >              FPW off  FPW on  DW on/FPW off >              CK on    CK on   CK on > one disk:     11078   10394    3296  [1G shared_buffers, 8G RAM] > sep log disk: 13605   12015    3412 > > one disk:      7731    66

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-17 Thread Greg Smith
On 01/17/2012 03:47 PM, Kevin Grittner wrote: I'm very curious about what impact DW would have on big servers with write-back cache that becomes saturated, like in Greg Smith's post here... My guess is that a percentage of the dbt-2 run results posted here are hitting that sort of problem. We

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-17 Thread Kevin Grittner
Dan Scales wrote: > The second set of numbers is for a hard disk with write cache > turned off, closer to internal hard disks of servers (people were > quite interested in that result). These runs are for 50-warehouse > 8-processor DBT2 60-minute run, with checkpoints every 5 minutes. > The RA

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-17 Thread Dan Scales
py to hear all comments/suggestions. Thanks, Dan - Original Message - From: "Dan Scales" To: "Heikki Linnakangas" Cc: "PG Hackers" , jks...@gmail.com, "David Fetter" Sent: Wednesday, January 11, 2012 1:25:21 PM Subject: Re: [HACKERS] [WIP] Do

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-12 Thread Simon Riggs
On Thu, Jan 12, 2012 at 12:09 AM, Tom Lane wrote: > Simon Riggs writes: >> On Wed, Jan 11, 2012 at 11:07 PM, Josh Berkus wrote: >>> On 1/11/12 1:25 PM, Dan Scales wrote: And just wanted to reiterate one other benefit of double writes -- it greatly reduces the size of the WAL logs. > >

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-11 Thread Aidan Van Dyk
On Wed, Jan 11, 2012 at 7:09 PM, Tom Lane wrote: > The question is how you prevent torn pages when a slave server crashes > during replay.  Right now, the presence of FPIs in the WAL stream, > together with the requirement that replay restart from a checkpoint, > is sufficient to guarantee that a

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-11 Thread Tom Lane
Simon Riggs writes: > On Wed, Jan 11, 2012 at 11:07 PM, Josh Berkus wrote: >> On 1/11/12 1:25 PM, Dan Scales wrote: >>> And just wanted to reiterate one other benefit of double writes -- it >>> greatly reduces the size of the WAL logs. >> Even if you're replicating? > Yes, but it will increase

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-11 Thread Simon Riggs
On Wed, Jan 11, 2012 at 11:07 PM, Josh Berkus wrote: > On 1/11/12 1:25 PM, Dan Scales wrote: >> And just wanted to reiterate one other benefit of double writes -- it >> greatly reduces the size of the WAL logs. > > Even if you're replicating? Yes, but it will increase random I/O on the standby w

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-11 Thread Josh Berkus
On 1/11/12 1:25 PM, Dan Scales wrote: > And just wanted to reiterate one other benefit of double writes -- it greatly > reduces the size of the WAL logs. Even if you're replicating? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hacke

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-11 Thread Dan Scales
...@gmail.com Sent: Wednesday, January 11, 2012 4:13:01 AM Subject: Re: [HACKERS] [WIP] Double-write with Fast Checksums On 10.01.2012 23:43, David Fetter wrote: > Please find attached a new revision of the double-write patch. While > this one still uses the checksums from VMware, it'

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-11 Thread Robert Haas
On Wed, Jan 11, 2012 at 9:47 AM, Aidan Van Dyk wrote: > It does this by moving the FPW/IO penalty from the commit time of a > backend dirtying the buffer first, to the eviction time of a backend > evicting a dirty buffer.  And if you're lucky enough that the > background writer is the only one wri

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-11 Thread Aidan Van Dyk
On Wed, Jan 11, 2012 at 7:13 AM, Heikki Linnakangas wrote: > At the moment, double-writes are done in one batch, fsyncing the > double-write area first and the data files immediately after that. That's > probably beneficial if you have a BBU, and/or a fairly large shared_buffers > setting, so tha

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-11 Thread Simon Riggs
On Wed, Jan 11, 2012 at 12:13 PM, Heikki Linnakangas wrote: > At the moment, double-writes are done in one batch, fsyncing the > double-write area first and the data files immediately after that. That's > probably beneficial if you have a BBU, and/or a fairly large shared_buffers > setting, so th

Re: [HACKERS] [WIP] Double-write with Fast Checksums

2012-01-11 Thread Heikki Linnakangas
On 10.01.2012 23:43, David Fetter wrote: Please find attached a new revision of the double-write patch. While this one still uses the checksums from VMware, it's been forward-ported to 9.2. I'd like to hold off on merging Simon's checksum patch into this one for now because there may be some in