Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-05-02 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: What we could have is the semantics of Return a buffer, with either correct contents or completely zeroed out. It would act just like ReadBuffer if the buffer was already in memory, and zero out the page otherwise. That's a bit strange semantics

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-28 Thread Simon Riggs
On Fri, 2007-04-27 at 12:22 +0100, Heikki Linnakangas wrote: Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: As regards the zero_damaged_pages question, I raised that some time ago but we didn't arrive at an explicit answer. All I would say is we can't allow invalid pages in the

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2007-04-27 at 12:22 +0100, Heikki Linnakangas wrote: Patch implementing that attached. I named the function ReadOrZeroBuffer. We already have an API quirk similar to this: relation extension. It seems strange to have two different kinds of special case API that are

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-28 Thread Heikki Linnakangas
I was actually thinking that we could slip this in 8.3. It's a simple, well-understood patch, which fixes a little data integrity quirk as well as gives a nice recovery speed up. Bruce Momjian wrote: I assume this is 8.4 material.

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-28 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: I was actually thinking that we could slip this in 8.3. It's a simple, well-understood patch, which fixes a little data integrity quirk as well as gives a nice recovery speed up. Yeah. It's arguably a bug fix, in fact, since it eliminates the

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Heikki Linnakangas
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: As regards the zero_damaged_pages question, I raised that some time ago but we didn't arrive at an explicit answer. All I would say is we can't allow invalid pages in the buffer manager at any time, whatever options we have requested,

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Alvaro Herrera
Heikki Linnakangas wrote: What we could have is the semantics of Return a buffer, with either correct contents or completely zeroed out. It would act just like ReadBuffer if the buffer was already in memory, and zero out the page otherwise. That's a bit strange semantics to have, but is

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Heikki Linnakangas
Alvaro Herrera wrote: Heikki Linnakangas wrote: What we could have is the semantics of Return a buffer, with either correct contents or completely zeroed out. It would act just like ReadBuffer if the buffer was already in memory, and zero out the page otherwise. That's a bit strange

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Alvaro Herrera
Heikki Linnakangas wrote: Alvaro Herrera wrote: Heikki Linnakangas wrote: What we could have is the semantics of Return a buffer, with either correct contents or completely zeroed out. It would act just like ReadBuffer if the buffer was already in memory, and zero out the page

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Bruce Momjian
I assume this is 8.4 material. --- Heikki Linnakangas wrote: Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: As regards the zero_damaged_pages question, I raised that some time ago but we didn't arrive at an

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Simon Riggs
On Fri, 2007-04-27 at 10:37 -0400, Bruce Momjian wrote: I assume this is 8.4 material. I think its a small enough, performance-only change to allow it at this time. It will provide considerable additional benefit for Warm Standby servers. -- Simon Riggs EnterpriseDB

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-26 Thread Jim Nasby
On Apr 25, 2007, at 2:48 PM, Heikki Linnakangas wrote: In recovery, with full_pages_writes=on, we read in each page only to overwrite the contents with a full page image. That's a waste of time, and can have a surprisingly large effect on recovery time. As a quick test on my laptop, I

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-26 Thread Zeugswetter Andreas ADI SD
So what happens if a backend is running with full_page_writes = off, someone edits postgresql.conf to turns it on and forgets to reload/ restart, and then we crash? You'll come up in recovery mode thinking that f_p_w was turned on, when in fact it wasn't. ISTM that we need to somehow

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-26 Thread Tom Lane
Jim Nasby [EMAIL PROTECTED] writes: So what happens if a backend is running with full_page_writes = off, someone edits postgresql.conf to turns it on and forgets to reload/ restart, and then we crash? You'll come up in recovery mode thinking that f_p_w was turned on, when in fact it

[HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Heikki Linnakangas
In recovery, with full_pages_writes=on, we read in each page only to overwrite the contents with a full page image. That's a waste of time, and can have a surprisingly large effect on recovery time. As a quick test on my laptop, I initialized a DBT-2 test with 5 warehouses, and let it run for

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Heikki Linnakangas
Heikki Linnakangas wrote: While working on this, this comment in ReadBuffer caught my eye: /* * During WAL recovery, the first access to any data page should * overwrite the whole page from the WAL; so a clobbered page * header is not reason to fail. Hence, when InRecovery

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Gregory Stark
Heikki Linnakangas [EMAIL PROTECTED] writes: While working on this, this comment in ReadBuffer caught my eye: /* * During WAL recovery, the first access to any data page should * overwrite the whole page from the WAL; so a clobbered page * header is not reason to

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Heikki Linnakangas
Gregory Stark wrote: So in short I think with your patch this piece of code no longer has a role. Either your patch kicks in and we never even look at the damaged page at all, or we should be treating it as corrupt data and just check zero_damaged_pages alone and not do anything special in

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Simon Riggs
On Wed, 2007-04-25 at 13:48 +0100, Heikki Linnakangas wrote: I was surprised how big a difference it makes, but when you think about it it's logical. Without the patch, it's doing roughly the same I/O as the test itself, reading in pages, modifying them, and writing them back. With the

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: As regards the zero_damaged_pages question, I raised that some time ago but we didn't arrive at an explicit answer. All I would say is we can't allow invalid pages in the buffer manager at any time, whatever options we have requested, otherwise other code

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Heikki Linnakangas
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: As regards the zero_damaged_pages question, I raised that some time ago but we didn't arrive at an explicit answer. All I would say is we can't allow invalid pages in the buffer manager at any time, whatever options we have requested,

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: Tom Lane wrote: but it'd make it safe to use in non-WAL contexts (I think there are other places where we know we are going to init the page and so a physical read is a waste of time). Is there? I can't think of any. Extending a relation doesn't