Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2009-01-27 Thread Tom Lane
Heikki Linnakangas writes: > (replying to a very old message, since I just bumped into this in review) > Alvaro Herrera wrote: >> The new message is not translatable, the original was. > Doesn't really matter since it's an elog(), not ereport(). ... which is wrong in itself, since it's certainly

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2009-01-27 Thread Heikki Linnakangas
(replying to a very old message, since I just bumped into this in review) Alvaro Herrera wrote: Simon Riggs wrote: On Fri, 2008-09-12 at 14:14 -0400, Alvaro Herrera wrote: Simon Riggs wrote: --- 5716,5725 CheckpointStats.ckpt_sync_end_t,

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 18:19 +0300, Heikki Linnakangas wrote: > The bottom line is that hot standby is a big feature, and probably a big > patch. No amount of version control will work around that. Finishing all > that in a few weeks is a very ambitious goal. I wish you luck, and I > wish I cou

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Heikki Linnakangas
Alvaro Herrera wrote: Simon Riggs wrote: I'm just grumpy because I can't see a way to do the patch-on-patch-on-patch that I'll need to make this all work for Nov 1. So big patch here we come. But that's just the way it is and I'll stop honking about it. This is one of the problems that DVCSs a

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Dave Page
On Tue, Oct 7, 2008 at 4:08 PM, Simon Riggs <[EMAIL PROTECTED]> wrote: > > Not convinced this is the right time to invest in side activities, but > if you think so, I'll look into it. > > Anybody wanting to write or link to a Simon's Guide, most welcome. Heikki will be presenting a talk about GIT

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Tom Lane
"Robert Haas" <[EMAIL PROTECTED]> writes: >>> I'm just grumpy because I can't see a way to do the >>> patch-on-patch-on-patch that I'll need to make this all work for Nov 1. >>> So big patch here we come. But that's just the way it is and I'll stop >>> honking about it. >> >> This is one of the pr

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 10:37 -0400, Alvaro Herrera wrote: > Simon Riggs wrote: > > > I'm just grumpy because I can't see a way to do the > > patch-on-patch-on-patch that I'll need to make this all work for Nov 1. > > So big patch here we come. But that's just the way it is and I'll stop > > honkin

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Robert Haas
>> I'm just grumpy because I can't see a way to do the >> patch-on-patch-on-patch that I'll need to make this all work for Nov 1. >> So big patch here we come. But that's just the way it is and I'll stop >> honking about it. > > This is one of the problems that DVCSs are supposed to solve ... have

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Alvaro Herrera
Simon Riggs wrote: > I'm just grumpy because I can't see a way to do the > patch-on-patch-on-patch that I'll need to make this all work for Nov 1. > So big patch here we come. But that's just the way it is and I'll stop > honking about it. This is one of the problems that DVCSs are supposed to so

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 10:08 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > A patch specifically marked as "required for other work" has been > > delayed by more than 5 weeks on queue and nobody was ever assigned to > > review it. That was exactly the problem CommitFests were

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > A patch specifically marked as "required for other work" has been > delayed by more than 5 weeks on queue and nobody was ever assigned to > review it. That was exactly the problem CommitFests were supposed to > resolve and from my perspective this is a syst

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Thu, 2008-10-02 at 19:07 -0400, Andrew Dunstan wrote: > Simon Riggs wrote: > > Just seen this patch has been bounced into November CommitFest, even > > though the new patch fixes all of the concerns raised. > > > > I'm concerned that this is going to make the final Hot Standby patch > > fairl

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-02 Thread Andrew Dunstan
Simon Riggs wrote: On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: Simon Riggs <[EMAIL PROTECTED]> writes: Version 7 After reading this for awhile, I realized that there is a rather fundamental problem with it: it switches into "consistent recovery" mode as soon as it's rea

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-02 Thread Simon Riggs
On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > Version 7 > > After reading this for awhile, I realized that there is a rather > fundamental problem with it: it switches into "consistent recovery" > mode as soon as it's read WAL beyond ControlFile-

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 11:24 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote: > >> ... If we crash and restart, we'll have to get to the end > >> of this file before we start letting backends in; which might be further > >> than

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote: >> ... If we crash and restart, we'll have to get to the end >> of this file before we start letting backends in; which might be further >> than we actually got before the crash, but not too much further be

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > I think we can get away with writing the LSN value to disk, as you > > suggested, but only every so often. No need to do it after every WAL > > record, just consistently every so often, so it gives us

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > I think we can get away with writing the LSN value to disk, as you > suggested, but only every so often. No need to do it after every WAL > record, just consistently every so often, so it gives us a point at > which we know we are safe. Huh? How does that

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 08:46 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > ... That kinda works, but the problem is that restartpoints are time based, > > not log based. We need them to be deterministic for us to rely upon them > > in the above way. > > Right, but the perfor

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > ... That kinda works, but the problem is that restartpoints are time based, > not log based. We need them to be deterministic for us to rely upon them > in the above way. Right, but the performance disadvantages of making them strictly log-distance-based a

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Sun, 2008-09-28 at 21:16 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > >> It does nothing AFAICS for the > >> problem that when restarting archive recovery from a restartpoint, > >> it's not clear when it is safe to start letting in backends. You need > >> to get past the

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-28 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: >> It does nothing AFAICS for the >> problem that when restarting archive recovery from a restartpoint, >> it's not clear when it is safe to start letting in backends. You need >> to get past the highest LSN that has made it out to disk, and there is >> no g

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-28 Thread Simon Riggs
On Sun, 2008-09-28 at 14:02 -0400, Tom Lane wrote: > It does nothing AFAICS for the > problem that when restarting archive recovery from a restartpoint, > it's not clear when it is safe to start letting in backends. You need > to get past the highest LSN that has made it out to disk, and there i

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-28 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: >> After reading this for awhile, I realized that there is a rather >> fundamental problem with it: it switches into "consistent recovery" >> mode as soon as it's read WAL beyond ControlFile->minRecoveryPoi

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-26 Thread Simon Riggs
On Fri, 2008-09-26 at 11:20 +0100, Simon Riggs wrote: > > After reading this for awhile, I realized that there is a rather > > fundamental problem with it: it switches into "consistent recovery" > > mode as soon as it's read WAL beyond ControlFile->minRecoveryPoint. > > In a crash recovery situat

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-26 Thread Simon Riggs
On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > Version 7 > Anyway, that's sufficiently bad that I'm bouncing the patch for > reconsideration. No problem, I understand this needs discussion. There's less detail here than first appears. There a

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-25 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > Version 7 After reading this for awhile, I realized that there is a rather fundamental problem with it: it switches into "consistent recovery" mode as soon as it's read WAL beyond ControlFile->minRecoveryPoint. In a crash recovery situation that typically

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-23 Thread Simon Riggs
On Mon, 2008-09-22 at 23:06 +0100, Simon Riggs wrote: > On Thu, 2008-09-18 at 10:09 -0400, Tom Lane wrote: > > Simon Riggs <[EMAIL PROTECTED]> writes: > > > On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: > > >> Do we really need a checkpoint there at all? > > > > > "Timelines only change at s

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-22 Thread Simon Riggs
On Thu, 2008-09-18 at 10:09 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: > >> Do we really need a checkpoint there at all? > > > "Timelines only change at shutdown checkpoints". > > Hmm. I *think* that that is just a deb

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Simon Riggs
On Thu, 2008-09-18 at 10:09 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: > >> Do we really need a checkpoint there at all? > > > "Timelines only change at shutdown checkpoints". > > Hmm. I *think* that that is just a deb

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: >> Do we really need a checkpoint there at all? > "Timelines only change at shutdown checkpoints". Hmm. I *think* that that is just a debugging crosscheck rather than a critical property. But yeah, it w

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Simon Riggs
On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > Having some trouble trying to get a clean state change from recovery to > > normal mode. > > > Startup needs to be able to write WAL at the end of recovery so it can > > write a ShutdownCheckpoint, ye

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > Having some trouble trying to get a clean state change from recovery to > normal mode. > Startup needs to be able to write WAL at the end of recovery so it can > write a ShutdownCheckpoint, yet must not be allowed to write WAL before > that. Other services

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Simon Riggs
On Thu, 2008-09-18 at 09:05 +0100, Simon Riggs wrote: > Feels like I should shutdown the bgwriter after recovery and then > allow it to be cranked up again after normal processing starts, and do > all of this through postmaster state changes. That way bgwriter > doesn't need to do a dynamic state

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Simon Riggs
On Tue, 2008-09-16 at 15:45 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> wrote: > > Testing takes a while on this, I probably won't complete it until > > Friday. So enclosed patch is for eyeballs only at this stage. > > What's the status on that patch? Having some trouble trying to g

Re: [HACKERS] Infrastructure changes for recovery

2008-08-07 Thread Simon Riggs
On Thu, 2008-08-07 at 10:56 -0400, Alvaro Herrera wrote: > Simon Riggs wrote: > > ii) have postmaster allow connections while Redo is taking place, as > one > > part of allowing query access to standby database > > is interesting and I'm sure it would be very welcome. Of course, it > is first n

Re: [HACKERS] Infrastructure changes for recovery

2008-08-07 Thread Simon Riggs
On Thu, 2008-08-07 at 10:56 -0400, Alvaro Herrera wrote: > > * have startup process signal postmaster again when it has completed > > recovery, so that the change of state is via explicit signal rather than > > death of the child process > > I'm not sure that this is very useful, because the sta

Re: [HACKERS] Infrastructure changes for recovery

2008-08-07 Thread Simon Riggs
On Thu, 2008-08-07 at 10:48 -0400, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > I propose to > > * have startup process signal postmaster when it starts Redo phase (if > > it starts it) > > Doesn't seem like "starts recovery" is the point at which you can start > letting clients

Re: [HACKERS] Infrastructure changes for recovery

2008-08-07 Thread Alvaro Herrera
Simon Riggs wrote: > I propose to > * have startup process signal postmaster when it starts Redo phase (if > it starts it) I think the first is a good idea -- at least, if you can get the startup process to use the normal ReadBuffer code path instead of XLogReadBuffer. I don't really know what's

Re: [HACKERS] Infrastructure changes for recovery

2008-08-07 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > I propose to > * have startup process signal postmaster when it starts Redo phase (if > it starts it) Doesn't seem like "starts recovery" is the point at which you can start letting clients into the DB. What you want is to reach a point at which you're su

[HACKERS] Infrastructure changes for recovery

2008-08-07 Thread Simon Riggs
I would like to propose some changes to the infrastructure for recovery. These changes are beneficial in themselves, but also form the basis for other work we might later contemplate. Currently * the startup process performs restartpoints during recovery * the death of the startup process is tied