Re: [HACKERS] Hot standby, recovery infra

2009-01-29 Thread Heikki Linnakangas
Heikki Linnakangas wrote: It looks like if you issue a fast shutdown during recovery, postmaster doesn't kill bgwriter. Hmm, seems like we haven't thought through how shutdown during consistent recovery is supposed to behave in general. Right now, smart shutdown doesn't do anything during

Re: [HACKERS] Hot standby, recovery infra

2009-01-29 Thread Heikki Linnakangas
Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-01-29 at 15:31 +0200, Heikki Linnakangas wrote: Now when we restart the recovery, we will never reach minSafeStartPoint, which is now 0/400, and we'll fail with the error that Fujii-san pointed out. We're already way past the min

[HACKERS] Hot standby, recovery infra

2009-01-28 Thread Heikki Linnakangas
I've been reviewing and massaging the so called recovery infra patch. To recap, the goal is to: - start background writer during (archive) recovery - skip the shutdown checkpoint at the end of recovery. Instead, the database is brought up immediately, and the bgwriter performs a normal online

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Simon Riggs
On Wed, 2009-01-28 at 12:04 +0200, Heikki Linnakangas wrote: I've been reviewing and massaging the so called recovery infra patch. Thanks. I feel quite good about this patch now. Given the amount of code churn, it requires testing, and I'll read it through one more time after sleeping over

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Fujii Masao
Hi, On Wed, Jan 28, 2009 at 7:04 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I've been reviewing and massaging the so called recovery infra patch. Great! I feel quite good about this patch now. Given the amount of code churn, it requires testing, and I'll read it

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Simon Riggs
On Wed, 2009-01-28 at 23:19 +0900, Fujii Masao wrote: @@ -355,6 +359,27 @@ BackgroundWriterMain(void) */ PG_SETMASK(UnBlockSig); + BgWriterRecoveryMode = IsRecoveryProcessingMode(); + + if (BgWriterRecoveryMode) + elog(DEBUG1, bgwriter starting during

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Fujii Masao
Hi, On Wed, Jan 28, 2009 at 11:47 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2009-01-28 at 23:19 +0900, Fujii Masao wrote: @@ -355,6 +359,27 @@ BackgroundWriterMain(void) */ PG_SETMASK(UnBlockSig); + BgWriterRecoveryMode = IsRecoveryProcessingMode(); + + if

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Simon Riggs
On Wed, 2009-01-28 at 23:54 +0900, Fujii Masao wrote: Why is InitXLOGAccess() called also here when bgwriter is started after recovery? That is already called by AuxiliaryProcessMain(). InitXLOGAccess() sets the timeline and also gets the latest record pointer. If the bgwriter is

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Heikki Linnakangas
Fujii Masao wrote: On Wed, Jan 28, 2009 at 7:04 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I feel quite good about this patch now. Given the amount of code churn, it requires testing, and I'll read it through one more time after sleeping over it. Simon, do you see

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Gregory Stark
I skimmed through the Hot Standby patch for a preliminary review. I noted the following things, some minor tweaks, some just questions. None of the things I noted are big issues unless some of the questions uncover issues. 1) This code is obviously a cut-pasto: + else if (strcmp(tok1,

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Simon Riggs
On Wed, 2009-01-28 at 18:55 +, Gregory Stark wrote: I skimmed through the Hot Standby patch for a preliminary review. I noted the following things, some minor tweaks, some just questions. None of the things I noted are big issues unless some of the questions uncover issues. Thanks for

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Joshua D. Drake
On Wed, 2009-01-28 at 19:27 +, Simon Riggs wrote: On Wed, 2009-01-28 at 18:55 +, Gregory Stark wrote: Agreed. As explained when I published that patch it is deliberately severe to allow testing of conflict resolution and feedback on it. I still *strongly* feel the default has to be

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Heikki Linnakangas
Gregory Stark wrote: 6) I still don't understand why you need unobserved_xids. We don't need this in normal running, an xid we don't know for certain is committed is exactly the same as a transaction we know is currently running or aborted. So why do you need it during HS? In normal operation,

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Simon Riggs
On Wed, 2009-01-28 at 18:55 +, Gregory Stark wrote: I still don't understand why you need unobserved_xids. We don't need this in normal running, an xid we don't know for certain is committed is exactly the same as a transaction we know is currently running or aborted. So why do you need

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Tom Lane
Joshua D. Drake j...@commandprompt.com writes: On Wed, 2009-01-28 at 19:27 +, Simon Riggs wrote: On Wed, 2009-01-28 at 18:55 +, Gregory Stark wrote: I still *strongly* feel the default has to be the non-destructive conservative -1. I don't. Primarily, we must support high

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Aidan Van Dyk
* Tom Lane t...@sss.pgh.pa.us [090128 15:02]: Well, those unexpectedly cancelled queries could have represented critical functionality too. I think this argument calls the entire approach into question. If there is no safe setting for the parameter then we need to find a way to not have

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Greg Stark
(sorry for top posting -- blame apple) I don't see anything dangerous with either setting. For use cases where the backup is the primary purpose then killing queries is fine. For use cases where the maching is a reporting machine then saving large amounts of archived logs is fine.

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Greg Stark
Put another way: your characterization is no more true than claiming there's no safe setting for statement_timeout since a large value means clog could overflow your disk and your tables could bloat. (I note we default statement_timeout to off though) -- Greg On 28 Jan 2009, at 19:56, Tom

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Simon Riggs
On Wed, 2009-01-28 at 21:41 +0200, Heikki Linnakangas wrote: So, you can think of the unobserved xids array as an extension of ProcArray. The entries are like light-weight PGPROC entries. In fact I proposed earlier to simply create dummy PGPROC entries instead. Which we don't do because we

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Jeff Davis
On Wed, 2009-01-28 at 19:27 +, Simon Riggs wrote: my failover was 12 hours behind when I needed it to be 10 seconds behind and I lost a $1 million because of downtime of Postgres The same could be said for warm standby right now. Or Slony-I, for that matter. I think that we can reasonably

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Heikki Linnakangas
Tom Lane wrote: Joshua D. Drake j...@commandprompt.com writes: On Wed, 2009-01-28 at 19:27 +, Simon Riggs wrote: On Wed, 2009-01-28 at 18:55 +, Gregory Stark wrote: I still *strongly* feel the default has to be the non-destructive conservative -1. I don't. Primarily, we must support

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Simon Riggs
On Wed, 2009-01-28 at 11:33 -0800, Joshua D. Drake wrote: On Wed, 2009-01-28 at 19:27 +, Simon Riggs wrote: On Wed, 2009-01-28 at 18:55 +, Gregory Stark wrote: Agreed. As explained when I published that patch it is deliberately severe to allow testing of conflict resolution and

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Simon Riggs
On Wed, 2009-01-28 at 14:56 -0500, Tom Lane wrote: Well, those unexpectedly cancelled queries could have represented critical functionality too. I think this argument calls the entire approach into question. If there is no safe setting for the parameter then we need to find a way to not

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Heikki Linnakangas
Simon Riggs wrote: The essential choice is What would you like the max failover time to be?. Some users want one server with max 5 mins behind, some want two servers, one with 0 seconds behind, one with 12 hours behind It's not quite that simple. Setting max_standby_delay=5mins means that

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Simon Riggs
On Wed, 2009-01-28 at 22:47 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: The essential choice is What would you like the max failover time to be?. Some users want one server with max 5 mins behind, some want two servers, one with 0 seconds behind, one with 12 hours behind It's

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Jeff Davis
On Wed, 2009-01-28 at 22:47 +0200, Heikki Linnakangas wrote: It's not quite that simple. Setting max_standby_delay=5mins means that you're willing to wait 5 minutes for each query to die. Which means that in worst case you have to stop for 5 minutes at every single vacuum record, and fall

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Robert Haas
I don't. Primarily, we must support high availability. It is much better if we get people saying I get my queries cancelled and we say RTFM and change parameter X, than if people say my failover was 12 hours behind when I needed it to be 10 seconds behind and I lost a $1 million because of

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Gregory Stark
Simon Riggs si...@2ndquadrant.com writes: On Wed, 2009-01-28 at 14:56 -0500, Tom Lane wrote: Well, those unexpectedly cancelled queries could have represented critical functionality too. I think this argument calls the entire approach into question. If there is no safe setting for the

Re: [HACKERS] Hot Standby (v9d)

2009-01-28 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I vote with Simon. The thing is that if you get some queries cancelled, you'll realize you have a problem. ... or if you don't, they couldn't have been all that critical. Having your failover be 12 hours behind (or 12 months behind) is something

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Fujii Masao
Hi, On Wed, Jan 28, 2009 at 11:19 PM, Fujii Masao masao.fu...@gmail.com wrote: I feel quite good about this patch now. Given the amount of code churn, it requires testing, and I'll read it through one more time after sleeping over it. Simon, do you see anything wrong with this? I also read

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Fujii Masao
Hi, On Wed, Jan 28, 2009 at 11:19 PM, Fujii Masao masao.fu...@gmail.com wrote: I feel quite good about this patch now. Given the amount of code churn, it requires testing, and I'll read it through one more time after sleeping over it. Simon, do you see anything wrong with this? I also read

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Simon Riggs
On Thu, 2009-01-29 at 12:18 +0900, Fujii Masao wrote: Hi, On Wed, Jan 28, 2009 at 11:19 PM, Fujii Masao masao.fu...@gmail.com wrote: I feel quite good about this patch now. Given the amount of code churn, it requires testing, and I'll read it through one more time after sleeping over

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Simon Riggs
On Thu, 2009-01-29 at 10:36 +0900, Fujii Masao wrote: Hi, On Wed, Jan 28, 2009 at 11:19 PM, Fujii Masao masao.fu...@gmail.com wrote: I feel quite good about this patch now. Given the amount of code churn, it requires testing, and I'll read it through one more time after sleeping over

Re: [HACKERS] Hot standby, recovery infra

2009-01-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-01-29 at 12:18 +0900, Fujii Masao wrote: Though this is a matter of taste, I think that it's weird that bgwriter runs with ThisTimeLineID = 0 during recovery. This is because XLogCtl-ThisTimeLineID is set at the end of recovery. ISTM this will be a cause of bug

[HACKERS] Hot standby, recovery infrastructure

2009-01-27 Thread Heikki Linnakangas
Regarding this comment: + /* +* Prior to 8.4 we wrote a Shutdown Checkpoint at the end of recovery. +* This could add minutes to the startup time, so we want bgwriter +* to perform it. This then frees the Startup process to complete so we can +* allow transactions and WAL

Re: [HACKERS] Hot standby, recovery infrastructure

2009-01-27 Thread Simon Riggs
On Tue, 2009-01-27 at 15:59 +0200, Heikki Linnakangas wrote: Regarding this comment: + /* +* Prior to 8.4 we wrote a Shutdown Checkpoint at the end of recovery. +* This could add minutes to the startup time, so we want bgwriter +* to perform it. This then frees the

Re: [HACKERS] Hot standby, recovery infrastructure

2009-01-27 Thread Heikki Linnakangas
Simon Riggs wrote: On Tue, 2009-01-27 at 15:59 +0200, Heikki Linnakangas wrote: Regarding this comment: + /* +* Prior to 8.4 we wrote a Shutdown Checkpoint at the end of recovery. +* This could add minutes to the startup time, so we want bgwriter +* to perform it. This then

Re: [HACKERS] Hot standby, recovery infrastructure

2009-01-27 Thread Simon Riggs
On Tue, 2009-01-27 at 17:50 +0200, Heikki Linnakangas wrote: Hmm, I think we have small issue if the last WAL segment restored from the archive is an incomplete one: All normal archive recoveries have complete WAL files, since an xlog switch record jumps past missing entries at the end of the

Re: [HACKERS] Hot standby, recovery infrastructure

2009-01-27 Thread Simon Riggs
On Tue, 2009-01-27 at 17:50 +0200, Heikki Linnakangas wrote: Just think standard-online-checkpoint and it all fits. Exactly that made me wonder why the first checkpoint needs to be any different. Not really following you to be honest, assuming this was a separate point to the other part

Re: [HACKERS] Hot standby, recovery infrastructure

2009-01-27 Thread Heikki Linnakangas
Simon Riggs wrote: On Tue, 2009-01-27 at 17:50 +0200, Heikki Linnakangas wrote: Hmm, I think we have small issue if the last WAL segment restored from the archive is an incomplete one: All normal archive recoveries have complete WAL files, since an xlog switch record jumps past missing

Re: [HACKERS] Hot standby, recovery infrastructure

2009-01-27 Thread Simon Riggs
On Tue, 2009-01-27 at 20:11 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Tue, 2009-01-27 at 17:50 +0200, Heikki Linnakangas wrote: Hmm, I think we have small issue if the last WAL segment restored from the archive is an incomplete one: All normal archive recoveries have

Re: [HACKERS] Hot standby, recovery infrastructure

2009-01-27 Thread Heikki Linnakangas
Simon Riggs wrote: On Tue, 2009-01-27 at 20:11 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Tue, 2009-01-27 at 17:50 +0200, Heikki Linnakangas wrote: Hmm, I think we have small issue if the last WAL segment restored from the archive is an incomplete one: All normal archive

Re: [HACKERS] Hot Standby (v9d)

2009-01-27 Thread Simon Riggs
On Sat, 2009-01-24 at 17:24 +1300, Mark Kirkwood wrote: I'm doing some test runs with this now. I notice an old flatfiles related bug has reappeared: Bug fix patch against git repo. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support ***

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-26 Thread Simon Riggs
On Sun, 2009-01-25 at 19:56 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: 2. Kill all connections by given user. Hmm, not used for anything, actually. Should remove the roleId argument from GetConflictingVirtualXIDs. No, because we still need to add code to kill-connected-users

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-26 Thread Simon Riggs
On Sun, 2009-01-25 at 12:13 +, Grzegorz Jaskiewicz wrote: On 2009-01-25, at 09:04, Simon Riggs wrote: On Sat, 2009-01-24 at 21:58 +0200, Heikki Linnakangas wrote: When replaying a DROP TABLE SPACE, you first try to remove the directory, and if that fails, you assume that it's

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-26 Thread Andres Freund
Hi, Simon Riggs wrote: On Sun, 2009-01-25 at 19:56 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: 2. Kill all connections by given user. Hmm, not used for anything, actually. Should remove the roleId argument from GetConflictingVirtualXIDs. No, because we still need to add code to

Re: [HACKERS] Hot standby, conflict resolution

2009-01-26 Thread Simon Riggs
On Sun, 2009-01-25 at 16:19 +, Simon Riggs wrote: On Fri, 2009-01-23 at 21:30 +0200, Heikki Linnakangas wrote: Ok, then I think we have a little race condition. The startup process doesn't get any reply indicating that the target backend has processed the SIGINT and set the

Re: [HACKERS] Hot standby, conflict resolution

2009-01-26 Thread Heikki Linnakangas
Simon Riggs wrote: Rather than signalling, we could use a hasconflict boolean for each proc in a shared data structure. It can be read without spinlock, but should only be written while holding spinlock. Each time we read a block we check if hasconflict is set. If it is, we grab spinlock,

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-25 Thread Simon Riggs
On Sat, 2009-01-24 at 21:58 +0200, Heikki Linnakangas wrote: When replaying a DROP TABLE SPACE, you first try to remove the directory, and if that fails, you assume that it's because it's in use as a temp tablespace in a read-only transaction. That sounds like you think there another

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-25 Thread Heikki Linnakangas
Simon Riggs wrote: On Sat, 2009-01-24 at 21:58 +0200, Heikki Linnakangas wrote: When replaying a DROP TABLE SPACE, you first try to remove the directory, and if that fails, you assume that it's because it's in use as a temp tablespace in a read-only transaction. That sounds like you think

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-25 Thread Grzegorz Jaskiewicz
On 2009-01-25, at 09:04, Simon Riggs wrote: On Sat, 2009-01-24 at 21:58 +0200, Heikki Linnakangas wrote: When replaying a DROP TABLE SPACE, you first try to remove the directory, and if that fails, you assume that it's because it's in use as a temp tablespace in a read-only transaction.

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-25 Thread Simon Riggs
On Sun, 2009-01-25 at 11:28 +0200, Heikki Linnakangas wrote: You then call ResolveRecoveryConflictWithVirtualXIDs to kill such transactions, and try removing the directory again. But ResolveRecoveryConflictWithVirtualXIDs doesn't wait for the target transaction to die anymore (or

Re: [HACKERS] Hot standby, conflict resolution

2009-01-25 Thread Simon Riggs
On Fri, 2009-01-23 at 21:30 +0200, Heikki Linnakangas wrote: Ok, then I think we have a little race condition. The startup process doesn't get any reply indicating that the target backend has processed the SIGINT and set the cached conflict LSN. The target backend might move ahead using

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-25 Thread Heikki Linnakangas
Simon Riggs wrote: 2. Kill all connections by given user. Hmm, not used for anything, actually. Should remove the roleId argument from GetConflictingVirtualXIDs. No, because we still need to add code to kill-connected-users if we drop role. Oh, I see, that's still a todo item. But we don't

Re: [HACKERS] Hot Standby (v9d)

2009-01-24 Thread Simon Riggs
On Sat, 2009-01-24 at 17:24 +1300, Mark Kirkwood wrote: version 9g - please use this for testing now I'm doing some test runs with this now. I notice an old flatfiles related bug has reappeared: I'm seeing an off-by-one error on xmax, in some cases. That then causes the flat file update

Re: [HACKERS] Hot Standby (v9d)

2009-01-24 Thread Simon Riggs
On Sat, 2009-01-24 at 11:20 +, Simon Riggs wrote: On Sat, 2009-01-24 at 17:24 +1300, Mark Kirkwood wrote: version 9g - please use this for testing now I'm doing some test runs with this now. I notice an old flatfiles related bug has reappeared: I'm seeing an off-by-one error

[HACKERS] Hot standby, dropping a tablespace

2009-01-24 Thread Heikki Linnakangas
When replaying a DROP TABLE SPACE, you first try to remove the directory, and if that fails, you assume that it's because it's in use as a temp tablespace in a read-only transaction. You then call ResolveRecoveryConflictWithVirtualXIDs to kill such transactions, and try removing the directory

Re: [HACKERS] Hot Standby (v9d)

2009-01-24 Thread Mark Kirkwood
Simon Riggs wrote: On Sat, 2009-01-24 at 11:20 +, Simon Riggs wrote: On Sat, 2009-01-24 at 17:24 +1300, Mark Kirkwood wrote: version 9g - please use this for testing now I'm doing some test runs with this now. I notice an old flatfiles related bug has reappeared:

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Heikki Linnakangas
Simon Riggs wrote: * Put corrected version into GIT * Produce outstanding items as patch-on-patch via GIT I've applied the hot standby patch and recovery infra v9 patch to branches in my git repository, and pushed those to: git://git.postgresql.org/git/~hlinnaka/pgsql.git You can clone

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Simon Riggs
On Fri, 2009-01-23 at 16:14 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: * Put corrected version into GIT * Produce outstanding items as patch-on-patch via GIT I've applied the hot standby patch and recovery infra v9 patch to branches in my git repository, and pushed those to:

[HACKERS] Hot standby, conflict resolution

2009-01-23 Thread Heikki Linnakangas
The FATAL and ERROR cancellation modes are quite different. In FATAL mode, you just want to kill the backend. The target connection doesn't need to know the LSN. In ERROR mode, you don't really want to interrupt the target backend. In ReadBuffer, you're checking a global variable,

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-23 at 16:14 +0200, Heikki Linnakangas wrote: I made a couple of minor changes after importing your patches. I've applied them both to v9g, attached here. If you wouldn't mind redoing the initial step, I will promise not to do anything else to the code,

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Heikki Linnakangas
@@ -1601,6 +1602,24 @@ BufferProcessRecoveryConflictsIfAny(volatile BufferDesc *bufHdr) { XLogRecPtr bufLSN = BufferGetLSN(bufHdr); + /* +* If the buffer is recent we may need to cancel ourselves +* rather than risk

Re: [HACKERS] Hot standby, conflict resolution

2009-01-23 Thread Simon Riggs
On Fri, 2009-01-23 at 17:51 +0200, Heikki Linnakangas wrote: In ERROR mode, you don't really want to interrupt the target backend. In ReadBuffer, you're checking a global variable, BufferRecoveryConflictPending on each call, and if it's set, you check the buffer's LSN against the LSN of

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Simon Riggs
On Fri, 2009-01-23 at 18:22 +0200, Heikki Linnakangas wrote: @@ -1601,6 +1602,24 @@ BufferProcessRecoveryConflictsIfAny(volatile BufferDesc *bufHdr) { XLogRecPtr bufLSN = BufferGetLSN(bufHdr); + /* +* If the buffer is

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Heikki Linnakangas
Simon Riggs wrote: If you have a serializable transaction with subtransactions that suffers a serializability error it only cancels the current subtransaction. That means it's snapshot is still valid and can be used again. By analogy, as long as a transaction does not see any data that is

Re: [HACKERS] Hot standby, conflict resolution

2009-01-23 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-23 at 17:51 +0200, Heikki Linnakangas wrote: ISTM that if ReadBuffer read the value directly from the PGPROC entry, there would be no need for the signaling (in the ERROR mode). That is possible and I considered it. If we did it that way we would need to

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Alvaro Herrera
Heikki Linnakangas wrote: Simon Riggs wrote: If you have a serializable transaction with subtransactions that suffers a serializability error it only cancels the current subtransaction. That means it's snapshot is still valid and can be used again. By analogy, as long as a transaction does

Re: [HACKERS] Hot standby, conflict resolution

2009-01-23 Thread Simon Riggs
On Fri, 2009-01-23 at 21:30 +0200, Heikki Linnakangas wrote: Correct me if I'm wrong, but I thought the idea of this new conflict resolution was that the startup process doesn't need to wait for the target backend to die. Instead, the target backend knows to commit suicide if it

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Greg Stark
If you have a serializable transaction with subtransactions that suffers a serializability error it only cancels the current subtransaction. This is a complete tangent to your work, but I wonder if this is really right. I mean it's not as if we could have srrialized the transaction

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Jeff Davis
On Fri, 2009-01-23 at 20:13 +, Greg Stark wrote: If you have a serializable transaction with subtransactions that suffers a serializability error it only cancels the current subtransaction. This is a complete tangent to your work, but I wonder if this is really right. I mean it's

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Mark Kirkwood
Simon Riggs wrote: On Thu, 2009-01-22 at 22:35 +, Simon Riggs wrote: * Bug fix v9 over next few days version 9g - please use this for testing now I'm doing some test runs with this now. I notice an old flatfiles related bug has reappeared: master: =# create database

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-20 Thread Heikki Linnakangas
Simon Riggs wrote: It would be useful to nibble away at the patch, committing all the necessary refactorings to make the patch work. That will reduce size of eventual commit. I committed this part now; one less thing to review. Please adjust the patch accordingly. -- Heikki Linnakangas

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-20 Thread Simon Riggs
On Tue, 2009-01-20 at 21:01 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: It would be useful to nibble away at the patch, committing all the necessary refactorings to make the patch work. That will reduce size of eventual commit. I committed this part now; one less thing to

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-20 Thread Heikki Linnakangas
Simon Riggs wrote: I did want you to commit those changes, but that was 8 days ago and much has changed since then. Now, I'm slightly worried that this may be a retrograde step. The last 3 days I've been refactoring the code to account for other requirements, as I have been discussing, and some

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 09:16 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2009-01-16 at 22:09 +0200, Heikki Linnakangas wrote: RecentGlobalXmin is just a hint, it lags behind the real oldest xmin that GetOldestXmin() would return. If another backend has a more recent

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 12:22 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: Well, steps 7 and 8 don't make sense. Your earlier comment was that it was possible for a WAL record to be written with a RecentGlobalXmin that was lower than other backends values. In step 9 the

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: One of us needs a coffee. Clearly, I put the kettle on... How does Transaction 4 have a RecentGlobalXmin of 2 in step (7), but at step (9) the value of RecentGlobalXmin has gone backwards? Looks like I mixed up the xids of the two transactions in steps 8 and 9. Let's

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 12:50 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: One of us needs a coffee. Clearly, I put the kettle on... I had one too, just in case. How does Transaction 4 have a RecentGlobalXmin of 2 in step (7), but at step (9) the value of RecentGlobalXmin has

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: Well, steps 7 and 8 don't make sense. Your earlier comment was that it was possible for a WAL record to be written with a RecentGlobalXmin that was lower than other backends values. In step 9 the RecentGlobalXmin is *not* lower than any other backend, it is the same. So if

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: GetSnapshotData() sets RecentGlobalXmin to the result of the snapshot's xmin. No. RecentGlobalXmin is set to the oldest *xmin* observed, across all running transactions. TransactionXmin is the xid of the oldest running transaction. IOW, RecentGlobalXmin is the xid of

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 14:00 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: GetSnapshotData() sets RecentGlobalXmin to the result of the snapshot's xmin. No. RecentGlobalXmin is set to the oldest *xmin* observed, across all running transactions. TransactionXmin is the xid of the

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: I suggest we handle this on the recovery side, not on the master, by deriving the xmin at the point the WAL record arrives. We would calculate it by looking at recovery procs only. That will likely give us a later value than we would get from the master, but that can't be

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 15:47 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: I suggest we handle this on the recovery side, not on the master, by deriving the xmin at the point the WAL record arrives. We would calculate it by looking at recovery procs only. That will likely give us a

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 12:54 +, Simon Riggs wrote: Some refactoring in this area is also required because we need to handle two other types of conflict to correctly support drop database and drop user, which is now underway. I've hung the drop database conflict code in dbase_redo(). For

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: I prefer the last one, but if you think otherwise, please shout. We're now emitting WAL records for relcache invalidations, IIRC. I wonder if those are useful for this? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 16:50 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: I prefer the last one, but if you think otherwise, please shout. We're now emitting WAL records for relcache invalidations, IIRC. I wonder if those are useful for this? Tom already objected to putting strange

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-18 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-16 at 22:09 +0200, Heikki Linnakangas wrote: RecentGlobalXmin is just a hint, it lags behind the real oldest xmin that GetOldestXmin() would return. If another backend has a more recent RecentGlobalXmin value, and has killed more recent tuples on the page,

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-16 Thread Heikki Linnakangas
I don't think RecentGlobalXmin is good enough here: ! /* !* We would like to set an accurate latestRemovedXid, but there !* is no easy way of obtaining a useful value. So we use the !* probably far

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-16 Thread Simon Riggs
On Fri, 2009-01-16 at 16:33 +0200, Heikki Linnakangas wrote: I don't think RecentGlobalXmin is good enough here: ! /* !* We would like to set an accurate latestRemovedXid, but there !* is no easy way of obtaining a useful

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-16 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-16 at 16:33 +0200, Heikki Linnakangas wrote: I don't think RecentGlobalXmin is good enough here: ! /* !* We would like to set an accurate latestRemovedXid, but there !* is no easy way of

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-16 Thread Simon Riggs
On Fri, 2009-01-16 at 22:09 +0200, Heikki Linnakangas wrote: RecentGlobalXmin is just a hint, it lags behind the real oldest xmin that GetOldestXmin() would return. If another backend has a more recent RecentGlobalXmin value, and has killed more recent tuples on the page, the

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Simon Riggs
On Fri, 2009-01-09 at 19:34 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: It would be useful to nibble away at the patch, committing all the necessary refactorings to make the patch work. That will reduce size of eventual commit. Agreed. This in particular is a change I feel

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Heikki Linnakangas
Simon Riggs wrote: The idea outlined before didn't deal with all call points for RecordIsCleanupRecord(), so doesn't actually work. Are we talking about the same thing? If we put the control of locking to the hands of the redo-function, I don't see why it couldn't use a lock of the right

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Heikki Linnakangas
Simon Riggs wrote: The idea outlined before didn't deal with all call points for RecordIsCleanupRecord(), so doesn't actually work. Hmm, grep finds two call points for that: ! case RECOVERY_TARGET_PAUSE_CLEANUP: ! /* !

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Simon Riggs
On Thu, 2009-01-15 at 18:05 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: The idea outlined before didn't deal with all call points for RecordIsCleanupRecord(), so doesn't actually work. Are we talking about the same thing? I guess not. Look at the other call points for that

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Simon Riggs
On Thu, 2009-01-15 at 18:16 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: Is the first really useful? I would understand advance until next cleanup record *that would block/kill queries*, but why would you want to advance until the next cleanup record? Minor difference there, but

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-15 Thread Simon Riggs
On Thu, 2009-01-08 at 22:31 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: If you want to do things a different way you need to say what you want to do and what effects those changes will have. I want to reduce the coupling between the primary and the master. The less they need to

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-15 Thread Simon Riggs
On Thu, 2009-01-15 at 22:10 +, Simon Riggs wrote: I notice that we are no longer able to record the databaseId for a connection, so query conflict resolution cannot be isolated to individual databases any longer. Wrong way round. Incoming WAL records from dbOid on them, so we can still

Re: [HACKERS] Hot standby and b-tree killed items

2009-01-13 Thread Simon Riggs
On Tue, 2008-12-30 at 18:31 +0200, Heikki Linnakangas wrote: You have to be careful to ignore the flags in read-only transactions that started in hot standby mode, even if recovery has since ended and we're in normal operation now. My initial implementation in v6 worked, but had a corner

<    4   5   6   7   8   9   10   11   >