Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-12-03 Thread Simon Riggs
On Wed, 2009-11-25 at 03:12 +, Greg Stark wrote: > On Wed, Nov 25, 2009 at 2:10 AM, Tom Lane wrote: > > As long as there's not anything the master actually does differently > > then I can't see where there'd be any performance testing to do. What's > > bothering me about this is that it seems

Re: [HACKERS] Hot Standby remaining issues

2009-12-02 Thread Heikki Linnakangas
Simon Riggs wrote: > Hmm, what happens if someone enables wal_standby_info in postgresql.conf > while the server is shutdown. It would still be a valid starting point > in that case. Yeah, true. > I'll just make a note, I think. Yeah, a manual (or automatic, if you just wait) checkpoint will pro

Re: [HACKERS] Hot Standby remaining issues

2009-12-02 Thread Simon Riggs
On Wed, 2009-12-02 at 16:41 +, Simon Riggs wrote: > On Tue, 2009-12-01 at 20:26 +0200, Heikki Linnakangas wrote: > > Simon Riggs wrote: > > > commit 02c3eadb766201db084b668daa271db4a900adc9 > > > Author: Simon Riggs > > > Date: Sat Nov 28 06:23:33 2009 + > > > > > > Added wal_standb

Re: [HACKERS] Hot Standby remaining issues

2009-12-02 Thread Simon Riggs
On Tue, 2009-12-01 at 20:26 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > commit 02c3eadb766201db084b668daa271db4a900adc9 > > Author: Simon Riggs > > Date: Sat Nov 28 06:23:33 2009 + > > > > Added wal_standby_info GUC to turn RM_STANDBY_ID messages on/off. > > Various co

Re: [HACKERS] Hot Standby remaining issues

2009-12-02 Thread Heikki Linnakangas
Simon Riggs wrote: > On Wed, 2009-12-02 at 12:49 +0200, Heikki Linnakangas wrote: > >> If a read-only transaction holds a lot of locks, consuming so much >> lock space that there's none left for the startup process to hold the >> lock it wants, it will abort and bring down postmaster. The patch >>

Re: [HACKERS] Hot Standby remaining issues

2009-12-02 Thread Simon Riggs
On Wed, 2009-12-02 at 12:49 +0200, Heikki Linnakangas wrote: > If a read-only transaction holds a lot of locks, consuming so much > lock space that there's none left for the startup process to hold the > lock it wants, it will abort and bring down postmaster. The patch > attempts to kill any confl

Re: [HACKERS] Hot Standby remaining issues

2009-12-02 Thread Heikki Linnakangas
Heikki Linnakangas wrote: > Simon Riggs wrote: >> @@ -654,10 +656,13 @@ LockAcquire(const LOCKTAG *locktag, >> elog(PANIC, "lock table corrupted"); >> } >> LWLockRelease(partitionLock); >> -ereport(ERROR, >> -

Re: [HACKERS] Hot Standby remaining issues

2009-12-01 Thread Heikki Linnakangas
Simon Riggs wrote: > commit 02c3eadb766201db084b668daa271db4a900adc9 > Author: Simon Riggs > Date: Sat Nov 28 06:23:33 2009 + > > Added wal_standby_info GUC to turn RM_STANDBY_ID messages on/off. > Various comments added also. > This patch makes it unsafe to start hot standby mode

Re: [HACKERS] Hot Standby remaining issues

2009-11-30 Thread Heikki Linnakangas
Simon Riggs wrote: > @@ -654,10 +656,13 @@ LockAcquire(const LOCKTAG *locktag, > elog(PANIC, "lock table corrupted"); > } > LWLockRelease(partitionLock); > - ereport(ERROR, > - (errcode(ERRCODE_OUT_OF_

Re: [HACKERS] Hot Standby remaining issues

2009-11-27 Thread Simon Riggs
On Wed, 2009-11-25 at 13:00 +0200, Heikki Linnakangas wrote: > I've put up a wiki page with the issues I see with the patch as it > stands. They're roughly categorized by seriousness. > > http://wiki.postgresql.org/wiki/Hot_Standby_TODO > > New issues can and probably will still pop up, let's add

Re: [HACKERS] Hot Standby and cancelling idle queries

2009-11-27 Thread Simon Riggs
On Thu, 2009-11-26 at 08:30 +0200, Heikki Linnakangas wrote: > I suspect you missed the context of this change. It's about the code > in tablespc.c, to kill all backends that might have a temporary file > in a tablespace that's being dropped. It's not about tuple visibility > but temporary files.

Re: [HACKERS] Hot Standby and cancelling idle queries

2009-11-25 Thread Heikki Linnakangas
Simon Riggs wrote: > Recent change: > > An idle-in-transaction transaction can also hold a temporary file. Think > of an open cursor, for example. Therefore, remove the distinction > between CONFLICT_MODE_ERROR and CONFLICT_MODE_ERROR_IF_NOT_IDLE, > idle-in-transaction backends need to be killed t

Re: [HACKERS] Hot Standby and cancelling idle queries

2009-11-25 Thread Tom Lane
Simon Riggs writes: > An idle-in-transaction transaction can also hold a temporary file. Think > of an open cursor, for example. Therefore, remove the distinction > between CONFLICT_MODE_ERROR and CONFLICT_MODE_ERROR_IF_NOT_IDLE, > idle-in-transaction backends need to be killed too when a tablespa

[HACKERS] Hot Standby and cancelling idle queries

2009-11-25 Thread Simon Riggs
Recent change: An idle-in-transaction transaction can also hold a temporary file. Think of an open cursor, for example. Therefore, remove the distinction between CONFLICT_MODE_ERROR and CONFLICT_MODE_ERROR_IF_NOT_IDLE, idle-in-transaction backends need to be killed too when a tablespace is droppe

Re: [HACKERS] Hot Standby remaining issues

2009-11-25 Thread Simon Riggs
On Wed, 2009-11-25 at 13:00 +0200, Heikki Linnakangas wrote: > I've put up a wiki page with the issues I see with the patch as it > stands. They're roughly categorized by seriousness. > > http://wiki.postgresql.org/wiki/Hot_Standby_TODO > > New issues can and probably will still pop up, let's add

[HACKERS] Hot Standby remaining issues

2009-11-25 Thread Heikki Linnakangas
I've put up a wiki page with the issues I see with the patch as it stands. They're roughly categorized by seriousness. http://wiki.postgresql.org/wiki/Hot_Standby_TODO New issues can and probably will still pop up, let's add them to the list as they're found so that we know what still needs to be

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-24 Thread Greg Stark
On Wed, Nov 25, 2009 at 3:26 AM, Tom Lane wrote: > Greg Stark writes: >> Well the only thing that's been discussed is having vacuum require a >> minimum age before considering a transaction visible to all to reduce >> the chance of conflicts on cleanup records. > > [ shrug... ]  Call me Cassandra

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-24 Thread Simon Riggs
On Wed, 2009-11-25 at 03:12 +, Greg Stark wrote: > On Wed, Nov 25, 2009 at 2:10 AM, Tom Lane wrote: > > As long as there's not anything the master actually does differently > > then I can't see where there'd be any performance testing to do. What's > > bothering me about this is that it seems

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-24 Thread Tom Lane
Greg Stark writes: > Well the only thing that's been discussed is having vacuum require a > minimum age before considering a transaction visible to all to reduce > the chance of conflicts on cleanup records. [ shrug... ] Call me Cassandra. I am not concerned about what has or has not been discu

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-24 Thread Greg Stark
On Wed, Nov 25, 2009 at 2:10 AM, Tom Lane wrote: > As long as there's not anything the master actually does differently > then I can't see where there'd be any performance testing to do.  What's > bothering me about this is that it seems likely that we'll find places > where the master has to do t

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-24 Thread Tom Lane
Simon Riggs writes: > Tom Lane wrote: >> There's no equivalent of XLogArchivingActive()? > We've tried hard to have it "just work". But I wonder whether we should > have a parameter to allow performance testing on the master? If nobody > finds any issues then we can remove it again, or at least m

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-24 Thread Simon Riggs
On Sat, 2009-11-21 at 23:00 +0200, Heikki Linnakangas wrote: > Tom Lane wrote: > > Heikki Linnakangas writes: > >> Tom Lane wrote: > >>> There's no equivalent of XLogArchivingActive()? > > > >> XLogArchivingMode() == false enables us to skip WAL-logging in > >> operations like CLUSTER or COPY, wh

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-24 Thread Simon Riggs
On Sat, 2009-11-21 at 20:20 +0200, Heikki Linnakangas wrote: > That causes some headaches for Hot Standby I say leave HS as it is and we can clean up when we do the VFectomy. It isn't really a headache, the code works easily enough. I agree its ugly and it should eventually be removed. Let's no

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-23 Thread Hannu Krosing
On Tue, 2009-11-24 at 09:24 +0200, Heikki Linnakangas wrote: > Greg Smith wrote: > > Heikki Linnakangas wrote: > >> So I guess what I'm asking is: Does anyone see any show-stoppers in > >> removing VACUUM FULL > > Here's the disclaimers attached to the new VACUUM REPLACE implementation > > from Ita

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-23 Thread Heikki Linnakangas
Greg Smith wrote: > Heikki Linnakangas wrote: >> So I guess what I'm asking is: Does anyone see any show-stoppers in >> removing VACUUM FULL > Here's the disclaimers attached to the new VACUUM REPLACE implementation > from Itagaki: > > "We still need traditional VACUUM FULL behavior for system cat

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-23 Thread Heikki Linnakangas
Itagaki Takahiro wrote: > VACUUM FULL is still reserved for system catalogs in my patch > because we cannot modify relfilenodes for the catalog tables. > Do you have solutions for it? Tom had an idea on that: http://archives.postgresql.org/message-id/19750.1252094...@sss.pgh.pa.us -- Heikki L

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-23 Thread Itagaki Takahiro
Heikki Linnakangas wrote: > So I guess what I'm asking is: Does anyone see any show-stoppers in > removing VACUUM FULL, and does anyone want to step up to the plate and > promise to do it before release? I'm working on "New VACUUM FULL" patch, that shrinks tables using CLUSTER-like rewrites. ht

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Heikki Linnakangas
Tom Lane wrote: > Heikki Linnakangas writes: >> Tom Lane wrote: >>> There's no equivalent of XLogArchivingActive()? > >> XLogArchivingMode() == false enables us to skip WAL-logging in >> operations like CLUSTER or COPY, which is a big optimization. I don't >> see anything like that in Hot Standby

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Tom Lane
Heikki Linnakangas writes: > Tom Lane wrote: >> There's no equivalent of XLogArchivingActive()? > XLogArchivingMode() == false enables us to skip WAL-logging in > operations like CLUSTER or COPY, which is a big optimization. I don't > see anything like that in Hot Standby. There is a few small th

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Heikki Linnakangas
Tom Lane wrote: > Heikki Linnakangas writes: >> Tom Lane wrote: >>> I don't see much problem with rejecting VAC FULL in a HS master, >>> whether or not it gets removed altogether. Why not just do that >>> rather than write a lot of kluges? > >> Hmm. At the moment, no action is required in the ma

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Tom Lane
Heikki Linnakangas writes: > Tom Lane wrote: >> I don't see much problem with rejecting VAC FULL in a HS master, >> whether or not it gets removed altogether. Why not just do that >> rather than write a lot of kluges? > Hmm. At the moment, no action is required in the master to allow hot > stand

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Heikki Linnakangas
Tom Lane wrote: > Heikki Linnakangas writes: >> So I guess what I'm asking is: Does anyone see any show-stoppers in >> removing VACUUM FULL, and does anyone want to step up to the plate and >> promise to do it before release? > > I don't see much problem with rejecting VAC FULL in a HS master, >

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Tom Lane
Heikki Linnakangas writes: > So I guess what I'm asking is: Does anyone see any show-stoppers in > removing VACUUM FULL, and does anyone want to step up to the plate and > promise to do it before release? I don't see much problem with rejecting VAC FULL in a HS master, whether or not it gets remo

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Greg Smith
Heikki Linnakangas wrote: So I guess what I'm asking is: Does anyone see any show-stoppers in removing VACUUM FULL Here's the disclaimers attached to the new VACUUM REPLACE implementation from Itagaki: "We still need traditional VACUUM FULL behavior for system catalog because we cannot change

[HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Heikki Linnakangas
VACUUM FULL does a peculiar hack: once it's done moving tuples, and before it truncates the relation, it calls RecordTransactionCommit to mark the transaction as committed in clog and WAL, but the transaction is still kept open in proc array. After it's done with truncating and other cleanup, norma

Re: [HACKERS] Hot standby, overflowed snapshots, testing

2009-11-15 Thread Robert Hodges
On 11/15/09 2:25 AM PST, "Simon Riggs" wrote: > On Sat, 2009-11-14 at 08:43 -0800, Robert Hodges wrote: > >> I can help set up automated basic tests for hot standby using 1+1 setups on >> Amazon. I¹m already working on tests for warm standby for our commercial >> Tungsten implementation and ne

Re: [HACKERS] Hot standby, race condition between recovery snapshot and commit

2009-11-15 Thread Heikki Linnakangas
Simon Riggs wrote: > On Sun, 2009-11-15 at 21:37 +0200, Heikki Linnakangas wrote: > >> Am I missing anything? > > Will review. Thanks! Please use the head of git branch, I already found one major oversight in what I posted that's fixed there... I should go to bed already. -- Heikki Linnakang

Re: [HACKERS] Hot standby, race condition between recovery snapshot and commit

2009-11-15 Thread Heikki Linnakangas
Oh, forgot to mention another thing that I've been pondering: Currently, the running-xacts record is written to the WAL after the checkpoint record. There's a small chance that you get an xlog switch in between. If that happens, it might take a long time after the checkpoint record until the stand

Re: [HACKERS] Hot standby, race condition between recovery snapshot and commit

2009-11-15 Thread Heikki Linnakangas
Simon Riggs wrote: > On Sat, 2009-11-14 at 14:59 +0200, Heikki Linnakangas wrote: >> I can't see any obvious way around that. > > Huh? We're only doing this strict locking approach because you insisted > that the looser approach was not acceptable. Take it easy, Simon. By obvious, I meant "trivi

Re: [HACKERS] Hot standby, overflowed snapshots, testing

2009-11-15 Thread Simon Riggs
On Sat, 2009-11-14 at 08:43 -0800, Robert Hodges wrote: > I can help set up automated basic tests for hot standby using 1+1 setups on > Amazon. I¹m already working on tests for warm standby for our commercial > Tungsten implementation and need to solve the problem of creating tests that > adapt

Re: [HACKERS] Hot standby, overflowed snapshots, testing

2009-11-14 Thread Robert Hodges
Hi Simon and Heikki, I can help set up automated basic tests for hot standby using 1+1 setups on Amazon. I¹m already working on tests for warm standby for our commercial Tungsten implementation and need to solve the problem of creating tests that adapt flexibly across different replication mecha

[HACKERS] Hot standby, race condition between recovery snapshot and commit

2009-11-14 Thread Heikki Linnakangas
There's a race condition between transaction commit and GetRunningTransactionData(). If GetRunningTransactionData() runs between the RecordTransactionCommit() and ProcArrayEndTransaction() calls in CommitTransaction(): > /* >* Here is where we really truly commit. >*/ >

Re: [HACKERS] Hot standby, overflowed snapshots, testing

2009-11-13 Thread Simon Riggs
On Fri, 2009-11-13 at 22:19 +0200, Heikki Linnakangas wrote: > I got the impression earlier that you had some test environment set up > to test hot standby. Can you share any details of what test cases > you've run? Fair question. The Sep 15 submission happened too quickly for us to mobilise test

[HACKERS] Hot standby, overflowed snapshots, testing

2009-11-13 Thread Heikki Linnakangas
In GetSnapshotData(), we set subcount to -1 if the snapshot was overflowed: > subcount = GetKnownAssignedTransactions(snapshot->subxip, > > &xmin, xmax, &overflow); > > /* >

Re: [HACKERS] Hot standby, freezing

2009-11-11 Thread Simon Riggs
On Wed, 2009-11-11 at 12:35 +0200, Heikki Linnakangas wrote: > While reading through the patch for what must be the 100th time by now, :-) > it occurred to me that this comment in heap_xlog_freeze: > > + /* > +* Freezing tuples does not require conflict processing > +*/ >

Re: [HACKERS] Hot Standby status

2009-11-11 Thread Simon Riggs
On Tue, 2009-11-10 at 15:11 -0500, Michael Glaesemann wrote: > I skimmed through the documentation to get a better handle on what > this will mean. Thanks for this and any further corrections/additions. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (p

Re: [HACKERS] Hot Standby status

2009-11-11 Thread Simon Riggs
On Tue, 2009-11-10 at 22:00 +0200, Devrim GÜNDÜZ wrote: > On Tue, 2009-11-10 at 20:36 +0200, Heikki Linnakangas wrote: > > Attached is the latest and greatest patch against CVS head, taken from > > the hs-riggs branch in my git repository. > > Is there a reason why recovery.conf.sample does not in

[HACKERS] Hot standby, freezing

2009-11-11 Thread Heikki Linnakangas
While reading through the patch for what must be the 100th time by now, it occurred to me that this comment in heap_xlog_freeze: + /* +* Freezing tuples does not require conflict processing +*/ is plain wrong. In the master, we can freeze the xmin of a tuple that's not yet v

Re: [HACKERS] Hot Standby status

2009-11-11 Thread Heikki Linnakangas
Devrim GÜNDÜZ wrote: > Is there a reason why recovery.conf.sample does not include (sample) > entries for recovery_connections and max_standby_delay? No, they probably should be included. I'll add them, thanks. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pg

Re: [HACKERS] Hot Standby status

2009-11-10 Thread Heikki Linnakangas
Michael Glaesemann wrote: > Other commands are in all-caps. Any reason INSERT, UPDATE, DELETE, > TRUNCATE, CREATE, DROP, ALTER, and COMMENT are not? No. Thanks, I'll fix that. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] Hot Standby status

2009-11-10 Thread Michael Glaesemann
On Nov 10, 2009, at 13:36 , Heikki Linnakangas wrote: Attached is the latest and greatest patch against CVS head, taken from the hs-riggs branch in my git repository. Awesome. Thank you, Simon and Heikki! I skimmed through the documentation to get a better handle on what this will mean.

Re: [HACKERS] Hot Standby status

2009-11-10 Thread Devrim GÜNDÜZ
On Tue, 2009-11-10 at 20:36 +0200, Heikki Linnakangas wrote: > Attached is the latest and greatest patch against CVS head, taken from > the hs-riggs branch in my git repository. Is there a reason why recovery.conf.sample does not include (sample) entries for recovery_connections and max_standby_de

Re: [HACKERS] Hot Standby and 64+ subxids (was COPY enhancements)

2009-11-09 Thread Tom Lane
Andrew Dunstan writes: > Simon Riggs wrote: >> HS writes a WAL record for subtransactions at the point that the subxid >> cache overflows for any single transaction. Current cache size = 64. >> Top-level transaction then writes one additional WAL record every >> additional 64 subxids after that. T

Re: [HACKERS] Hot Standby and 64+ subxids (was COPY enhancements)

2009-11-09 Thread Andrew Dunstan
Simon Riggs wrote: On Thu, 2009-10-08 at 11:50 -0400, Tom Lane wrote: Robert Haas writes: Subcommitting every single row is going to be really painful, especially after Hot Standby goes in and we have to issue a WAL record after every 64 subtransactions (AIUI). Yikes ... I ha

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-22 Thread Simon Riggs
On Thu, 2009-10-22 at 09:41 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Thu, 2009-10-22 at 07:55 +0300, Heikki Linnakangas wrote: > >> Making some effort to transfer locks instead of acquiring+releasing > >> would eliminate the need for having extra lock space available when > >> s

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Heikki Linnakangas
Simon Riggs wrote: > On Thu, 2009-10-22 at 07:55 +0300, Heikki Linnakangas wrote: >> Making some effort to transfer locks instead of acquiring+releasing >> would eliminate the need for having extra lock space available when >> switching from hot standby mode to normal operation. > > This isn't ver

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Simon Riggs
On Thu, 2009-10-22 at 07:55 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Wed, 2009-10-21 at 23:02 +0300, Heikki Linnakangas wrote: > >> Hmm, dunno about that, but there is one problem with the "grant to dummy > >> proc, then release in startup process" approach. What if there isn't

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Heikki Linnakangas
Simon Riggs wrote: > On Wed, 2009-10-21 at 23:02 +0300, Heikki Linnakangas wrote: >> Hmm, dunno about that, but there is one problem with the "grant to dummy >> proc, then release in startup process" approach. What if there isn't >> enough shared memory available to re-acquire the lock for the dumm

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Simon Riggs
On Wed, 2009-10-21 at 23:02 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Wed, 2009-10-21 at 19:37 +0300, Heikki Linnakangas wrote: > > > >> So, I'm quite eager to just revert all those lock_twophase_recover() > >> changes, and always rely on the "grant lock to dummy proc, then > >>

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Heikki Linnakangas
Simon Riggs wrote: > On Wed, 2009-10-21 at 19:37 +0300, Heikki Linnakangas wrote: > >> So, I'm quite eager to just revert all those lock_twophase_recover() >> changes, and always rely on the "grant lock to dummy proc, then >> release >> it in startup process" method. If we don't want to rely on th

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Simon Riggs
On Wed, 2009-10-21 at 19:37 +0300, Heikki Linnakangas wrote: > So, I'm quite eager to just revert all those lock_twophase_recover() > changes, and always rely on the "grant lock to dummy proc, then > release > it in startup process" method. If we don't want to rely on that, > PostPrepare_Locks is

[HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Heikki Linnakangas
The Hot Standby patch changes lock_twophase_recover() so that when we're starting up from Hot Standby mode to normal operation, as opposed to crash recovery, we assume that all AccessExcusiveLocks are already held by the startup process and instead of acquiring them anew they are transferred from t

Re: [HACKERS] Hot standby, pausing recovery

2009-10-20 Thread Simon Riggs
On Tue, 2009-10-20 at 09:43 +0100, Simon Riggs wrote: > On Tue, 2009-10-20 at 10:33 +0300, Heikki Linnakangas wrote: > > At this point, I'd like to cut out all those control functions to > > pause/stop at various points from the patch. > > OK Maybe OK, I should say. Some parts are important for t

Re: [HACKERS] Hot standby, pausing recovery

2009-10-20 Thread Simon Riggs
On Tue, 2009-10-20 at 10:33 +0300, Heikki Linnakangas wrote: > At this point, I'd like to cut out all those control functions to > pause/stop at various points from the patch. OK -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] Hot standby, pausing recovery

2009-10-20 Thread Heikki Linnakangas
Simon Riggs wrote: > On Fri, 2009-10-16 at 09:33 +0300, Heikki Linnakangas wrote: >> If you pause recovery, and then continue, we reset to "target none" >> mode, even if a stopping point was set previously. > > Yes, currently. Resetting the target mode will do what you want, rather > than continu

Re: [HACKERS] Hot standby, pausing recovery

2009-10-16 Thread Simon Riggs
On Fri, 2009-10-16 at 09:33 +0300, Heikki Linnakangas wrote: > In advance-mode, we will merrilly skip over a WAL record that's a > recovery stop target. Is that a bug or a feature? Merrily?!? I saw it more as a sombre stepping motion. Advance currently means set target at next record and continu

[HACKERS] Hot standby, pausing recovery

2009-10-15 Thread Heikki Linnakangas
This if-block is misplaced: > case RECOVERY_TARGET_STOP_IMMEDIATE: > case RECOVERY_TARGET_STOP_XID: > case RECOVERY_TARGET_STOP_TIME: > paused = false; > br

Re: [HACKERS] Hot standby, recovery_starts_paused

2009-10-15 Thread Simon Riggs
On Fri, 2009-10-16 at 01:43 +0300, Heikki Linnakangas wrote: > recovery_starts_paused is useless as it is. It pauses the recovery right > after the first WAL record, all right, but before we see a running-xacts > record, we won't let any backends in. And if you can't connect, you > can't unpause,

[HACKERS] Hot standby, recovery_starts_paused

2009-10-15 Thread Heikki Linnakangas
recovery_starts_paused is useless as it is. It pauses the recovery right after the first WAL record, all right, but before we see a running-xacts record, we won't let any backends in. And if you can't connect, you can't unpause, so it's stuck forever. It should probably behave as "pause after reac

Re: [HACKERS] Hot standby status

2009-10-15 Thread Heikki Linnakangas
Simon Riggs wrote: > On Thu, 2009-10-15 at 10:33 +0300, Heikki Linnakangas wrote: >> Let me know if I'm missing something. And please feel free to help, by >> testing, by reviewing and commenting on the patch, or by addressing any >> of the above issues. I will continue working on this, but this is

Re: [HACKERS] Hot standby status

2009-10-15 Thread Simon Riggs
On Thu, 2009-10-15 at 10:33 +0300, Heikki Linnakangas wrote: > There's been a lot of churn in hot standby since the beginning of the > commitfest, so I thought it would be good to summarize where we are. > > Attached is the latest and greatest patch against CVS head, taken from > the hs-riggs bra

Re: [HACKERS] Hot standby, xlog_xact_assignment and unreported subxids.

2009-10-14 Thread Simon Riggs
On Wed, 2009-10-14 at 14:43 +0300, Heikki Linnakangas wrote: > In the hot standby patch, we have this comment in procarray.c: > > > It is > > * important that the XLOG_XACT_ASSIGNMENT record contain *all* subxids > > * not just those so far unreported because the sole purpose is to ensure > >

[HACKERS] Hot standby, xlog_xact_assignment and unreported subxids.

2009-10-14 Thread Heikki Linnakangas
In the hot standby patch, we have this comment in procarray.c: > It is > * important that the XLOG_XACT_ASSIGNMENT record contain *all* subxids > * not just those so far unreported because the sole purpose is to ensure > * we can remove the xids from KnownAssignedXids. As the patch stands, t

[HACKERS] Hot Standby and 64+ subxids (was COPY enhancements)

2009-10-12 Thread Simon Riggs
On Thu, 2009-10-08 at 11:50 -0400, Tom Lane wrote: > Robert Haas writes: > > Subcommitting every single row is going to be really painful, > > especially after Hot Standby goes in and we have to issue a WAL record > > after every 64 subtransactions (AIUI). > > Yikes ... I had not been following t

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Bruce Momjian
Andrew Dunstan wrote: > > > Robert Haas wrote: > > But at least for simple features I think that there would be a value > > in separating the patch author's work from the committer's adjustments. > > > > > > That is just going to make life harder for committers. > > There are plenty of things

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Joshua D. Drake
On Fri, 2009-10-09 at 14:05 -0400, Andrew Dunstan wrote: > > Robert Haas wrote: > > But at least for simple features I think that there would be a value > > in separating the patch author's work from the committer's adjustments. > > > > > > That is just going to make life harder for committers.

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Andrew Dunstan
Robert Haas wrote: But at least for simple features I think that there would be a value in separating the patch author's work from the committer's adjustments. That is just going to make life harder for committers. There are plenty of things with my name on them that are not exactly what

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Robert Haas
On Oct 9, 2009, at 1:21 PM, Bruce Momjian wrote: Simon Riggs wrote: You have posted patches that I have said I don't agree with. My name is going to be on this when it goes in, so I don't think it makes any sense to force that commit to include changes I don't agree with. I cannot prevent

Re: [HACKERS] Hot Standby on git

2009-10-09 Thread Heikki Linnakangas
Simon Riggs wrote: >> Tom Lane wrote: >>> [ scratches head ... ] Why is hot standby messing with this sort of >>> thing at all? It sounds like a performance optimization that should >>> be considered separately, and *later*. >> Yeah, I too considered just ripping it out. Simon is worried that >>

Re: [HACKERS] Hot Standby on git

2009-10-09 Thread Heikki Linnakangas
While playing with conflict resolution, I bumped into this: postgres=# begin ISOLATION LEVEL SERIALIZABLE; BEGIN postgres=# SELECT * FROM foo; id | data +-- 12 | (1 row) postgres=# SELECT * FROM foo; id | data +-- 12 | (1 row) postgres=# SELECT * FROM foo; id | data +---

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Bruce Momjian
Simon Riggs wrote: > You have posted patches that I have said I don't agree with. My name is > going to be on this when it goes in, so I don't think it makes any sense > to force that commit to include changes I don't agree with. I cannot > prevent you making changes afterwards, nor would I wish to

Re: [HACKERS] Hot Standby 0.2.1

2009-10-08 Thread Simon Riggs
On Thu, 2009-10-08 at 12:49 +0100, Simon Riggs wrote: > I'll get cracking on some changes. This will probably be next week now, just in case you're wondering when I'll start adding patches. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@po

Re: [HACKERS] Hot Standby 0.2.1

2009-10-08 Thread Simon Riggs
On Thu, 2009-10-08 at 07:33 -0400, Heikki Linnakangas wrote: > Simon Riggs wrote: > > I would still like you to make a clear statement that the contents of > > that repository are BSD licenced open source contributions. > > Ok. All the content in the repository at > git://git.postgresql.org/git/u

Re: [HACKERS] Hot Standby 0.2.1

2009-10-08 Thread Heikki Linnakangas
Simon Riggs wrote: > I would still like you to make a clear statement that the contents of > that repository are BSD licenced open source contributions. Ok. All the content in the repository at git://git.postgresql.org/git/users/heikki/postgres.git is released under the same BSD license as Postgre

Re: [HACKERS] Hot Standby 0.2.1

2009-10-08 Thread Simon Riggs
On Wed, 2009-10-07 at 23:19 -0400, Heikki Linnakangas wrote: > Heikki Linnakangas wrote: > > I've made public the version I'm working on. That's the version I'm > > ultimately going to commit. It would be a lot more helpful if you > > provided these patches over that version. Otherwise I have to r

Re: [HACKERS] Hot Standby 0.2.1

2009-10-07 Thread Heikki Linnakangas
Heikki Linnakangas wrote: > I've made public the version I'm working on. That's the version I'm > ultimately going to commit. It would be a lot more helpful if you > provided these patches over that version. Otherwise I have to refactor > them over that codebase, possibly introducing new bugs. Act

Re: [HACKERS] Hot Standby 0.2.1

2009-10-07 Thread Heikki Linnakangas
Simon Riggs wrote: > On Tue, 2009-09-22 at 12:53 +0300, Heikki Linnakangas wrote: >> It looks like the standby tries to remove XID 4323 from the >> known-assigned hash table, but it's not there because it was removed >> and set in pg_subtrans by an XLOG_XACT_ASSIGNMENT record earlier. I >> guess we

Re: [HACKERS] Hot Standby 0.2.1

2009-10-07 Thread Simon Riggs
On Wed, 2009-09-23 at 12:07 +0300, Heikki Linnakangas wrote: > we need be careful to avoid putting any extra work into the normal > recovery path. Otherwise bugs in hot standby related code can cause > crash recovery to fail. Re-checked code and found a couple of additional places that needed tes

Re: [HACKERS] Hot Standby 0.2.1

2009-10-07 Thread Simon Riggs
On Tue, 2009-09-22 at 12:53 +0300, Heikki Linnakangas wrote: > It looks like the standby tries to remove XID 4323 from the > known-assigned hash table, but it's not there because it was removed > and set in pg_subtrans by an XLOG_XACT_ASSIGNMENT record earlier. I > guess we should just not throw a

Re: [HACKERS] Hot Standby on git

2009-10-07 Thread Simon Riggs
On Fri, 2009-10-02 at 10:32 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > I will docuemnt the recommendation to set max_standby_delay = 0 if > > performing an archive recovery (and explain why). > > Hmm, not sure if that's such a good piece of advice either. It will mean > waiting for

Re: [HACKERS] Hot Standby on git

2009-10-06 Thread Simon Riggs
On Thu, 2009-10-01 at 18:47 +0300, Heikki Linnakangas wrote: > And if you could please review the changes I've been doing, just to > make sure I haven't inadvertently introduced new bugs. That has > happened before, as you've rightfully reminded me :-). You posted 17 patches here. I've reviewed

Re: [HACKERS] Hot Standby on git

2009-10-06 Thread Simon Riggs
On Sun, 2009-09-27 at 13:57 +0300, Heikki Linnakangas wrote: > Per Simon's request, for the benefit of the archive, here's all the > changes I've done on the patch since he posted the initial version for > review for this commitfest as incremental patches. This is extracted > from my git reposito

Re: [HACKERS] Hot Standby on git

2009-10-06 Thread Simon Riggs
On Tue, 2009-10-06 at 01:10 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > We discussed briefly your change > > 0011-Replace-per-proc-counters-of-loggable-locks-with-per.patch. > > > > I don't see how that helps at all. The objective of lock counters was to > > know if we can skip acq

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Simon Riggs
On Mon, 2009-10-05 at 18:30 -0400, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Mon, 2009-09-28 at 11:25 +0300, Heikki Linnakangas wrote: > >> Heikki Linnakangas wrote: > >>> Per Simon's request, for the benefit of the archive, here's all the > >>> changes I've done on the patch since he

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Heikki Linnakangas
Simon Riggs wrote: > On Mon, 2009-09-28 at 11:25 +0300, Heikki Linnakangas wrote: >> Heikki Linnakangas wrote: >>> Per Simon's request, for the benefit of the archive, here's all the >>> changes I've done on the patch since he posted the initial version for >>> review for this commitfest as increme

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Heikki Linnakangas
Simon Riggs wrote: > We discussed briefly your change > 0011-Replace-per-proc-counters-of-loggable-locks-with-per.patch. > > I don't see how that helps at all. The objective of lock counters was to > know if we can skip acquiring an LWlock on all lock partitions. This > change keeps the lock coun

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Simon Riggs
On Mon, 2009-10-05 at 10:19 -0400, Tom Lane wrote: > Simon Riggs writes: > > I don't see how that helps at all. The objective of lock counters was to > > know if we can skip acquiring an LWlock on all lock partitions. This > > change keeps the lock counters yet acquires the locks we were trying t

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Tom Lane
Simon Riggs writes: > I don't see how that helps at all. The objective of lock counters was to > know if we can skip acquiring an LWlock on all lock partitions. This > change keeps the lock counters yet acquires the locks we were trying to > avoid. This change needs some justification since it is

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Simon Riggs
On Sun, 2009-09-27 at 13:57 +0300, Heikki Linnakangas wrote: > Per Simon's request, for the benefit of the archive, here's all the > changes I've done on the patch since he posted the initial version for > review for this commitfest as incremental patches. This is extracted > from my git repositor

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Simon Riggs
On Mon, 2009-09-28 at 11:25 +0300, Heikki Linnakangas wrote: > Heikki Linnakangas wrote: > > Per Simon's request, for the benefit of the archive, here's all the > > changes I've done on the patch since he posted the initial version for > > review for this commitfest as incremental patches. This is

<    1   2   3   4   5   6   7   8   9   10   >