Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 10:04 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: @@ -7061,6 +7061,15 @@ ShutdownXLOG(int code, Datum arg) else { /* +* Take a snapshot of running transactions and write this to WAL. +* This allows us to reconstruct the

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-10-02 at 10:04 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: @@ -7061,6 +7061,15 @@ ShutdownXLOG(int code, Datum arg) else { /* +* Take a snapshot of running transactions and write this to WAL. +* This allows us to

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 10:32 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I will docuemnt the recommendation to set max_standby_delay = 0 if performing an archive recovery (and explain why). Hmm, not sure if that's such a good piece of advice either. It will mean waiting for

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 10:43 +0300, Heikki Linnakangas wrote: It seems dangerous to write a WAL record after the shutdown checkpoint. It will be overwritten by subsequent startup, which is a recipe for trouble. I've said its a corner case and not worth spending time on. I'm putting it in at

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-10-02 at 10:43 +0300, Heikki Linnakangas wrote: It seems dangerous to write a WAL record after the shutdown checkpoint. It will be overwritten by subsequent startup, which is a recipe for trouble. I've said its a corner case and not worth spending time on.

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 11:26 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2009-10-02 at 10:43 +0300, Heikki Linnakangas wrote: It seems dangerous to write a WAL record after the shutdown checkpoint. It will be overwritten by subsequent startup, which is a recipe for

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 11:26 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2009-10-02 at 10:43 +0300, Heikki Linnakangas wrote: It seems dangerous to write a WAL record after the shutdown checkpoint. It will be overwritten by subsequent startup, which is a recipe for

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: I'd rather just skip this for now. It's a minor case anyway and there's nothing stopping writing their own RunningXactData records with a function, if it is needed. I can add a function for that. That won't help. There's no way to have it in a right place wrt. the shutdown

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 13:52 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I'd rather just skip this for now. It's a minor case anyway and there's nothing stopping writing their own RunningXactData records with a function, if it is needed. I can add a function for that. That won't

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: I will add code to make a shutdown checkpoint be a valid starting place for Hot Standby, as long as there are no in-doubt prepared transactions. That way we know there are no xids still running and no locks, without needing to write a record to say so. Ok, I can live with

Re: [HACKERS] Hot Standby 0.2.1

2009-10-02 Thread Simon Riggs
On Sun, 2009-09-27 at 14:59 +0300, Heikki Linnakangas wrote: The locking in smgr_redo_commit and smgr_redo_abort doesn't look right. First of all, smgr_redo_abort is not holding XidGenLock and ProcArrayLock while modifying ShmemVariableCache-nextXid and ShmemVariableCache-latestCompletedXid,

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 18:04 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I will add code to make a shutdown checkpoint be a valid starting place for Hot Standby, as long as there are no in-doubt prepared transactions. That way we know there are no xids still running and no locks,

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Heikki Linnakangas
+ /* +* If our initial RunningXactData had an overflowed snapshot then we +* knew we were missing some subxids from our snapshot. We can use +* this data as an initial snapshot, but we cannot yet mark it valid. +* We know that the missing subxids are equal to or earlier than +

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Wed, 2009-09-30 at 18:45 +0300, Heikki Linnakangas wrote: Regarding this in InitStandbyDelayTimers: + /* +* If replication delay is enormously huge, just treat that as +* zero and work up from there. This prevents us from acting +* foolishly when replaying old log files. +

Re: [HACKERS] Hot Standby 0.2.1

2009-10-01 Thread Simon Riggs
On Sun, 2009-09-27 at 15:35 +0300, Heikki Linnakangas wrote: TransactionIdIsInProgress() doesn't consult the known-assigned-xids structure. That's a problem: in the standby, TransactionIdIsInProgress() can return false for a transaction that is still running in the master. HeapTupleSatisfies*

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Thu, 2009-10-01 at 14:29 +0300, Heikki Linnakangas wrote: + /* +* If our initial RunningXactData had an overflowed snapshot then we +* knew we were missing some subxids from our snapshot. We can use +* this data as an initial snapshot, but we cannot yet mark it valid. +*

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Wed, 2009-09-30 at 09:33 +0300, Heikki Linnakangas wrote: Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquire first before changing anything. Why? Looking at the other functions in that

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-09-30 at 09:33 +0300, Heikki Linnakangas wrote: Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquire first before changing anything. Why? Looking at the other

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Heikki Linnakangas
Simon Riggs wrote: Hmm, yes. ISTM that I'm still calculating latestRunningXid the old way while assuming it is calculated the new way. The new way is just to grab nextXid since we have XidGenLock and do TransactionIdRetreat() on it. Ok, good, that's what I thought too. I'll fix that. --

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Thu, 2009-10-01 at 18:47 +0300, Heikki Linnakangas wrote: Could you look into these two TODO items you listed on the wiki page: Unless we agree otherwise, if its listed on the Wiki page then I will work on it. Maybe not as when you might like it, but I am working through the list. 5 new

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Thu, 2009-10-01 at 18:48 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: Hmm, yes. ISTM that I'm still calculating latestRunningXid the old way while assuming it is calculated the new way. The new way is just to grab nextXid since we have XidGenLock and do TransactionIdRetreat() on

Re: [HACKERS] Hot Standby on git

2009-09-30 Thread Heikki Linnakangas
Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquire first before changing anything. Why? Looking at the other functions in that file, all others that access both files are happy to acquire one

Re: [HACKERS] Hot Standby on git

2009-09-30 Thread Heikki Linnakangas
Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquired first before changing anything. Why? Looking at the other functions in that file, all others that access both files are happy to acquire one

Re: [HACKERS] Hot Standby on git

2009-09-30 Thread Heikki Linnakangas
Regarding this in InitStandbyDelayTimers: + /* +* If replication delay is enormously huge, just treat that as +* zero and work up from there. This prevents us from acting +* foolishly when replaying old log files. +*/ + if (*currentDelay_ms 0) + *currentDelay_ms = 0; +

Re: [HACKERS] Hot Standby on git

2009-09-28 Thread Heikki Linnakangas
Heikki Linnakangas wrote: Per Simon's request, for the benefit of the archive, here's all the changes I've done on the patch since he posted the initial version for review for this commitfest as incremental patches. This is extracted from my git repository at

Re: [HACKERS] Hot Standby on git

2009-09-27 Thread Heikki Linnakangas
Per Simon's request, for the benefit of the archive, here's all the changes I've done on the patch since he posted the initial version for review for this commitfest as incremental patches. This is extracted from my git repository at git://git.postgresql.org/git/users/heikki/postgres.git. --

Re: [HACKERS] Hot Standby 0.2.1

2009-09-27 Thread Heikki Linnakangas
The locking in smgr_redo_commit and smgr_redo_abort doesn't look right. First of all, smgr_redo_abort is not holding XidGenLock and ProcArrayLock while modifying ShmemVariableCache-nextXid and ShmemVariableCache-latestCompletedXid, respectively, like smgr_redo_commit is. Attached patch fixes that.

Re: [HACKERS] Hot Standby 0.2.1

2009-09-27 Thread Heikki Linnakangas
TransactionIdIsInProgress() doesn't consult the known-assigned-xids structure. That's a problem: in the standby, TransactionIdIsInProgress() can return false for a transaction that is still running in the master. HeapTupleSatisfies* functions can incorrectly set HEAP_XMIN/XMAX_INVALID hint bits

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Robert Haas
On Sat, Sep 26, 2009 at 5:49 AM, Simon Riggs si...@2ndquadrant.com wrote: Just a note to say that Hot Standby patch is now on git repository  git://git.postgresql.org/git/users/simon/postgres Branch name: hot_standby Awesome! Thanks for taking the time to get this set up. The complete

[HACKERS] Hot Standby on git

2009-09-26 Thread Simon Riggs
Just a note to say that Hot Standby patch is now on git repository git://git.postgresql.org/git/users/simon/postgres Branch name: hot_standby The complete contents of that repository are BSD licenced contributions to the PostgreSQL project. Any further changes to that will be by agreement

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Simon Riggs
On Sat, 2009-09-26 at 09:29 -0400, Robert Haas wrote: I estimate that making the remaining changes noted on the Wiki and fully testing them will take at least 2 weeks. Gabriele Bartolini is assisting in this area, though neither of us are able to work full time on this. We still have

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Mark Mielke
On 09/26/2009 10:04 AM, Simon Riggs wrote: If you think there's something useful I could do, let me know and I'll take a look. I feel like I need a better way of unit testing new code. Some of the code in the patch is to handle corner cases, so recreating them is fairly hard. It is a

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Dan Colish
On Sat, Sep 26, 2009 at 10:45:17AM -0400, Mark Mielke wrote: On 09/26/2009 10:04 AM, Simon Riggs wrote: If you think there's something useful I could do, let me know and I'll take a look. I feel like I need a better way of unit testing new code. Some of the code in the patch is to handle

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Josh Berkus
I feel like I need a better way of unit testing new code. Some of the code in the patch is to handle corner cases, so recreating them is fairly hard. It is a nagging feeling that I am missing some knowledge here and would welcome some insight, or research, into better ways of doing general

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Mark Mielke
On 09/26/2009 02:28 PM, Dan Colish wrote: There are a variety of projects dedicated to creating C unit test frameworks. I don't have a lot of experience with them, but I have heard good things about check and cunit. Here's a link I found with a longer list of frameworks.

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread David E. Wheeler
On Sep 26, 2009, at 12:33 PM, Josh Berkus wrote: There's always pgtap. Whenever we find a new corner case, we add it to the development test suite. Also, for C TAP, there's [libtap](http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap ). You can then use `prove` which you likely already

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Alvaro Herrera
Mark Mielke escribió: Most real life code gets a little more complicated. For example, what if we want to simulate a network failure or out of disk space condition? What if we want to test out what happens when the Y2038 date is reached? This requires either complex test case setup that is

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Heikki Linnakangas
Looking at the startup sequence now. I see that you modified ExtendSUBTRANS so that it doesn't wipe out previously set values if it's called with out-of-order xids. I guess that works, although I think it can leave pages unzeroed if it's called with a large enough gap between xids, so that the

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Thu, 2009-09-24 at 13:33 +0300, Heikki Linnakangas wrote: Heikki Linnakangas wrote: The problem becomes a lot easier if we accept that it's OK to have a lock included in the running-xacts snapshot and also appear in a XLOG_RELATION_LOCK record later. The standby should handle that

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Wed, 2009-09-23 at 19:07 +0300, Heikki Linnakangas wrote: Rather than keep the numHeldLocks counters per-proc in proc array, I think it would be simpler to have a single (or one per lock partition) counter in shared memory in lock.c. It's just an optimization to make it faster to find out

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Wed, 2009-09-23 at 17:45 +0300, Heikki Linnakangas wrote: Heikki Linnakangas wrote: Simon Riggs wrote: On Wed, 2009-09-23 at 11:13 +0300, Heikki Linnakangas wrote: I note that we don't emit RunningXacts after a shutdown checkpoint. So if recovery starts at a shutdown checkpoint, we

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-09-23 at 19:07 +0300, Heikki Linnakangas wrote: Rather than keep the numHeldLocks counters per-proc in proc array, I think it would be simpler to have a single (or one per lock partition) counter in shared memory in lock.c. It's just an optimization to make it

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Fri, 2009-09-25 at 11:49 +0300, Heikki Linnakangas wrote: Looking at the startup sequence now. I see that you modified ExtendSUBTRANS so that it doesn't wipe out previously set values if it's called with out-of-order xids. I guess that works, although I think it can leave pages unzeroed

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Fri, 2009-09-25 at 13:23 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Wed, 2009-09-23 at 17:45 +0300, Heikki Linnakangas wrote: XactClearRecoveryTransactions() when we see a shutdown checkpoint, which clears all recovery locks. But doesn't that prematurely clear all locks

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Heikki Linnakangas
Simon Riggs wrote: Definitely need to cope with them for Hot Standby. My point was general one to say that behaviour is very non-useful for users with prepared transactions. It just causes manual effort by a DBA each time the system is shutdown. The transaction manager is supposed to

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-09-23 at 17:45 +0300, Heikki Linnakangas wrote: XactClearRecoveryTransactions() when we see a shutdown checkpoint, which clears all recovery locks. But doesn't that prematurely clear all locks belonging to prepared transactions as well? Much better to read

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Fri, 2009-09-25 at 13:14 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Wed, 2009-09-23 at 19:07 +0300, Heikki Linnakangas wrote: Rather than keep the numHeldLocks counters per-proc in proc array, I think it would be simpler to have a single (or one per lock partition)

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Heikki Linnakangas
In XidInMVCCSnapshot: if (snapshot-takenDuringRecovery) { /* * If the snapshot contains full subxact data, the fastest way to check * things is just to compare the given XID against both subxact XIDs and * top-level

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Fri, 2009-09-25 at 15:50 +0300, Heikki Linnakangas wrote: Hang on, isn't this 180 degrees backwards? A witty riposte escapes me; yes, the if test is !correct. What I find amazing is that it passed the test where I put it doing make installcheck in an infinite loop for a long time. I guess

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Alvaro Herrera
Simon Riggs wrote: What I find amazing is that it passed the test where I put it doing make installcheck in an infinite loop for a long time. I guess that means the regression tests hardly touch the concurrency code at all, which now I think about it makes sense but I still find that very

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Fri, 2009-09-25 at 16:30 -0400, Alvaro Herrera wrote: Simon Riggs wrote: What I find amazing is that it passed the test where I put it doing make installcheck in an infinite loop for a long time. I guess that means the regression tests hardly touch the concurrency code at all, which

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Andrew Dunstan
Alvaro Herrera wrote: Simon Riggs wrote: What I find amazing is that it passed the test where I put it doing make installcheck in an infinite loop for a long time. I guess that means the regression tests hardly touch the concurrency code at all, which now I think about it makes sense but

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Alvaro Herrera
Andrew Dunstan wrote: Alvaro Herrera wrote: Try installcheck-parallel, which should be a bit better. It's probably not yet good enough though because it always runs the same tests concurrently. It is also quite easy to set up your own schedule. ... except you have to be careful with

Re: [HACKERS] Hot Standby 0.2.1

2009-09-24 Thread Heikki Linnakangas
Heikki Linnakangas wrote: The problem becomes a lot easier if we accept that it's OK to have a lock included in the running-xacts snapshot and also appear in a XLOG_RELATION_LOCK record later. The standby should handle that gracefully already. If we just remove RecoveryInfoLock, that can

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Heikki Linnakangas
Looking at the way cache invalidations are handled in two-phase transactions, it would be simpler if we store the shared cache invalidation messages in the twophase state file header, like we store deleted relations and subxids. This allows them to be copied to the COMMIT_PREPARED WAL record, so

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Heikki Linnakangas
The logic in the lock manager to track the number of held AccessExclusiveLocks (with ProcArrayIncrementNumHeldLocks and ProcArrayDecrementNumHeldLocks) seems to be broken. I added an Assertion into ProcArrayDecrementNumHeldLocks: --- a/src/backend/storage/ipc/procarray.c +++

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Simon Riggs
On Wed, 2009-09-23 at 11:13 +0300, Heikki Linnakangas wrote: Looking at the way cache invalidations are handled in two-phase transactions, it would be simpler if we store the shared cache invalidation messages in the twophase state file header, like we store deleted relations and subxids.

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Simon Riggs
On Wed, 2009-09-23 at 12:07 +0300, Heikki Linnakangas wrote: it highlights that we need be careful to avoid putting any extra work into the normal recovery path. Otherwise bugs in hot standby related code can cause crash recovery to fail. Excellent point. I will put in additional protective

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Simon Riggs
On Wed, 2009-09-23 at 12:07 +0300, Heikki Linnakangas wrote: seems to be broken Agreed. Patch withdrawn for correction and rework. Nothing serious, but not much point doing further testing to all current issues resolved. Tracking of issues raised and later solved via Wiki. -- Simon Riggs

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-09-23 at 11:13 +0300, Heikki Linnakangas wrote: I note that we don't emit RunningXacts after a shutdown checkpoint. So if recovery starts at a shutdown checkpoint, we don't let read-only backends in until the first online checkpoint. Could we treat a shutdown

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Heikki Linnakangas
Heikki Linnakangas wrote: Simon Riggs wrote: On Wed, 2009-09-23 at 11:13 +0300, Heikki Linnakangas wrote: I note that we don't emit RunningXacts after a shutdown checkpoint. So if recovery starts at a shutdown checkpoint, we don't let read-only backends in until the first online checkpoint.

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Jeff Janes
On Tue, Sep 22, 2009 at 10:02 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Heikki Linnakangas escribió: Simon Riggs wrote: On Mon, 2009-09-21 at 19:42 -0700, Jeff Janes wrote: jjanes=# begin; BEGIN jjanes=# lock table pgbench_history in access exclusive mode; LOCK TABLE

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes: Unfortunately, isolation level serializable is not truly serializable. Usually it is good enough, but when it isn't good enough and you need an explicit table lock (a very rare but not nonexistent situation), I think it should either lock the table in

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-09-23 at 12:07 +0300, Heikki Linnakangas wrote: seems to be broken Agreed. Looking at the relation lock stuff a bit more... When someone tries to acquire an AccessExclusiveLock, but can't get it immediately, we sleep while holding RecoveryInfoLock. That

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Josh Berkus
Simon, Patch withdrawn for correction and rework. Nothing serious, but not much point doing further testing to all current issues resolved. :-( Good thing we went for 4 CFs. Is there a GIT branch of Simon's current working version up somewhere? -- Josh Berkus PostgreSQL Experts Inc.

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Heikki Linnakangas
Josh Berkus wrote: Patch withdrawn for correction and rework. Nothing serious, but not much point doing further testing to all current issues resolved. :-( Good thing we went for 4 CFs. I think we should try to hammer this in in this commitfest. None of the issues found this far are too

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Josh Berkus
Heikki, I think we should try to hammer this in in this commitfest. None of the issues found this far are too serious, nothing that requires major rewrites. It would certainly be valuable to get users testing it starting with Alpha2 instead of waiting 2 months. -- Josh Berkus PostgreSQL

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Heikki Linnakangas
Simon Riggs wrote: On Mon, 2009-09-21 at 14:01 +0100, Simon Riggs wrote: On Mon, 2009-09-21 at 13:50 +0300, Heikki Linnakangas wrote: is this that we seem to be missing conflict resolution for GiST index tuples deleted by the kill_prior_tuples mechanism. Unless I'm missing something, we

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Simon Riggs
On Mon, 2009-09-21 at 19:42 -0700, Jeff Janes wrote: On Tue, Sep 15, 2009 at 2:41 PM, Simon Riggs si...@2ndquadrant.com wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1 Patch against CVS HEAD (now):

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Heikki Linnakangas
Simon Riggs wrote: On Mon, 2009-09-21 at 19:42 -0700, Jeff Janes wrote: jjanes=# begin; BEGIN jjanes=# lock table pgbench_history in access exclusive mode; LOCK TABLE jjanes=# select count(*) from pgbench_history; count 519104 (1 row) jjanes=# select count(*) from

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Simon Riggs
On Tue, 2009-09-22 at 11:04 +0300, Heikki Linnakangas wrote: By me, yes. WAL replay does not require a table lock to progress. Any changes are protected with block-level locks. It does acquire a table lock and cancel conflicting queries when it is about to replay something that would

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Heikki Linnakangas
In testing, it looks like there's still something wrong with the subtransaction handling. I created a test function to create a large number of subtransactions: CREATE LANGUAGE plpgsql; CREATE TABLE bar (id int4); CREATE OR REPLACE FUNCTION subxids (n integer) RETURNS void LANGUAGE plpgsql AS $$

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Simon Riggs
On Tue, 2009-09-22 at 12:53 +0300, Heikki Linnakangas wrote: In testing, it looks like there's still something wrong with the subtransaction handling. I created a test function to create a large number of subtransactions: OK, looking at this now. Thanks for the report. -- Simon Riggs

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Simon Riggs
On Tue, 2009-09-22 at 12:53 +0300, Heikki Linnakangas wrote: It looks like the standby tries to remove XID 4323 from the known-assigned hash table, but it's not there because it was removed and set in pg_subtrans by an XLOG_XACT_ASSIGNMENT record earlier. I guess we should just not throw an

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Simon Riggs
On Tue, 2009-09-22 at 12:53 +0300, Heikki Linnakangas wrote: In testing, it looks like there's still something wrong with the subtransaction handling. I created a test function to create a large number of subtransactions: CREATE LANGUAGE plpgsql; CREATE TABLE bar (id int4); CREATE OR

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Alvaro Herrera
Heikki Linnakangas escribió: Simon Riggs wrote: On Mon, 2009-09-21 at 19:42 -0700, Jeff Janes wrote: jjanes=# begin; BEGIN jjanes=# lock table pgbench_history in access exclusive mode; LOCK TABLE jjanes=# select count(*) from pgbench_history; count 519104 (1 row)

Re: [HACKERS] Hot Standby 0.2.1

2009-09-22 Thread Bruce Momjian
Simon Riggs wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1 Patch against CVS HEAD (now): clean apply, compile, no known bugs. OVERVIEW Anyone who is interested in how the hot standby behaves should

Re: [HACKERS] Hot Standby 0.2.1

2009-09-21 Thread Heikki Linnakangas
Simon Riggs wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1 Patch against CVS HEAD (now): clean apply, compile, no known bugs. Thanks! Attached is some minor comment and fixes, and some dead code removal.

Re: [HACKERS] Hot Standby 0.2.1

2009-09-21 Thread Simon Riggs
On Mon, 2009-09-21 at 13:50 +0300, Heikki Linnakangas wrote: The only bug I've found ! is this that we seem to be missing conflict resolution for GiST index tuples deleted by the kill_prior_tuples mechanism. Unless I'm missing something, we need similar handling there that we have in

Re: [HACKERS] Hot Standby 0.2.1

2009-09-21 Thread Simon Riggs
On Mon, 2009-09-21 at 13:50 +0300, Heikki Linnakangas wrote: The documentation talks about setting and checking default_transaction_read_only, but I think it doesn't say anything about transaction_read_only, which I find odd. This in particular: Users will be able to tell whether their

Re: [HACKERS] Hot Standby 0.2.1

2009-09-21 Thread Simon Riggs
On Mon, 2009-09-21 at 14:01 +0100, Simon Riggs wrote: On Mon, 2009-09-21 at 13:50 +0300, Heikki Linnakangas wrote: is this that we seem to be missing conflict resolution for GiST index tuples deleted by the kill_prior_tuples mechanism. Unless I'm missing something, we need similar

Re: [HACKERS] Hot Standby 0.2.1

2009-09-21 Thread Robert Haas
On Mon, Sep 21, 2009 at 9:01 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2009-09-21 at 13:50 +0300, Heikki Linnakangas wrote: The only bug I've found ! Yeah, wow. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Hot Standby 0.2.1

2009-09-21 Thread Bruce Momjian
Simon Riggs wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1 Patch against CVS HEAD (now): clean apply, compile, no known bugs. Wow, great! Simon has allowed us to pass a great milestone in Postgres

Re: [HACKERS] Hot Standby 0.2.1

2009-09-21 Thread Jeff Janes
On Tue, Sep 15, 2009 at 2:41 PM, Simon Riggs si...@2ndquadrant.com wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1 Patch against CVS HEAD (now): clean apply, compile, no known bugs. OVERVIEW You can

Re: [HACKERS] Hot Standby 0.2.1

2009-09-18 Thread Simon Riggs
On Thu, 2009-09-17 at 19:01 -0400, Robert Haas wrote: I'm going to put the index-only scans aside for now to focus on hot standby and streaming replication. Both are big patches, so there's plenty of work in those two alone, and not only for me. What is the best way to attack this?

Re: [HACKERS] Hot Standby 0.2.1

2009-09-18 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: Also, stepping back from me personally, should we try to assign some additional reviewers to these patches? Is there some way we can divide up review tasks among multiple people so that we're not repeating each others work? Thoughts appreciated,

Re: [HACKERS] Hot Standby 0.2.1

2009-09-18 Thread Jeff Janes
On Tue, Sep 15, 2009 at 2:41 PM, Simon Riggs si...@2ndquadrant.com wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1 Patch against CVS HEAD (now): clean apply, compile, no known bugs. Hi Simon, Is there

Re: [HACKERS] Hot Standby 0.2.1

2009-09-18 Thread Simon Riggs
On Fri, 2009-09-18 at 07:23 -0700, Jeff Janes wrote: On Tue, Sep 15, 2009 at 2:41 PM, Simon Riggs si...@2ndquadrant.com wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1

Re: [HACKERS] Hot Standby 0.2.1

2009-09-18 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Fri, 2009-09-18 at 07:23 -0700, Jeff Janes wrote: Is there a reason that you remove the WAL_DEBUG shown below? WAL_DEBUG is not removed by the patch, though that section of code is removed, as you observe. I recall an earlier bug report by

Re: [HACKERS] Hot Standby 0.2.1

2009-09-18 Thread Simon Riggs
On Fri, 2009-09-18 at 11:14 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Fri, 2009-09-18 at 07:23 -0700, Jeff Janes wrote: Is there a reason that you remove the WAL_DEBUG shown below? WAL_DEBUG is not removed by the patch, though that section of code is removed,

Re: [HACKERS] Hot Standby 0.2.1

2009-09-18 Thread Marcos Luis Ortiz Valmaseda
Enviados: Jueves, 17 de Septiembre 2009 20:53:24 GMT -10:00 Hawai Asunto: Re: [HACKERS] Hot Standby 0.2.1 On Thu, 2009-09-17 at 19:01 -0400, Robert Haas wrote: I'm going to put the index-only scans aside for now to focus on hot standby and streaming replication. Both are big patches, so there's

Re: [HACKERS] Hot Standby 0.2.1

2009-09-17 Thread Heikki Linnakangas
Robert Haas wrote: On Wed, Sep 16, 2009 at 6:05 PM, Josh Berkus j...@agliodbs.com wrote: Now that Simon has submitted this, can some of the heavy-hitters here review it? Heikki? Nobody's name is next to it. I don't think anyone is planning to ignore this patch, but it wasn't included in

Re: [HACKERS] Hot Standby 0.2.1

2009-09-17 Thread Simon Riggs
On Thu, 2009-09-17 at 09:54 +0300, Heikki Linnakangas wrote: This is a pretty small CommitFest, so there shouldn't be any shortage of reviewers, though Heikki's time may be stretched a little thin, since Streaming Replication is also in the queue, and he is working on index-only scans.

Re: [HACKERS] Hot Standby 0.2.1

2009-09-17 Thread Robert Haas
On Thu, Sep 17, 2009 at 2:54 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Robert Haas wrote: On Wed, Sep 16, 2009 at 6:05 PM, Josh Berkus j...@agliodbs.com wrote: Now that Simon has submitted this, can some of the heavy-hitters here review it?  Heikki? Nobody's name is

Re: [HACKERS] Hot Standby 0.2.1

2009-09-16 Thread Bernd Helmle
--On 15. September 2009 22:41:59 +0100 Simon Riggs si...@2ndquadrant.com wrote: http://wiki.postgresql.org/images/0/01/Hot_Standby_Recovery_Functions.pdf This doesn't work for me, it seems the correct link is http://wiki.postgresql.org/images/1/10/Hot_Standby_Recovery_Functions.pdf ?

Re: [HACKERS] Hot Standby 0.2.1

2009-09-16 Thread Josh Berkus
All, Now that Simon has submitted this, can some of the heavy-hitters here review it? Heikki? Nobody's name is next to it. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Hot Standby 0.2.1

2009-09-16 Thread Robert Haas
On Wed, Sep 16, 2009 at 6:05 PM, Josh Berkus j...@agliodbs.com wrote: Now that Simon has submitted this, can some of the heavy-hitters here review it?  Heikki? Nobody's name is next to it. I don't think anyone is planning to ignore this patch, but it wasn't included in the first round of

Re: [HACKERS] Hot Standby 0.2.1

2009-09-15 Thread David Fetter
On Tue, Sep 15, 2009 at 10:41:59PM +0100, Simon Riggs wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1 Patch against CVS HEAD (now): clean apply, compile, no known bugs. Kudos Cheers, David. --

Re: [HACKERS] Hot Standby, max_connections and max_prepared_transactions

2009-09-04 Thread Simon Riggs
On Fri, 2009-09-04 at 01:25 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Thu, 2009-09-03 at 22:22 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I propose we just accept that both max_connections and max_prepared_transactions need to be set correctly for

Re: [HACKERS] Hot Standby, max_connections and max_prepared_transactions

2009-09-04 Thread Robert Haas
On Thu, Sep 3, 2009 at 5:50 PM, Simon Riggssi...@2ndquadrant.com wrote: On Thu, 2009-09-03 at 22:22 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I propose we just accept that both max_connections and max_prepared_transactions need to be set correctly for recovery to work. This will

<    1   2   3   4   5   6   7   8   9   10   >