Re: [HACKERS] Hot Standby and VACUUM FULL

2010-02-01 Thread Simon Riggs
On Mon, 2010-02-01 at 10:06 -0500, Tom Lane wrote: > the assumption that the file is less than one disk block, > it should be just as atomic as pg_control updates are. IIRC there were 173 relations affected by this. 4 bytes each we would have more than 512 bytes. ISTM you need to treat some of t

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-02-01 Thread Tom Lane
Greg Stark writes: > On Mon, Feb 1, 2010 at 8:54 AM, Simon Riggs wrote: >>> Disallow catalog relocation inside subtransactions, to avoid having >>> to handle subxact abort effects on the local-map-changes state. >>> This could be implemented if desired, but doesn't seem worth it >>> at least in f

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-02-01 Thread Tom Lane
Heikki Linnakangas writes: > Tom Lane wrote: >> Once the updated map file is moved into place, the relocation is effectively >> committed even if we subsequently abort the transaction. We can make that >> window pretty narrow but not remove it completely. > We could include the instructions to u

Re: [HACKERS] Hot Standby and deadlock detection

2010-02-01 Thread Simon Riggs
On Mon, 2010-02-01 at 09:40 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > The way this would work is if Startup waits on a buffer pin we > > immediately send out a request to all backends to cancel themselves if > > they are holding the buffer pin required && waiting on a lock. We then

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-02-01 Thread Greg Stark
On Mon, Feb 1, 2010 at 8:54 AM, Simon Riggs wrote: >> Disallow catalog relocation inside subtransactions, to avoid having >> to handle subxact abort effects on the local-map-changes state. >> This could be implemented if desired, but doesn't seem worth it >> at least in first pass. > > Agreed, not

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-02-01 Thread Simon Riggs
On Sun, 2010-01-31 at 22:49 -0500, Tom Lane wrote: > Simon Riggs writes: > > I'll do a little work towards step (1) just so we can take a more > > informed view once you've had a better look at just what (2) involves. > > I spent a couple of hours reading code and believe that I've identified > a

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-02-01 Thread Heikki Linnakangas
Tom Lane wrote: > Hm ... do we want an LWLock per map file, or is one lock to rule them all > sufficient? LWLock per database seems problematic. With an HW lock there > wouldn't be a problem with that. HW lock would allow concurrent updates of > the map files of different DBs, but is that worth

Re: [HACKERS] Hot Standby and deadlock detection

2010-02-01 Thread Heikki Linnakangas
Simon Riggs wrote: > The way this would work is if Startup waits on a buffer pin we > immediately send out a request to all backends to cancel themselves if > they are holding the buffer pin required && waiting on a lock. We then > sleep until max_standby_delay. When max_standby_delay = -1 we only

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Bruce Momjian
FYI, getting rid of VACUUM FULL in a .0 release does make more sense than doing it in a .1 release. --- Tom Lane wrote: > Simon Riggs writes: > > I'll do a little work towards step (1) just so we can take a more > > informe

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Tom Lane
Simon Riggs writes: > I'll do a little work towards step (1) just so we can take a more > informed view once you've had a better look at just what (2) involves. I spent a couple of hours reading code and believe that I've identified all the pain points for allowing relocation of system catalogs (

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Simon Riggs
On Sun, 2010-01-31 at 15:14 -0500, Tom Lane wrote: > If the only benefit of getting rid of VACUUM FULL were simplifying > Hot Standby, I'd agree with you. But there are numerous other benefits. > The double-commit hack you mention is something we need to get rid of > for general system stability

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Simon Riggs
On Sun, 2010-01-31 at 15:14 -0500, Tom Lane wrote: > Simon Riggs writes: > > On Sun, 2010-01-31 at 14:35 -0500, Tom Lane wrote: > >> Anyway, it's still not apparent to me exactly what the connection is > >> between VACUUM FULL and Hot Standby. I remember that we said HS didn't > >> work with VACU

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Tom Lane
Simon Riggs writes: > On Sun, 2010-01-31 at 14:35 -0500, Tom Lane wrote: >> Anyway, it's still not apparent to me exactly what the connection is >> between VACUUM FULL and Hot Standby. I remember that we said HS didn't >> work with VACUUM FULL (INPLACE) but I don't recall why that is, and the [

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Simon Riggs
On Sun, 2010-01-31 at 14:44 -0500, Tom Lane wrote: > Simon Riggs writes: > > Having said that I now realise a few things I didn't before: > > > * Approach (2) effects the core of Postgres, even if you don't use Hot > > Standby. > > * I've had to remove 7 sanity checks to get the first few VACUUMs

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Simon Riggs
On Sun, 2010-01-31 at 14:35 -0500, Tom Lane wrote: > Simon Riggs writes: > > On Sat, 2010-01-30 at 15:17 -0500, Tom Lane wrote: > >> Simon Riggs writes: > >>> The options to do this were and still are: > >>> (1) Add WAL messages for non-transactional relcache invalidations > >>> (2) Allow system

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Tom Lane
Simon Riggs writes: > Having said that I now realise a few things I didn't before: > * Approach (2) effects the core of Postgres, even if you don't use Hot > Standby. > * I've had to remove 7 sanity checks to get the first few VACUUMs > working. ISTM that removing various basic checks in the code

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Tom Lane
Simon Riggs writes: > On Sat, 2010-01-30 at 15:17 -0500, Tom Lane wrote: >> Simon Riggs writes: >>> The options to do this were and still are: >>> (1) Add WAL messages for non-transactional relcache invalidations >>> (2) Allow system relations to be cluster-ed/vacuum full-ed. >> Refresh my memor

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Simon Riggs
On Sun, 2010-01-31 at 21:00 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Sat, 2010-01-30 at 15:17 -0500, Tom Lane wrote: > >> Simon Riggs writes: > >>> The last item on my list before close is making VACUUM FULL and Hot > >>> Standby play nicely together. > >>> The options to do th

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-31 Thread Heikki Linnakangas
Simon Riggs wrote: > On Sat, 2010-01-30 at 15:17 -0500, Tom Lane wrote: >> Simon Riggs writes: >>> The last item on my list before close is making VACUUM FULL and Hot >>> Standby play nicely together. >>> The options to do this were and still are: >>> (1) Add WAL messages for non-transactional rel

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-30 Thread Simon Riggs
On Sat, 2010-01-30 at 15:17 -0500, Tom Lane wrote: > Simon Riggs writes: > > The last item on my list before close is making VACUUM FULL and Hot > > Standby play nicely together. > > > The options to do this were and still are: > > > (1) Add WAL messages for non-transactional relcache invalidati

Re: [HACKERS] Hot Standby and VACUUM FULL

2010-01-30 Thread Tom Lane
Simon Riggs writes: > The last item on my list before close is making VACUUM FULL and Hot > Standby play nicely together. > The options to do this were and still are: > (1) Add WAL messages for non-transactional relcache invalidations > (2) Allow system relations to be cluster-ed/vacuum full-ed.

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-30 Thread Simon Riggs
On Fri, 2010-01-29 at 14:52 +, Greg Stark wrote: > Can you explain what it does in > more detail so we can understand why it's necessary for a sensible set > of features? I've slimmed down the patch to make it clearer what it does, having committed some refactoring. Problem: Currently when w

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Joshua D. Drake
On Fri, 2010-01-29 at 11:41 -0800, Josh Berkus wrote: > All, > > Is there a working list of HS must-fix items somewhere which people > agree on? Or are we still lacking consensus? VACUUM FULL, I believe is one. Joshua D. Drake > > --Josh Berkus > -- PostgreSQL.org Major Contributor Comman

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Josh Berkus
All, Is there a working list of HS must-fix items somewhere which people agree on? Or are we still lacking consensus? --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Joshua D. Drake
On Fri, 2010-01-29 at 18:08 +, Simon Riggs wrote: > On Fri, 2010-01-29 at 12:23 -0500, Robert Haas wrote: > > Two months on, there is > > zero sign of any activity on that front > > I'm surprised that you call 14 commits in 28 days following a publicly > available priority list: "zero sign of

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Robert Haas
On Fri, Jan 29, 2010 at 1:08 PM, Simon Riggs wrote: > On Fri, 2010-01-29 at 12:23 -0500, Robert Haas wrote: >> Two months on, there is >> zero sign of any activity on that front > > I'm surprised that you call 14 commits in 28 days following a publicly > available priority list: "zero sign of acti

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 12:23 -0500, Robert Haas wrote: > Two months on, there is > zero sign of any activity on that front I'm surprised that you call 14 commits in 28 days following a publicly available priority list: "zero sign of activity". Further discussion seems pointless. -- Simon Riggs

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Joshua D. Drake
On Fri, 2010-01-29 at 12:23 -0500, Robert Haas wrote: > Exactly. It would be nice to see 9.0 come out in 2010, and we're not > going to get there unless we start fixing the issues that are actually > release-blockers, rather than adding new features. Hot Standby was > committed with at least one

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Robert Haas
On Fri, Jan 29, 2010 at 11:32 AM, Joshua D. Drake wrote: > On Fri, 2010-01-29 at 12:56 +, Simon Riggs wrote: > >> I think we should extend the time available to make sure we have a >> sensible set of features for 9.0. The heat of this discussion tells me >> that we are going to be lacking fea

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Stefan Kaltenbrunner
Simon Riggs wrote: On Fri, 2010-01-29 at 11:20 +0100, Stefan Kaltenbrunner wrote: Simon Riggs wrote: On Fri, 2010-01-29 at 11:10 +0100, Stefan Kaltenbrunner wrote: yeah and we keep finding major bugs nearly daily Facts, please? 5 seconds of time spent on archives.postgresql.org show at leas

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 14:52 +, Greg Stark wrote: > You said "I think we should extend the time available to make sure we > have a sensible set of features for 9.0." Perhaps part of the problem > is that I couldn't understand what your patch did from the description > you posted and can't eval

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Joshua D. Drake
On Fri, 2010-01-29 at 12:56 +, Simon Riggs wrote: > I think we should extend the time available to make sure we have a > sensible set of features for 9.0. The heat of this discussion tells me > that we are going to be lacking features that are must-have to someone, > whether or not they are in

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 16:44 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > I removed code that you mentioned was > > buggy because I don't have time to fix it and it is not high enough up > > the priority list. We have discussed all of these things before yet you > > raise them again as

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Greg Stark
The fundamental disagreement here is over what qualifies as a "wishlist" item, aka a feature or added functionality. And what qualifies as a must-fix bug. Priorities are context sensitive. If this were early in the cycle then working on bigger impact features like conflict resolution code might be

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Heikki Linnakangas
Simon Riggs wrote: > I removed code that you mentioned was > buggy because I don't have time to fix it and it is not high enough up > the priority list. We have discussed all of these things before yet you > raise them again as if those things have never been said. *sigh*. Yeah, we've been through

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 11:33 +0200, Heikki Linnakangas wrote: > I even *fixed* that already, but you decided to take it out before > committing. I then added it to the list of must-fix items in the TODO > list, but you took that out too. I have no objection to doing things > in smaller steps, but t

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 07:01 -0500, Robert Haas wrote: > On Fri, Jan 29, 2010 at 5:08 AM, Simon Riggs wrote: > > On Fri, 2010-01-29 at 11:33 +0200, Heikki Linnakangas wrote: > > > >> So what was the clear result? > > > > I have spoken clearly enough. You were welcome to attend the Hot Standby > > U

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Robert Haas
On Fri, Jan 29, 2010 at 5:08 AM, Simon Riggs wrote: > On Fri, 2010-01-29 at 11:33 +0200, Heikki Linnakangas wrote: > >> So what was the clear result? > > I have spoken clearly enough. You were welcome to attend the Hot Standby > User Group. The fact that you did not expresses your own priorities >

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 11:20 +0100, Stefan Kaltenbrunner wrote: > Simon Riggs wrote: > > On Fri, 2010-01-29 at 11:10 +0100, Stefan Kaltenbrunner wrote: > > > >> yeah and we keep finding major bugs nearly daily > > > > Facts, please? > > > > 5 seconds of time spent on archives.postgresql.org show

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Stefan Kaltenbrunner
Simon Riggs wrote: On Fri, 2010-01-29 at 11:12 +0100, Stefan Kaltenbrunner wrote: There are many features we should add. I will add them in priority order until forced to stop. we are past the point of adding new features for 9.0 imho So presumably we cannot add the new feature to start hot

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Stefan Kaltenbrunner
Simon Riggs wrote: On Fri, 2010-01-29 at 11:10 +0100, Stefan Kaltenbrunner wrote: yeah and we keep finding major bugs nearly daily Facts, please? 5 seconds of time spent on archives.postgresql.org show at least the following SR/HS related bugs in the last 7 days or so: http://archives.p

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 11:12 +0100, Stefan Kaltenbrunner wrote: > > > > There are many features we should add. I will add them in priority order > > until forced to stop. > > we are past the point of adding new features for 9.0 imho So presumably we cannot add the new feature to start hot standb

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Magnus Hagander
2010/1/29 Stefan Kaltenbrunner : > Simon Riggs wrote: >> >> On Fri, 2010-01-29 at 11:33 +0200, Heikki Linnakangas wrote: >> >>> So what was the clear result? >> >> I have spoken clearly enough. You were welcome to attend the Hot Standby >> User Group. The fact that you did not expresses your own pr

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 11:10 +0100, Stefan Kaltenbrunner wrote: > yeah and we keep finding major bugs nearly daily Facts, please? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://ww

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Stefan Kaltenbrunner
Simon Riggs wrote: On Fri, 2010-01-29 at 11:33 +0200, Heikki Linnakangas wrote: So what was the clear result? I have spoken clearly enough. You were welcome to attend the Hot Standby User Group. The fact that you did not expresses your own priorities quite well, ISTM. Your protestations to kn

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Stefan Kaltenbrunner
Heikki Linnakangas wrote: Simon Riggs wrote: Conflict resolution improvements are important to include in this release, as discussed many times. Proposal given here http://archives.postgresql.org/pgsql-hackers/2009-12/msg01175.php presents a viable design to improve this. Following patch is a c

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 11:33 +0200, Heikki Linnakangas wrote: > So what was the clear result? I have spoken clearly enough. You were welcome to attend the Hot Standby User Group. The fact that you did not expresses your own priorities quite well, ISTM. Your protestations to know more about the wis

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Heikki Linnakangas
Simon Riggs wrote: > On Fri, 2010-01-29 at 08:26 +0200, Heikki Linnakangas wrote: >> Simon Riggs wrote: >>> Conflict resolution improvements are important to include in this >>> release, as discussed many times. Proposal given here >>> http://archives.postgresql.org/pgsql-hackers/2009-12/msg01175.p

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 09:20 +0100, Guillaume Smet wrote: > On Fri, Jan 29, 2010 at 9:03 AM, Simon Riggs wrote: > > That was not the feedback I have received. Nobody has commented on that > > to me, though many have commented on the need for the current patch. As > > mentioned, I went to the troubl

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Guillaume Smet
On Fri, Jan 29, 2010 at 9:03 AM, Simon Riggs wrote: > That was not the feedback I have received. Nobody has commented on that > to me, though many have commented on the need for the current patch. As > mentioned, I went to the trouble of running a meeting to gain additional > feedback and the resu

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 08:26 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > Conflict resolution improvements are important to include in this > > release, as discussed many times. Proposal given here > > http://archives.postgresql.org/pgsql-hackers/2009-12/msg01175.php > > presents a viab

Re: [HACKERS] Hot Standby: Relation-specific deferred conflict resolution

2010-01-28 Thread Heikki Linnakangas
Simon Riggs wrote: > Conflict resolution improvements are important to include in this > release, as discussed many times. Proposal given here > http://archives.postgresql.org/pgsql-hackers/2009-12/msg01175.php > presents a viable design to improve this. > > Following patch is a complete working i

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Heikki Linnakangas
Tom Lane wrote: > Speaking of which, just where is the defense that makes sure that > walsender and walreceiver are compatible? We should be checking not > only version, but all of the configuration variables that are embedded > in pg_control. That happens at startup when pg_control is read, befo

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Tom Lane
Heikki Linnakangas writes: > Tom Lane wrote: >> Whether or not anyone bothers with the timestamp message, I think adding >> a message type header is a Must Fix item. A protocol with no provision >> for extension is certainly going to bite us in the rear before long. > Agreed a message type heade

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Heikki Linnakangas
Tom Lane wrote: > Heikki Linnakangas writes: >> Simon Riggs wrote: >>> Do we need a new record type for that, is there a handy record type to >>> bounce from? > >> After starting streaming, slices of WAL are sent as CopyData messages. >> The CopyData payload begins with an XLogRecPtr, followed by

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Tom Lane
Heikki Linnakangas writes: > Simon Riggs wrote: >> Do we need a new record type for that, is there a handy record type to >> bounce from? > After starting streaming, slices of WAL are sent as CopyData messages. > The CopyData payload begins with an XLogRecPtr, followed by the WAL > data. That pay

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Heikki Linnakangas
Simon Riggs wrote: > What were the blockers that prevented sync rep from being included? I > must have missed the discussion on that part. For one, figuring out how to send back the notifications about WAL applied in standby, and all the IPC required for that. Streaming replication is a complex e

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Heikki Linnakangas
Fujii Masao wrote: > On Mon, Jan 18, 2010 at 6:35 PM, Fujii Masao wrote: >> On Mon, Jan 18, 2010 at 5:45 PM, Simon Riggs wrote: >>> How accurate is this now? With regard to remaining items of work. >>> http://wiki.postgresql.org/wiki/Streaming_Replication >> Not accurate. I'll correct that and pr

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Simon Riggs
On Mon, 2010-01-18 at 20:18 +0900, Fujii Masao wrote: > On Mon, Jan 18, 2010 at 6:35 PM, Fujii Masao wrote: > > On Mon, Jan 18, 2010 at 5:45 PM, Simon Riggs wrote: > >> How accurate is this now? With regard to remaining items of work. > >> http://wiki.postgresql.org/wiki/Streaming_Replication > >

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Fujii Masao
On Mon, Jan 18, 2010 at 6:35 PM, Fujii Masao wrote: > On Mon, Jan 18, 2010 at 5:45 PM, Simon Riggs wrote: >> How accurate is this now? With regard to remaining items of work. >> http://wiki.postgresql.org/wiki/Streaming_Replication > > Not accurate. I'll correct that and provide the link from > "

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Fujii Masao
On Mon, Jan 18, 2010 at 5:45 PM, Simon Riggs wrote: > How accurate is this now? With regard to remaining items of work. > http://wiki.postgresql.org/wiki/Streaming_Replication Not accurate. I'll correct that and provide the link from "v8.5 Open Items page" to that. Regards, -- Fujii Masao NIPP

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Simon Riggs
On Mon, 2010-01-18 at 08:28 +0200, Heikki Linnakangas wrote: > > Do we need a new record type for that, is there a handy record type to > > bounce from? > > After starting streaming, slices of WAL are sent as CopyData messages. > The CopyData payload begins with an XLogRecPtr, followed by the WAL

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-18 Thread Simon Riggs
On Mon, 2010-01-18 at 08:28 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Sun, 2010-01-17 at 22:57 +0200, Heikki Linnakangas wrote: > >>> It would be good if there was a keepalive WAL record with a > >> timestamp on it generated every N seconds while in streaming mode. > >> > >> Yeah

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-17 Thread Heikki Linnakangas
Simon Riggs wrote: > On Sun, 2010-01-17 at 22:57 +0200, Heikki Linnakangas wrote: >>> It would be good if there was a keepalive WAL record with a >> timestamp on it generated every N seconds while in streaming mode. >> >> Yeah, that would help. In streaming replication we could also send >> such ti

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-17 Thread Simon Riggs
On Sun, 2010-01-17 at 22:57 +0200, Heikki Linnakangas wrote: > > > It would be good if there was a keepalive WAL record with a > timestamp on it generated every N seconds while in streaming mode. > > Yeah, that would help. In streaming replication we could also send > such timestamp as a separate

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-17 Thread Heikki Linnakangas
Simon Riggs wrote: > On Fri, 2010-01-15 at 20:50 +0200, Heikki Linnakangas wrote: > >>> So in either case, when we are waiting for new input we reset the timer >>> as soon as new WAL is received. The resolution/accuracy of standby_delay >>> will be no more than the time taken to replay a single fi

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-16 Thread Joshua D. Drake
On Sat, 2010-01-16 at 18:20 +, Simon Riggs wrote: > On Sat, 2010-01-16 at 11:37 -0500, Tom Lane wrote: > > Simon Riggs writes: > > > I'm wondering if it wouldn't just be easier to put in a plugin for > > > recovery conflict handling, so the user can decide what to do > > > themselves. That see

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-16 Thread Simon Riggs
On Sat, 2010-01-16 at 11:37 -0500, Tom Lane wrote: > Simon Riggs writes: > > I'm wondering if it wouldn't just be easier to put in a plugin for > > recovery conflict handling, so the user can decide what to do > > themselves. That seems like a better plan than chewing through these > > issues now.

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-16 Thread Simon Riggs
On Fri, 2010-01-15 at 20:50 +0200, Heikki Linnakangas wrote: > That would change the meaning of max_standby_delay. Currently, it's the > delay between *generating* and applying a WAL record, your proposal > would change it to mean delay between receiving and applying it. That > seems a lot less us

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-16 Thread Tom Lane
Simon Riggs writes: > I'm wondering if it wouldn't just be easier to put in a plugin for > recovery conflict handling, so the user can decide what to do > themselves. That seems like a better plan than chewing through these > issues now. Making it a plugin doesn't solve anything. This is not th

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-16 Thread Simon Riggs
On Sat, 2010-01-16 at 14:08 +0100, Dimitri Fontaine wrote: > Simon Riggs writes: > > On Fri, 2010-01-15 at 20:50 +0200, Heikki Linnakangas wrote: > > Yes, it does. And I know you're thinking along those lines because we > > are concurrently discussing how to handle re-connection after updates. >

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-16 Thread Dimitri Fontaine
Simon Riggs writes: > On Fri, 2010-01-15 at 20:50 +0200, Heikki Linnakangas wrote: > Yes, it does. And I know you're thinking along those lines because we > are concurrently discussing how to handle re-connection after updates. With my State Machine proposal, we could only apply max_standby_delay

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-15 Thread Simon Riggs
On Fri, 2010-01-15 at 20:50 +0200, Heikki Linnakangas wrote: > > So in either case, when we are waiting for new input we reset the timer > > as soon as new WAL is received. The resolution/accuracy of standby_delay > > will be no more than the time taken to replay a single file. This > > shouldn't

Re: [HACKERS] Hot Standby and handling max_standby_delay

2010-01-15 Thread Heikki Linnakangas
Simon Riggs wrote: > We need to calculate a more accurate time since WAL arrived to make > max_standby_delay sensible in all cases. Difficult to know exactly when > to record new timestamps for received WAL. So, proposal is... > > if (Base time is earlier than WAL record time) > standby_dela

Re: [HACKERS] Hot Standby and query cancel

2010-01-14 Thread Andres Freund
On Thursday 14 January 2010 13:21:07 Simon Riggs wrote: > On Wed, 2010-01-13 at 19:23 +, Simon Riggs wrote: > > On Wed, 2010-01-13 at 19:58 +0100, Andres Freund wrote: > > > > I am still testing patch, so should be confident to commit tomorrow > > > > barring issues. > > > > > > I have only lo

Re: [HACKERS] Hot Standby and query cancel

2010-01-14 Thread Simon Riggs
On Wed, 2010-01-13 at 19:23 +, Simon Riggs wrote: > On Wed, 2010-01-13 at 19:58 +0100, Andres Freund wrote: > > > > I am still testing patch, so should be confident to commit tomorrow > > > barring issues. > > I have only looked at briefly because right now I dont have the time (going > > to

Re: [HACKERS] Hot Standby and query cancel

2010-01-13 Thread Simon Riggs
On Wed, 2010-01-13 at 19:58 +0100, Andres Freund wrote: > > I am still testing patch, so should be confident to commit tomorrow > > barring issues. > I have only looked at briefly because right now I dont have the time (going > to > eat at a friends place...) but I think I spotted an issue: > Th

Re: [HACKERS] Hot Standby and query cancel

2010-01-13 Thread Andres Freund
Hi Simon, On Wednesday 13 January 2010 19:24:22 Simon Riggs wrote: > We've been chewing around query cancel on Hot Standby and I think things > have got fairly confusing, hence a new thread. Good idea. > I enclose a patch that includes all the things that we all agree on so > far, in my understan

Re: [HACKERS] Hot standby documentation

2010-01-13 Thread Simon Riggs
On Thu, 2010-01-07 at 18:34 +0900, Fujii Masao wrote: > On Thu, Jan 7, 2010 at 6:09 PM, Joshua Tolley wrote: > > Having concluded I really need to start playing with hot standby, I started > > looking for documentation on the subject. I found what I was looking for; I > > also found this page[1],

Re: [HACKERS] Hot standby documentation

2010-01-07 Thread Fujii Masao
On Thu, Jan 7, 2010 at 6:09 PM, Joshua Tolley wrote: > Having concluded I really need to start playing with hot standby, I started > looking for documentation on the subject. I found what I was looking for; I > also found this page[1], which, it seems, ought to mention hot standby. > Comments? > >

Re: [HACKERS] Hot Standby and cancelling idle queries

2009-12-26 Thread Andres Freund
On Wednesday 25 November 2009 17:25:43 Tom Lane wrote: > Simon Riggs writes: > > An idle-in-transaction transaction can also hold a temporary file. Think > > of an open cursor, for example. Therefore, remove the distinction > > between CONFLICT_MODE_ERROR and CONFLICT_MODE_ERROR_IF_NOT_IDLE, > > i

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-17 Thread Simon Riggs
On Thu, 2009-12-17 at 13:38 +, Simon Riggs wrote: > > I see now that in the presence of prepared transactions, we would fail > > to clean up failed transations with XID > the oldest prepared > > transaction > > Good! I just spotted that also, just prior to posting my fix, so > rewriting it ag

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-17 Thread Simon Riggs
On Thu, 2009-12-17 at 15:18 +0200, Heikki Linnakangas wrote: > That removed piece of code was executed in the standby whenever we saw a > shutdown checkpoint. It calls ProcArrayApplyRecoveryInfo(), which calls > ExpireOldKnownAssignedTransactionIds() and StandbyReleaseOldLocks() to > clean up know

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-17 Thread Heikki Linnakangas
Simon Riggs wrote: > On Thu, 2009-12-17 at 13:24 +0200, Heikki Linnakangas wrote: > >> Hmm, looking at the code, I think Simon threw that baby with the >> bathwater when he removed support for starting standby from a shutdown >> checkpoint. > > Hmm, I think that code was just for starting points

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-17 Thread Simon Riggs
On Thu, 2009-12-17 at 13:24 +0200, Heikki Linnakangas wrote: > Hmm, looking at the code, I think Simon threw that baby with the > bathwater when he removed support for starting standby from a shutdown > checkpoint. Hmm, I think that code was just for starting points only. It would not have been e

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-17 Thread Simon Riggs
On Thu, 2009-12-17 at 19:55 +0900, Hiroyuki Yamada wrote: > Well, I've read some more and have a question. > > The implementation assumes that transactions write COMMIT/ABORT WAL at the end > of them, while it does not seem to write ABORT WAL in immediate shutdown. So, > > 1. acquire ACCESS EXCL

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-17 Thread Heikki Linnakangas
Hiroyuki Yamada wrote: > The implementation assumes that transactions write COMMIT/ABORT WAL at the end > of them, while it does not seem to write ABORT WAL in immediate shutdown. So, > > 1. acquire ACCESS EXCLUSIVE lock in table A in xact 1 > 2. execute immediate shutdown of the active node > 3.

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-17 Thread Hiroyuki Yamada
>On Wed, 2009-12-16 at 19:35 +0900, Hiroyuki Yamada wrote: > >> * There is a window beween gathering lock information in >> GetRunningTransactionLocks() >>and writing WAL in LogAccessExclusiveLocks(). >> * In current lock redo algorithm, locks are released when the transaction >> holding t

Re: [HACKERS] Hot Standby, release candidate?

2009-12-17 Thread Simon Riggs
On Thu, 2009-12-17 at 12:01 +0200, Peter Eisentraut wrote: > On sön, 2009-12-13 at 19:20 +, Simon Riggs wrote: > > Barring resolving a few points and subject to even more testing, this > > is the version I expect to commit to CVS on Wednesday. > > So it's Thursday now. Please keep us updated

Re: [HACKERS] Hot Standby, release candidate?

2009-12-17 Thread Peter Eisentraut
On sön, 2009-12-13 at 19:20 +, Simon Riggs wrote: > Barring resolving a few points and subject to even more testing, this > is the version I expect to commit to CVS on Wednesday. So it's Thursday now. Please keep us updated on the schedule, as we need to decide when to wrap alpha3 and whether

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-16 Thread Simon Riggs
On Wed, 2009-12-16 at 19:35 +0900, Hiroyuki Yamada wrote: > * There is a window beween gathering lock information in > GetRunningTransactionLocks() >and writing WAL in LogAccessExclusiveLocks(). > * In current lock redo algorithm, locks are released when the transaction > holding the lock

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-16 Thread Simon Riggs
On Wed, 2009-12-16 at 19:35 +0900, Hiroyuki Yamada wrote: > Sorry for annoying you. Not at all! Good to get fresh eyes on this. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-16 Thread Heikki Linnakangas
Hiroyuki Yamada wrote: > By the way, reading LogStandbySnapshot() and GetRunningTransactionLocks() > raised following questions. > > * There is a window beween gathering lock information in > GetRunningTransactionLocks() >and writing WAL in LogAccessExclusiveLocks(). > * In current lock red

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-16 Thread Hiroyuki Yamada
>On Wed, 2009-12-16 at 18:08 +0900, Hiroyuki Yamada wrote: >> >That fixes or explains all known issues, from me. Are there any other >> >things you know about that I haven't responded to? Do you think we have >> >addressed every issue, except deferred items? >> > >> >I will be looking to commit to

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-16 Thread Simon Riggs
On Wed, 2009-12-16 at 18:08 +0900, Hiroyuki Yamada wrote: > >That fixes or explains all known issues, from me. Are there any other > >things you know about that I haven't responded to? Do you think we have > >addressed every issue, except deferred items? > > > >I will be looking to commit to CVS la

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-16 Thread Hiroyuki Yamada
>That fixes or explains all known issues, from me. Are there any other >things you know about that I haven't responded to? Do you think we have >addressed every issue, except deferred items? > >I will be looking to commit to CVS later today; waiting on any >objections. > Is following problem repo

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-16 Thread Simon Riggs
On Wed, 2009-12-16 at 09:10 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > Investigating how that could come about, it looks like there is some > > fairly strange stuff going on here. StandbyRecoverPreparedTransactions() > > is never called at all. > > I told you so: > http://archives.p

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-15 Thread Heikki Linnakangas
Simon Riggs wrote: > Investigating how that could come about, it looks like there is some > fairly strange stuff going on here. StandbyRecoverPreparedTransactions() > is never called at all. I told you so: http://archives.postgresql.org/message-id/4b260b5e.3010...@enterprisedb.com StandbyRecoverP

Re: [HACKERS] Hot Standby and prepared transactions

2009-12-15 Thread Simon Riggs
On Tue, 2009-12-15 at 18:49 +, Simon Riggs wrote: > This looks like the code is setting the parent to be zero. [Sorry, that comment is completely off, misread the line number.] The assertion is failing because the parent entry for that subxid had already been set. Investigating how that coul

Re: [HACKERS] Hot Standby, release candidate?

2009-12-14 Thread Greg Smith
Simon Riggs wrote: On Mon, 2009-12-14 at 11:44 +0200, Peter Eisentraut wrote: On mån, 2009-12-14 at 08:54 +, Simon Riggs wrote: Wednesday because that seemed a good delay to allow review. Josh and I had discussed the value of getting patch into Alpha3, so that was my wish and aim.

<    1   2   3   4   5   6   7   8   9   10   >