Re: [HACKERS] sync rep design architecture (was "disposition of remaining patches")

2011-02-26 Thread Daniel Farina
On Fri, Feb 25, 2011 at 8:40 AM, Greg Smith wrote: > I didn't get the Streaming Rep + Hot Standby features I wanted in 9.0 either. >  But committing what was reasonable to include in that version let me march > forward with very useful new code, doing another year of development on my > own pro

Re: [HACKERS] sync rep design architecture (was "disposition of remaining patches")

2011-02-25 Thread Greg Smith
Daniel Farina wrote: Server A syncreps to Server B Now I want to provision server A-prime, which will eventually take the place of A. Server A syncreps to Server B Server A syncreps to Server A-prime Right now, as it stands, the syncrep patch will be happy as soon as the data has been fsynced

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Greg Smith
Simon Riggs wrote: Based upon that series of conversations, I've reworked the design so that there is (currently) only a single standby offering sync rep at any one time. Other standbys can request to be sync standbys but they only become the sync standby if the first one fails. Which was simple

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Josh Berkus
> I was just thinking that you could prepend a reject line at the right > place in the file. Hmmm, that's worth thinking about. How do others feel about this? -- -- Josh Berkus PostgreSQL Experts Inc.

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Dimitri Fontaine
Josh Berkus writes: >> What about the HBA here? > > Hmmm. That's tempting; an "synchronous" HBA instead of a GUC? But that > doesn't solve the problem of "standby #6 is failing, I want to kick it > off synch rep". > > I'd be opposed to having a GUC *and* an HBA. making DBAs set things > indepen

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Josh Berkus
> What about the HBA here? Hmmm. That's tempting; an "synchronous" HBA instead of a GUC? But that doesn't solve the problem of "standby #6 is failing, I want to kick it off synch rep". I'd be opposed to having a GUC *and* an HBA. making DBAs set things independantly in two places just frustra

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Dimitri Fontaine
Josh Berkus writes: > So, again, I don't agree that a separate synchrep permission is useful, > or warranted. +1 > However, your arguments *do* make me backpedal on the issue of having a > list of synch rep roles on the master. I can easily imagine a DBA > needing to rapidly disable synch rep i

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Robert Haas
On Tue, Jan 4, 2011 at 2:50 PM, Stefan Kaltenbrunner wrote: > On 01/04/2011 07:51 PM, Simon Riggs wrote: >> >> On Tue, 2011-01-04 at 10:28 -0800, Josh Berkus wrote: >> >>> The relevant question is: which configuration(s) can we have ready for >>> the next CommitFest and alpha release? >> >> Based

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Stefan Kaltenbrunner
On 01/04/2011 07:51 PM, Simon Riggs wrote: On Tue, 2011-01-04 at 10:28 -0800, Josh Berkus wrote: The relevant question is: which configuration(s) can we have ready for the next CommitFest and alpha release? Based upon that series of conversations, I've reworked the design so that there is (cu

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Joshua D. Drake
> I'm not feeling well now, so I'm going to go to bed, not just to avoid > snapping at people. Even given that short interlude, I see no problem > about delivery. Cool! Thanks Simon. Feel better. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Robert Haas
On Sun, Jan 2, 2011 at 4:19 PM, Simon Riggs wrote: > On Sun, 2011-01-02 at 18:54 +0200, Heikki Linnakangas wrote: > >> I believe we all agree that there's different use cases that require >> different setups. Both "first-past-the-post" and "wait-for-all-to-ack" >> have their uses. > > Robert's ana

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Simon Riggs
On Tue, 2011-01-04 at 10:28 -0800, Josh Berkus wrote: > The relevant question is: which configuration(s) can we have ready for > the next CommitFest and alpha release? Based upon that series of conversations, I've reworked the design so that there is (currently) only a single standby offering syn

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Josh Berkus
On 1/2/11 12:35 AM, Heikki Linnakangas wrote: > Very likely. A synchronous standby can bring the master to a halt, while > an asynchronous one is rather harmless. If I were a DBA, and the data > wasn't very sensitive, I would liberally hand out async privileges to my > colleagues to set up reportin

Re: [HACKERS] Sync Rep Design

2011-01-04 Thread Josh Berkus
All, This is a pointless argument. Eventually, we will be implementing all possible sync rep configurations, because different users *need* different configurations. Some users care more about durability, some more about availability, and some more about response time. And you can't have all th

Re: [HACKERS] Sync Rep Design

2011-01-03 Thread Simon Riggs
On Sun, 2011-01-02 at 20:53 +, Simon Riggs wrote: > On Sun, 2011-01-02 at 12:13 -0800, MARK CALLAGHAN wrote: > > On Thu, Dec 30, 2010 at 9:02 PM, Robert Haas wrote: > > > > > > > > > I see now that you've tried to design this feature in a way that is > > > similar to MySQL's offering, which d

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Hannu Krosing
On 2.1.2011 5:36, Robert Haas wrote: On Sat, Jan 1, 2011 at 6:54 AM, Simon Riggs wrote: Yes, working out the math is a good idea. Things are much clearer if we do that. Let's assume we have 98% availability on any single server. 1. Having one primary and 2 standbys, either of which can acknow

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Simon Riggs
On Sun, 2011-01-02 at 18:54 +0200, Heikki Linnakangas wrote: > I believe we all agree that there's different use cases that require > different setups. Both "first-past-the-post" and "wait-for-all-to-ack" > have their uses. Robert's analysis is that "first-past-the-post" doesn't actually impro

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Simon Riggs
On Sun, 2011-01-02 at 12:13 -0800, MARK CALLAGHAN wrote: > On Thu, Dec 30, 2010 at 9:02 PM, Robert Haas wrote: > > > > > > I see now that you've tried to design this feature in a way that is > > similar to MySQL's offering, which does have some value. But it > > appears to me that the documentat

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Jeff Janes
On Sat, Jan 1, 2011 at 8:35 AM, Simon Riggs wrote: > On Sat, 2011-01-01 at 05:13 -0800, Jeff Janes wrote: >> On 12/31/10, Simon Riggs wrote: > >> > 2. "sync" does not guarantee that the updates to the standbys are in any >> > way coordinated. You can run a query on one standby and get one answer

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread MARK CALLAGHAN
On Thu, Dec 30, 2010 at 9:02 PM, Robert Haas wrote: > > > I see now that you've tried to design this feature in a way that is > similar to MySQL's offering, which does have some value.  But it > appears to me that the documentation you've written here is > substantially similar to the MySQL 5.5 r

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Simon Riggs
On Sun, 2011-01-02 at 11:11 -0600, Kevin Grittner wrote: > Simon Riggs wrote: > > > Do you agree that requiring response from 2 sync standbys, or > > locking up, gives us 94% server availability, but 99.9992% data > > durability? > > I'm not sure how to answer that. The calculations so far ha

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Kevin Grittner
Simon Riggs wrote: > Do you agree that requiring response from 2 sync standbys, or > locking up, gives us 94% server availability, but 99.9992% data > durability? I'm not sure how to answer that. The calculations so far have been based around up-time and the probabilities that you have a mach

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Heikki Linnakangas
On 02.01.2011 15:41, Simon Riggs wrote: On Sat, 2011-01-01 at 23:36 -0500, Robert Haas wrote: On Sat, Jan 1, 2011 at 6:54 AM, Simon Riggs wrote: Yes, working out the math is a good idea. Things are much clearer if we do that. Let's assume we have 98% availability on any single server. 1. Hav

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Simon Riggs
On Sun, 2011-01-02 at 08:08 -0600, Kevin Grittner wrote: > I think you're talking about different metrics, and you're both > right. With two servers configured in sync rep your chance of having > an available (running) server is 99.9992%. The chance that you know > that you have one that is tota

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Kevin Grittner
Simon Riggs wrote: > On Sat, 2011-01-01 at 23:36 -0500, Robert Haas wrote: >> On Sat, Jan 1, 2011 at 6:54 AM, Simon Riggs wrote: >>> Yes, working out the math is a good idea. Things are much clearer >>> if we do that. >>> >>> Let's assume we have 98% availability on any single server. >>> >>> 1.

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Simon Riggs
On Sat, 2011-01-01 at 23:36 -0500, Robert Haas wrote: > On Sat, Jan 1, 2011 at 6:54 AM, Simon Riggs wrote: > > Yes, working out the math is a good idea. Things are much clearer if we > > do that. > > > > Let's assume we have 98% availability on any single server. > > > > 1. Having one primary and

Base Backup Streaming (was: [HACKERS] Sync Rep Design)

2011-01-02 Thread Dimitri Fontaine
Heikki Linnakangas writes: > BTW, there's a bunch of replication related stuff that we should work to > close, that are IMHO more important than synchronous replication. Like > making the standby follow timeline changes, to make failovers smoother, and > the facility to stream a base-backup over t

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Hannu Krosing
On 2.1.2011 5:36, Robert Haas wrote: On Sat, Jan 1, 2011 at 6:54 AM, Simon Riggs wrote: Yes, working out the math is a good idea. Things are much clearer if we do that. Let's assume we have 98% availability on any single server. 1. Having one primary and 2 standbys, either of which can acknow

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Simon Riggs
On Sun, 2011-01-02 at 10:35 +0200, Heikki Linnakangas wrote: > > > Frankly, if Simon hadn't already submitted code, I'd be pushing for > > single-standby-only for 9.1, instead of "any one". > > Yes, we are awfully late, but let's not panic. Yes, we're about a year late. Getting a simple feature

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Simon Riggs
On Sat, 2011-01-01 at 22:11 -0500, Aidan Van Dyk wrote: > On Sat, Jan 1, 2011 at 6:08 PM, Simon Riggs wrote: > > On Sat, 2011-01-01 at 14:40 -0800, Josh Berkus wrote: > > > >> Standby in general deals with the A,D,R triangle (Availability, > >> Durability, Response time). "Any one" configuration

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Simon Riggs
On Sun, 2011-01-02 at 10:35 +0200, Heikki Linnakangas wrote: > BTW, there's a bunch of replication related stuff that we should work > to close, that are IMHO more important than synchronous replication. > Like making the standby follow timeline changes, to make failovers > smoother, and the facil

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Stefan Kaltenbrunner
On 01/02/2011 09:35 AM, Heikki Linnakangas wrote: On 02.01.2011 00:40, Josh Berkus wrote: On 1/1/11 5:59 AM, Stefan Kaltenbrunner wrote: well you keep saying that but to be honest I cannot really even see a usecase for me - what is "only a random one of a set of servers is sync at any time and

Re: [HACKERS] Sync Rep Design

2011-01-02 Thread Heikki Linnakangas
On 02.01.2011 00:40, Josh Berkus wrote: On 1/1/11 5:59 AM, Stefan Kaltenbrunner wrote: well you keep saying that but to be honest I cannot really even see a usecase for me - what is "only a random one of a set of servers is sync at any time and I don't really know which one". My usecases would a

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Robert Haas
On Sat, Jan 1, 2011 at 6:54 AM, Simon Riggs wrote: > Yes, working out the math is a good idea. Things are much clearer if we > do that. > > Let's assume we have 98% availability on any single server. > > 1. Having one primary and 2 standbys, either of which can acknowledge, > and we never lock up

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Aidan Van Dyk
On Sat, Jan 1, 2011 at 6:08 PM, Simon Riggs wrote: > On Sat, 2011-01-01 at 14:40 -0800, Josh Berkus wrote: > >> Standby in general deals with the A,D,R triangle (Availability, >> Durability, Response time).  "Any one" configuration is the A,R >> configuration, and the only reason to go out with it

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Simon Riggs
On Sat, 2011-01-01 at 14:40 -0800, Josh Berkus wrote: > Standby in general deals with the A,D,R triangle (Availability, > Durability, Response time). "Any one" configuration is the A,R > configuration, and the only reason to go out with it for 9.1 is > because it's simpler to implement than the D

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Simon Riggs
On Sat, 2011-01-01 at 21:41 +0200, Heikki Linnakangas wrote: > On 31.12.2010 23:18, Hannu Krosing wrote: > > On 31.12.2010 13:40, Heikki Linnakangas wrote: > >> That thread makes no mention of how to specify which standbys are > >> synchronous and which are not. > > The simplest way would be to hav

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Josh Berkus
On 1/1/11 5:59 AM, Stefan Kaltenbrunner wrote: > well you keep saying that but to be honest I cannot really even see a > usecase for me - what is "only a random one of a set of servers is sync > at any time and I don't really know which one". > My usecases would al involved 2 sync standbys and 1 or

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Heikki Linnakangas
On 31.12.2010 23:18, Hannu Krosing wrote: On 31.12.2010 13:40, Heikki Linnakangas wrote: That thread makes no mention of how to specify which standbys are synchronous and which are not. The simplest way would be to have separate database users for sync and async standbys ? That would allow any

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Heikki Linnakangas
On 01.01.2011 19:03, Simon Riggs wrote: On Sat, 2011-01-01 at 17:37 +0100, Stefan Kaltenbrunner wrote: On 01/01/2011 05:28 PM, Dimitri Fontaine wrote: Stefan Kaltenbrunner writes: well you keep saying that but to be honest I cannot really even see a usecase for me - what is "only a random on

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Simon Riggs
On Sat, 2011-01-01 at 18:49 +0100, Stefan Kaltenbrunner wrote: > hmm maybe my "surviving" standbys(the case I'm wondering about is > whole > datacenter failures which might take out more than just the master) > was > not clear - consider three boxes, one master and two standby and > semisync rep

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Stefan Kaltenbrunner
On 01/01/2011 06:29 PM, Simon Riggs wrote: On Sat, 2011-01-01 at 18:13 +0100, Stefan Kaltenbrunner wrote: On 01/01/2011 05:55 PM, Simon Riggs wrote: It appears to me there has been substantial confusion over alternatives, because of a misunderstanding about how synchronisation works. Requiring

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Simon Riggs
On Sat, 2011-01-01 at 17:28 +0100, Dimitri Fontaine wrote: > something visible through a system view for > users? This as been asked for before and I was thinking there was a > consensus on this. Yes, it will be there. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Dev

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Simon Riggs
On Sat, 2011-01-01 at 18:13 +0100, Stefan Kaltenbrunner wrote: > On 01/01/2011 05:55 PM, Simon Riggs wrote: > > > > It appears to me there has been substantial confusion over alternatives, > > because of a misunderstanding about how synchronisation works. Requiring > > confirmation that standbys ar

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Stefan Kaltenbrunner
On 01/01/2011 05:55 PM, Simon Riggs wrote: On Sat, 2011-01-01 at 16:12 +0100, Stefan Kaltenbrunner wrote: I still would like to get a statement on why simon thinks that the design heikki and others have proposed I've explained in huge detail why I think what I think, nor avoided any technical

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Simon Riggs
On Sat, 2011-01-01 at 17:37 +0100, Stefan Kaltenbrunner wrote: > On 01/01/2011 05:28 PM, Dimitri Fontaine wrote: > > Stefan Kaltenbrunner writes: > >> well you keep saying that but to be honest I cannot really even see a > >> usecase for me - what is "only a random one of a set of servers is sync

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Simon Riggs
On Sat, 2011-01-01 at 16:12 +0100, Stefan Kaltenbrunner wrote: > I still would like to get a statement on why simon thinks that > the design heikki and others have proposed I've explained in huge detail why I think what I think, nor avoided any technical issue. It appears to me there has been

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Stefan Kaltenbrunner
On 01/01/2011 05:28 PM, Dimitri Fontaine wrote: Stefan Kaltenbrunner writes: well you keep saying that but to be honest I cannot really even see a usecase for me - what is "only a random one of a set of servers is sync at any time and I don't really know which one". It looks easy enough to ge

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Simon Riggs
On Sat, 2011-01-01 at 05:13 -0800, Jeff Janes wrote: > On 12/31/10, Simon Riggs wrote: > > On Fri, 2010-12-31 at 09:27 +0100, Stefan Kaltenbrunner wrote: > > > >> Maybe it has been discussed but I still don't see way it makes any > >> sense. If I declare a standby a sync standby I better want it s

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Dimitri Fontaine
Stefan Kaltenbrunner writes: > well you keep saying that but to be honest I cannot really even see a > usecase for me - what is "only a random one of a set of servers is sync at > any time and I don't really know which one". It looks easy enough to get to know which one it is. Surely the primary

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Stefan Kaltenbrunner
On 01/01/2011 03:15 PM, Robert Haas wrote: On Sat, Jan 1, 2011 at 9:03 AM, Stefan Kaltenbrunner wrote: that is exactly my point - if have no guarantee that your SYNC standby is actually sync there is no use for it being used in business cases that require sync replication. If we cannot support

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Robert Haas
On Sat, Jan 1, 2011 at 9:03 AM, Stefan Kaltenbrunner wrote: > that is exactly my point - if have no guarantee that your SYNC standby is > actually sync there is no use for it being used in business cases that > require sync replication. > If we cannot support that usecase I would either like to se

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Stefan Kaltenbrunner
On 01/01/2011 02:13 PM, Jeff Janes wrote: On 12/31/10, Simon Riggs wrote: On Fri, 2010-12-31 at 09:27 +0100, Stefan Kaltenbrunner wrote: Maybe it has been discussed but I still don't see way it makes any sense. If I declare a standby a sync standby I better want it sync - not "maybe sync". co

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Stefan Kaltenbrunner
On 12/31/2010 08:15 PM, Simon Riggs wrote: On Fri, 2010-12-31 at 14:40 +0200, Heikki Linnakangas wrote: On 31.12.2010 13:48, Simon Riggs wrote: I see significant real-world issues with configuring replication using multiple named servers, as described in the link above: All of these points o

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Jeff Janes
On 12/31/10, Simon Riggs wrote: > On Fri, 2010-12-31 at 09:27 +0100, Stefan Kaltenbrunner wrote: > >> Maybe it has been discussed but I still don't see way it makes any >> sense. If I declare a standby a sync standby I better want it sync - not >> "maybe sync". consider the case of a 1 master and

Re: [HACKERS] Sync Rep Design

2011-01-01 Thread Simon Riggs
On Fri, 2010-12-31 at 22:18 +0100, Hannu Krosing wrote: > On 31.12.2010 13:40, Heikki Linnakangas wrote: > > > > Sounds good. > > > > I still don't like the synchronous_standbys='' and > > synchronous_replication=on combination, though. IMHO that still > > amounts to letting the standby control t

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Josh Berkus
On 12/31/10 4:40 AM, Robert Haas wrote: > Someone may have proposed this before, but one way of getting standby > naming "for free" would be to make the standby names the same as the > roles used to log in, rather than adding a separate parameter. We > could just recommend to people that they use

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Hannu Krosing
On 31.12.2010 13:40, Heikki Linnakangas wrote: Sounds good. I still don't like the synchronous_standbys='' and synchronous_replication=on combination, though. IMHO that still amounts to letting the standby control the behavior on master, and it makes it impossible to temporarily add an async

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Simon Riggs
On Fri, 2010-12-31 at 14:40 +0200, Heikki Linnakangas wrote: > On 31.12.2010 13:48, Simon Riggs wrote: > > > > I see significant real-world issues with configuring replication using > > multiple named servers, as described in the link above: > > All of these points only apply to specifying *multip

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Simon Riggs
On Fri, 2010-12-31 at 07:33 -0500, Aidan Van Dyk wrote: > On Fri, Dec 31, 2010 at 5:26 AM, Simon Riggs wrote: > > > Your picture above is a common misconception. I will add something to > > the docs to explain this. > > > 2. "sync" does not guarantee that the updates to the standbys are in any >

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Robert Haas
On Fri, Dec 31, 2010 at 8:48 AM, Stefan Kaltenbrunner wrote: >> What's weird about using the role name?  That's our standard way of >> distinguishing between two or more users.  Why invent something new? > > wel a user is not a host/server for me - given there is no real benefit from > using disti

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Stefan Kaltenbrunner
On 12/31/2010 02:39 PM, Robert Haas wrote: On Fri, Dec 31, 2010 at 7:57 AM, Heikki Linnakangas wrote: On 31.12.2010 14:40, Robert Haas wrote: Someone may have proposed this before, but one way of getting standby naming "for free" would be to make the standby names the same as the roles used

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Robert Haas
On Fri, Dec 31, 2010 at 7:57 AM, Heikki Linnakangas wrote: > On 31.12.2010 14:40, Robert Haas wrote: >> >> Someone may have proposed this before, but one way of getting standby >> naming "for free" would be to make the standby names the same as the >> roles used to log in, rather than adding a sep

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Heikki Linnakangas
On 31.12.2010 14:40, Robert Haas wrote: Someone may have proposed this before, but one way of getting standby naming "for free" would be to make the standby names the same as the roles used to log in, rather than adding a separate parameter. We could just recommend to people that they use a sepa

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Heikki Linnakangas
On 31.12.2010 13:48, Simon Riggs wrote: On Fri, 2010-12-31 at 12:06 +0200, Heikki Linnakangas wrote: Regarding the rest of the proposal, I would still prefer the UI discussed here: http://archives.postgresql.org/message-id/4cae030a.2060...@enterprisedb.com It ought to be the same amount of wo

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Robert Haas
On Fri, Dec 31, 2010 at 6:48 AM, Simon Riggs wrote: > I suppose we might regard the feature set I am proposing as being the > same as making synchronous_standbys a USERSET parameter, and allowing > just two options: > "none" - allowing the user to specify async if they wish it > "*" - allowing peo

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Aidan Van Dyk
On Fri, Dec 31, 2010 at 5:26 AM, Simon Riggs wrote: > Your picture above is a common misconception. I will add something to > the docs to explain this. > 2. "sync" does not guarantee that the updates to the standbys are in any > way coordinated. You can run a query on one standby and get one ans

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Simon Riggs
On Thu, 2010-12-30 at 20:26 -0700, Joshua Tolley wrote: > 2) initiate fsync on the primary first > >- In this case, the slave is always slightly behind. If if your > > primary falls over, you don't give commit messages to the clients, > but > > if it recovers, it might have committed data, and

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Simon Riggs
On Fri, 2010-12-31 at 12:06 +0200, Heikki Linnakangas wrote: > Regarding the rest of the proposal, I would still prefer the UI > discussed here: > > http://archives.postgresql.org/message-id/4cae030a.2060...@enterprisedb.com > > It ought to be the same amount of work to implement, and provides

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Simon Riggs
On Fri, 2010-12-31 at 12:06 +0200, Heikki Linnakangas wrote: > On 31.12.2010 09:50, Hannu Krosing wrote: > > On 30.12.2010 22:27, Robert Haas wrote: > >> On Thu, Dec 30, 2010 at 2:04 PM, Simon Riggs > >> wrote: > >>> synchronous_replication (boolean) > >>> Specifies whether transaction commit will

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Stefan Kaltenbrunner
On 12/31/2010 11:06 AM, Heikki Linnakangas wrote: On 31.12.2010 09:50, Hannu Krosing wrote: On 30.12.2010 22:27, Robert Haas wrote: On Thu, Dec 30, 2010 at 2:04 PM, Simon Riggs wrote: synchronous_replication (boolean) Specifies whether transaction commit will wait for WAL records to be replica

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Simon Riggs
On Fri, 2010-12-31 at 09:27 +0100, Stefan Kaltenbrunner wrote: > Maybe it has been discussed but I still don't see way it makes any > sense. If I declare a standby a sync standby I better want it sync - not > "maybe sync". consider the case of a 1 master and two identical sync > standbys - one

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Heikki Linnakangas
On 31.12.2010 09:50, Hannu Krosing wrote: On 30.12.2010 22:27, Robert Haas wrote: On Thu, Dec 30, 2010 at 2:04 PM, Simon Riggs wrote: synchronous_replication (boolean) Specifies whether transaction commit will wait for WAL records to be replicated before the command returns a "success" indicati

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Stefan Kaltenbrunner
On 12/30/2010 10:23 PM, Simon Riggs wrote: On Thu, 2010-12-30 at 21:42 +0100, Stefan Kaltenbrunner wrote: Synchronous replication offers the ability to guarantee that all changes made by a transaction have been transferred to at least one remote standby server. This is an extension to the stand

Re: [HACKERS] Sync Rep Design

2010-12-31 Thread Stefan Kaltenbrunner
On 12/30/2010 10:27 PM, Simon Riggs wrote: On Thu, 2010-12-30 at 22:08 +0100, Stefan Kaltenbrunner wrote: On 12/30/2010 10:01 PM, Simon Riggs wrote: On Thu, 2010-12-30 at 15:51 -0500, Robert Treat wrote: Still, one thing that has me concerned is that in the case of two slaves, you don't know

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Hannu Krosing
On 31.12.2010 6:02, Robert Haas wrote: On Thu, Dec 30, 2010 at 8:57 PM, Simon Riggs wrote: I'm not very clear what your response has to do with Stefan's comments. My general perspective is that MySQL released a simple design a year ahead of us, which should be to our collective shame. I will b

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Hannu Krosing
On 30.12.2010 22:27, Robert Haas wrote: On Thu, Dec 30, 2010 at 2:04 PM, Simon Riggs wrote: synchronous_replication (boolean) Specifies whether transaction commit will wait for WAL records to be replicated before the command returns a "success" indication to the client.

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Robert Haas
On Thu, Dec 30, 2010 at 8:57 PM, Simon Riggs wrote: > I'm not very clear what your response has to do with Stefan's comments. > > My general perspective is that MySQL released a simple design a year > ahead of us, which should be to our collective shame. I will be working > towards delivering some

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Joshua Tolley
On Thu, Dec 30, 2010 at 03:24:09PM -0500, Aidan Van Dyk wrote: > On Thu, Dec 30, 2010 at 3:07 PM, Robert Treat wrote: > > >> If primary crashes while commits are waiting for acknowledgement, those > >> transactions will be marked fully committed if the primary database > >> recovers, no matter ho

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 16:47 -0500, Robert Haas wrote: > > It also does not address the more general (not sync rep specific) problem of > > how to deal with max_keep_segments which is a wart and I was hoping we could > > get rid of in 9.1 - but it would require a real standby registration or at > >

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 18:47 -0600, Jim Nasby wrote: > On Dec 30, 2010, at 3:27 PM, Robert Haas wrote: > >> synchronous_replication (boolean) > >>Specifies whether transaction commit will wait for WAL records > >>to be replicated before the command returns a "success" > >>ind

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 16:47 -0500, Robert Haas wrote: > On Thu, Dec 30, 2010 at 3:42 PM, Stefan Kaltenbrunner > wrote: > >> synchronous replication for high performance applications. This feature > >> is unique to PostgreSQL. > > > > that seems to be a bit too much marketing for a reference level

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 16:27 -0500, Robert Haas wrote: > I think it's a bad idea to use the same parameter to mean different > things on the master and standby. Obviously if you phrase it like that, nobody would disagree. I would say I have used the same parameter on both sides in a balanced way

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Jim Nasby
On Dec 30, 2010, at 3:27 PM, Robert Haas wrote: >> synchronous_replication (boolean) >>Specifies whether transaction commit will wait for WAL records >>to be replicated before the command returns a "success" >>indication to the client. > > The word "replicated" here could b

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Robert Haas
On Thu, Dec 30, 2010 at 3:42 PM, Stefan Kaltenbrunner wrote: >> synchronous replication for high performance applications. This feature >> is unique to PostgreSQL. > > that seems to be a bit too much marketing for a reference level document +1. > It also does not address the more general (not sy

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 22:08 +0100, Stefan Kaltenbrunner wrote: > On 12/30/2010 10:01 PM, Simon Riggs wrote: > > On Thu, 2010-12-30 at 15:51 -0500, Robert Treat wrote: > > > >> Still, one thing that has me concerned is that in the case of two > >> slaves, you don't know which one is the more up-to-d

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Robert Haas
On Thu, Dec 30, 2010 at 2:04 PM, Simon Riggs wrote: > We use a single parameter to enable synchronous replication, set in > postgresql.conf on both primary and standby servers: > > synchronous_replication = off (default) | on > > On the primary, synchronous_replication can be set for particular us

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 21:42 +0100, Stefan Kaltenbrunner wrote: > > > > Synchronous replication offers the ability to guarantee that all changes > > made by a transaction have been transferred to at least one remote > > standby server. This is an extension to the standard level of durability > > off

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 22:11 +0200, Marti Raudsepp wrote: > I think a comment about the "head-of-line blocking" nature of > streaming repliaction is in order. If you execute massive writes in > async mode and then run a transaction in sync mode, its commit will be > delayed until all the async tran

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Stefan Kaltenbrunner
On 12/30/2010 10:01 PM, Simon Riggs wrote: On Thu, 2010-12-30 at 15:51 -0500, Robert Treat wrote: Still, one thing that has me concerned is that in the case of two slaves, you don't know which one is the more up-to-date one if you need to failover. It'd be nice if you could just guarantee they

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 15:51 -0500, Robert Treat wrote: > Still, one thing that has me concerned is that in the case of two > slaves, you don't know which one is the more up-to-date one if you > need to failover. It'd be nice if you could just guarantee they both > are... Regrettably, nobody can k

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 15:07 -0500, Robert Treat wrote: > > When allow_standalone_primary is set, a user will stop waiting once > the > > replication_timeout has been reached for their specific session. > Users > > are not waiting for a specific standby to reply, they are waiting > for a > > reply f

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Robert Treat
On Thu, Dec 30, 2010 at 3:36 PM, Simon Riggs wrote: > > On Thu, 2010-12-30 at 15:07 -0500, Robert Treat wrote: > > > If more than one standby server specifies synchronous_replication, > > then > > > whichever standby replies first will release waiting commits. > > > I don't want you to think I am

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Aidan Van Dyk
On Thu, Dec 30, 2010 at 3:07 PM, Robert Treat wrote: >> If primary crashes while commits are waiting for acknowledgement, those >> transactions will be marked fully committed if the primary database >> recovers, no matter how allow_standalone_primary is set. > > This seems backwards; if you are w

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Stefan Kaltenbrunner
On 12/30/2010 08:04 PM, Simon Riggs wrote: On Thu, 2010-12-30 at 18:42 +0100, Stefan Kaltenbrunner wrote: it would help if this would just be a simple text-only description of the design that people can actually comment on inline. I don't think sending technical design proposals as a pdf (which

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 15:07 -0500, Robert Treat wrote: > > We use a single parameter to enable synchronous replication, set in > > postgresql.conf on both primary and standby servers: > > > > synchronous_replication = off (default) | on > > > > On the primary, synchronous_replication can be set for

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 15:07 -0500, Robert Treat wrote: > > If more than one standby server specifies synchronous_replication, > then > > whichever standby replies first will release waiting commits. > I don't want you to think I am setting an expectation, but I'm curious > about the possibility of

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Marti Raudsepp
Most of your doc uses the terms "primary" and "standby", but a few instances of "master" and "slave" have slipped in. I think it's better to stick to consistent terminology. On Thu, Dec 30, 2010 at 21:04, Simon Riggs wrote: > With synchronous replication options specified at the application level

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Robert Treat
On Thu, Dec 30, 2010 at 2:04 PM, Simon Riggs wrote: > On Thu, 2010-12-30 at 18:42 +0100, Stefan Kaltenbrunner wrote: > > > it would help if this would just be a simple text-only description of > > the design that people can actually comment on inline. I don't think > > sending technical design pr

Re: [HACKERS] Sync Rep Design

2010-12-30 Thread Simon Riggs
On Thu, 2010-12-30 at 18:42 +0100, Stefan Kaltenbrunner wrote: > it would help if this would just be a simple text-only description of > the design that people can actually comment on inline. I don't think > sending technical design proposals as a pdf (which seems to be written > in doc-style a

  1   2   >