Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-25 Thread Fujii Masao
On Sat, Mar 19, 2011 at 4:29 AM, Simon Riggs wrote: > On Fri, 2011-03-18 at 20:19 +0100, Markus Wanner wrote: >> Simon, >> >> On 03/18/2011 05:19 PM, Simon Riggs wrote: >> >>> Simon Riggs  wrote: >> In PostgreSQL other users cannot observe the commit until an >> acknowledgement has been

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-23 Thread Robert Haas
On Wed, Mar 23, 2011 at 8:16 AM, Markus Wanner wrote: > On 03/23/2011 12:52 PM, Robert Haas wrote: >> Yes.  What this won't do is let you build a big load-balancing network >> (at least not without great caution about what you assume). > > This sounds too strong to me.  Session-aware load balancin

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-23 Thread Markus Wanner
On 03/23/2011 12:52 PM, Robert Haas wrote: > Yes. What this won't do is let you build a big load-balancing network > (at least not without great caution about what you assume). This sounds too strong to me. Session-aware load balancing is pretty common these days. It's the default mode of PgBou

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-23 Thread Robert Haas
On Wed, Mar 23, 2011 at 3:27 AM, Markus Wanner wrote: > On 03/22/2011 09:33 PM, Robert Haas wrote: >> We might have a version of synchronous replication that works this way >> some day, but it's not the version were shipping with 9.1.  The slave >> acknowledges the WAL records when they hit the di

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-23 Thread Markus Wanner
On 03/22/2011 09:33 PM, Robert Haas wrote: > We might have a version of synchronous replication that works this way > some day, but it's not the version were shipping with 9.1. The slave > acknowledges the WAL records when they hit the disk (i.e. fsync) not > when they are applied; WAL apply can l

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-22 Thread Robert Haas
On Fri, Mar 18, 2011 at 5:08 PM, Aidan Van Dyk wrote: > On Fri, Mar 18, 2011 at 3:41 PM, Markus Wanner wrote: >> On 03/18/2011 08:29 PM, Simon Riggs wrote: >>> We could do that easily enough, actually, if we wished. >>> >>> Do we wish? >> >> I personally don't see any problem letting a standby sh

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-22 Thread Aidan Van Dyk
On Fri, Mar 18, 2011 at 3:41 PM, Markus Wanner wrote: > On 03/18/2011 08:29 PM, Simon Riggs wrote: >> We could do that easily enough, actually, if we wished. >> >> Do we wish? > > I personally don't see any problem letting a standby show a snapshot > before the master.  I'd consider it unneeded ne

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-19 Thread Markus Wanner
On 03/18/2011 10:48 PM, Kevin Grittner wrote: > the least of the evils. I guess we should document it, though, so > nobody has a false expectation that seeing something on the replica > means that a connection looking at the master will see something > that current. Agreed. Note, however, that e

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 5:48 PM, Kevin Grittner wrote: > Robert Haas wrote: >> Well, the idea is that we don't want to let people depend on the >> value until it's guaranteed to be durably committed. > > OK, so if you see it on the replica, you know it is in at least two > places.  I guess that m

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Robert Haas wrote: > Well, the idea is that we don't want to let people depend on the > value until it's guaranteed to be durably committed. OK, so if you see it on the replica, you know it is in at least two places. I guess that makes sense. It kinda "feels" wrong to see a view of the repli

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 5:24 PM, Kevin Grittner wrote: > Robert Haas wrote: > >> Since the current solution is intended to support data-loss-free >> failover, but NOT to guarantee a consistent view of the world from >> a SQL level, I doubt it's worth paying any price for this. > > Well, that brin

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 16:24 -0500, Kevin Grittner wrote: > Robert Haas wrote: > > > Since the current solution is intended to support data-loss-free > > failover, but NOT to guarantee a consistent view of the world from > > a SQL level, I doubt it's worth paying any price for this. > > Well, t

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 17:08 -0400, Aidan Van Dyk wrote: > On Fri, Mar 18, 2011 at 3:41 PM, Markus Wanner wrote: > > On 03/18/2011 08:29 PM, Simon Riggs wrote: > >> We could do that easily enough, actually, if we wished. > >> > >> Do we wish? > > > > I personally don't see any problem letting a sta

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Robert Haas wrote: > Since the current solution is intended to support data-loss-free > failover, but NOT to guarantee a consistent view of the world from > a SQL level, I doubt it's worth paying any price for this. Well, that brings us back to the question of why we would want to suppress the

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 3:29 PM, Simon Riggs wrote: > On Fri, 2011-03-18 at 20:19 +0100, Markus Wanner wrote: >> Simon, >> >> On 03/18/2011 05:19 PM, Simon Riggs wrote: >> >>> Simon Riggs  wrote: >> In PostgreSQL other users cannot observe the commit until an >> acknowledgement has been

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
On 03/18/2011 08:29 PM, Simon Riggs wrote: > We could do that easily enough, actually, if we wished. > > Do we wish? I personally don't see any problem letting a standby show a snapshot before the master. I'd consider it unneeded network traffic. But then again, I'm completely biased. Regards

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Simon Riggs wrote: > On Fri, 2011-03-18 at 20:19 +0100, Markus Wanner wrote: >> >>> Simon Riggs wrote: >> In PostgreSQL other users cannot observe the commit until an >> acknowledgement has been received. >> >> On other nodes as well? To me that means the standby needs to >> hold ba

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 20:19 +0100, Markus Wanner wrote: > Simon, > > On 03/18/2011 05:19 PM, Simon Riggs wrote: > >>> Simon Riggs wrote: > In PostgreSQL other users cannot observe the commit until an > acknowledgement has been received. > > On other nodes as well? To me that means the

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
On 03/18/2011 05:27 PM, Kevin Grittner wrote: > Basically, what Heikki addresses. It has to be committed after > crash and recovery, and deal with replicas which may or may not have > been notified and may or may not have applied the transaction. Huh? I'm not quite following here. Committing ad

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
Simon, On 03/18/2011 05:19 PM, Simon Riggs wrote: >>> Simon Riggs wrote: In PostgreSQL other users cannot observe the commit until an acknowledgement has been received. On other nodes as well? To me that means the standby needs to hold back COMMIT of an ACKed transaction, until receiv

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
On 03/18/2011 06:35 PM, Greg Stark wrote: > I think promising that the COMMIT doesn't return until the transaction > and all previous transactions are replicated is enough. We don't have > to promise that nobody else will see it either. Those same > transactions eventually have to commit as well N

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Greg Stark
On Fri, Mar 18, 2011 at 4:33 PM, Robert Haas wrote: > The fundamental problem here is that once you update CLOG and flush > the corresponding WAL record, there is no going backward.  You can > hold the system in some intermediate state where the transaction still > holds locks and is excluded from

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Robert Haas wrote: > Simon Riggs wrote: >> No, only in the case where you choose not to failover to the >> standby when you crash, which would be a fairly strange choice >> after the effort to set up the standby. In a correctly configured >> and operated cluster what I say above is fully correc

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 12:19 PM, Simon Riggs wrote: > On Fri, 2011-03-18 at 17:47 +0200, Heikki Linnakangas wrote: >> On 18.03.2011 16:52, Kevin Grittner wrote: >> > Simon Riggs  wrote: >> > >> >> In PostgreSQL other users cannot observe the commit until an >> >> acknowledgement has been received

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
> On 18.03.2011 16:52, Kevin Grittner wrote: >> Simon Riggs wrote: >> >>> In PostgreSQL other users cannot observe the commit until an >>> acknowledgement has been received. >> >> Really? I hadn't picked up on that. That makes for a lot of >> complication on crash-and-recovery of a master, but

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 17:47 +0200, Heikki Linnakangas wrote: > On 18.03.2011 16:52, Kevin Grittner wrote: > > Simon Riggs wrote: > > > >> In PostgreSQL other users cannot observe the commit until an > >> acknowledgement has been received. > > > > Really? I hadn't picked up on that. That makes fo

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread MARK CALLAGHAN
On Fri, Mar 18, 2011 at 2:37 PM, Markus Wanner wrote: > Hi, > > On 03/18/2011 02:40 PM, Kevin Grittner wrote: >> Then the only thing you would consider sync replication, as far as I >> can see, is two phase commit > > I think waiting for the ACK before actually making the changes from the > transa

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread MARK CALLAGHAN
On Fri, Mar 18, 2011 at 2:19 PM, Markus Wanner wrote: > Their documentation [1] isn't entirely clear on that first: "the master > blocks after the commit is done and waits until at least one > semisynchronous slave acknowledges that it has received all events for > the transaction" and the "slave

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Heikki Linnakangas
On 18.03.2011 16:52, Kevin Grittner wrote: Simon Riggs wrote: In PostgreSQL other users cannot observe the commit until an acknowledgement has been received. Really? I hadn't picked up on that. That makes for a lot of complication on crash-and-recovery of a master, but if we can pull it of

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
On 03/18/2011 03:52 PM, Kevin Grittner wrote: > Really? I hadn't picked up on that. That makes for a lot of > complication on crash-and-recovery of a master What complication do you have in mind here? I think of it the opposite way (at least for Postgres, that is): committing a transaction that

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 11:07 -0400, Robert Haas wrote: > On Fri, Mar 18, 2011 at 10:55 AM, Greg Stark wrote: > > On Thu, Mar 17, 2011 at 5:46 PM, Robert Haas wrote: > >> What makes more sense to me after having thought about this more > >> carefully is to simply make a blanket rule that when > >>

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Simon Riggs wrote: > In PostgreSQL other users cannot observe the commit until an > acknowledgement has been received. Really? I hadn't picked up on that. That makes for a lot of complication on crash-and-recovery of a master, but if we can pull it off, that's really cool. If we do that and

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
Hi, On 03/18/2011 02:40 PM, Kevin Grittner wrote: > Then the only thing you would consider sync replication, as far as I > can see, is two phase commit I think waiting for the ACK before actually making the changes from the transaction visible (COMMIT) would suffice for disallowing such an incons

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
Mark, On 03/18/2011 02:16 PM, MARK CALLAGHAN wrote: > We didn't invent the term, we just implemented something that Heikki > Tuuri briefly described, for example: > http://bugs.mysql.com/bug.php?id=7440 Oh, okay, good to know who to blame ;-) However, I didn't mean to offend anybody. > I do not

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 13:16 +, MARK CALLAGHAN wrote: > On Fri, Mar 18, 2011 at 9:27 AM, Markus Wanner wrote: > > Google invented the term "semi-syncronous" for something that's > > essentially the same that we have, now, I think. However, I full > > heartedly hate that term (based on the reas

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
MARK CALLAGHAN wrote: > Markus Wanner wrote: >> Google invented the term "semi-syncronous" for something that's >> essentially the same that we have, now, I think. However, I full >> heartedly hate that term (based on the reasoning that there's no >> semi-pregnant, either). To be fair, what

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 9:16 AM, MARK CALLAGHAN wrote: > On Fri, Mar 18, 2011 at 9:27 AM, Markus Wanner wrote: >> Google invented the term "semi-syncronous" for something that's >> essentially the same that we have, now, I think.  However, I full >> heartedly hate that term (based on the reasonin

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread MARK CALLAGHAN
On Fri, Mar 18, 2011 at 9:27 AM, Markus Wanner wrote: > Google invented the term "semi-syncronous" for something that's > essentially the same that we have, now, I think.  However, I full > heartedly hate that term (based on the reasoning that there's no > semi-pregnant, either). We didn't invent

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
Hi, sorry for being late to join that bike-shedding discussion. On 03/07/2011 05:09 PM, Alvaro Herrera wrote: > I think these terms are used inconsistenly enough across the industry > that what would make the most sense would be to use the common term and > document accurately what we mean by it,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Alvaro Herrera
Excerpts from Andrew Dunstan's message of lun mar 07 12:51:49 -0300 2011: > > On 03/07/2011 10:46 AM, Heikki Linnakangas wrote: > > Hmm, I've read that wikipedia definition before, but the "atomic" part > > never caught my eye. You do get zero data loss with what we have; if a > > meteor strike

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Andrew Dunstan
On 03/07/2011 10:46 AM, Heikki Linnakangas wrote: On 07.03.2011 17:03, Andrew Dunstan wrote: This is about expectations. The thing that worries me is that the use of this term might cause some people NOT to use 2PC because they think they are getting an equivalent guarantee, when in fact they

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Heikki Linnakangas
On 07.03.2011 17:03, Andrew Dunstan wrote: This is about expectations. The thing that worries me is that the use of this term might cause some people NOT to use 2PC because they think they are getting an equivalent guarantee, when in fact they are not. And that's hardly unreasonable. Here for exa

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Kevin Grittner
Andrew Dunstan wrote: > Synchronous replication - guarantees "zero data loss" by the > means of atomic write operation, i.e. write either completes on > both sides or not at all. So far, so good. > Write is not considered complete until acknowledgement by both > local and remote st

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Aidan Van Dyk
On Mon, Mar 7, 2011 at 2:29 PM, Aidan Van Dyk wrote: > They they are either already hosed or already using 2PC. Sorry, to expand on my all too brief comment, even *without* replication, they are hosed. Once you issue commit, you have know knowledge if the commit is durable, (or even posibly see

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Andrew Dunstan
On 03/07/2011 09:29 AM, Aidan Van Dyk wrote: On Mon, Mar 7, 2011 at 2:21 PM, Andrew Dunstan wrote: For me, that's enough to call it "synchronous replication". It provides a useful guarantee to the client. But you could argue for an even stricter definition, requiring atomicity so that if a t

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Kevin Grittner
Heikki Linnakangas wrote: > if you pull the power plug, the transaction that was just being > committed might be committed on the master, but not yet on the > standby. > For me, that's enough to call it "synchronous replication". It > provides useful guarantee to the client. I don't think mo

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Aidan Van Dyk
On Mon, Mar 7, 2011 at 2:21 PM, Andrew Dunstan wrote: >> For me, that's enough to call it "synchronous replication". It provides a >> useful guarantee to the client. But you could argue for an even stricter >> definition, requiring atomicity so that if a transaction is not successfully >> replica

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Andrew Dunstan
On 03/07/2011 09:02 AM, Heikki Linnakangas wrote: On 07.03.2011 15:30, Andrew Dunstan wrote: Previously, Simon said: Truly "synchronous" requires two-phase commit, which this never was. So I too am confused about how it's now become "truly synchronous". Are we saying this give the same or

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Heikki Linnakangas
On 07.03.2011 15:30, Andrew Dunstan wrote: Previously, Simon said: Truly "synchronous" requires two-phase commit, which this never was. So I too am confused about how it's now become "truly synchronous". Are we saying this give the same or better guarantees than a 2PC setup? The guarantee w

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Andrew Dunstan
On 03/07/2011 02:29 AM, Heikki Linnakangas wrote: On 07.03.2011 01:28, Simon Riggs wrote: On Sun, 2011-03-06 at 18:09 -0500, Andrew Dunstan wrote: On 03/06/2011 05:51 PM, Simon Riggs wrote: Efficient transaction-controlled synchronous replication. I'm glad this is in, but I thought we agr

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Simon Riggs
On Mon, 2011-03-07 at 17:44 +0900, Fujii Masao wrote: > The above check should be required also after pg_ctl reload since > synchronous_standby_names can be changed by SIGHUP? > Or how about just removing that? If the patch I submitted is > committed,empty synchronous_standby_names and max_wal_sen

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-06 Thread Heikki Linnakangas
On 07.03.2011 09:48, Simon Riggs wrote: On Mon, 2011-03-07 at 09:29 +0200, Heikki Linnakangas wrote: I presume you didn't make allow_synchronous_standby=off the default behavior. Sorry, s/allow_synchronous_standby/allow_standalone_master You presume incorrectly. Ok, ok then. Thank you! Lo

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-06 Thread Simon Riggs
On Mon, 2011-03-07 at 09:29 +0200, Heikki Linnakangas wrote: > I presume you didn't make allow_synchronous_standby=off the default > behavior. You presume incorrectly. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-06 Thread Heikki Linnakangas
On 07.03.2011 01:28, Simon Riggs wrote: On Sun, 2011-03-06 at 18:09 -0500, Andrew Dunstan wrote: On 03/06/2011 05:51 PM, Simon Riggs wrote: Efficient transaction-controlled synchronous replication. I'm glad this is in, but I thought we agreed NOT to call it "synchronous replication". The d

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-06 Thread Simon Riggs
On Sun, 2011-03-06 at 18:09 -0500, Andrew Dunstan wrote: > > On 03/06/2011 05:51 PM, Simon Riggs wrote: > > Efficient transaction-controlled synchronous replication. > > > > I'm glad this is in, but I thought we agreed NOT to call it "synchronous > replication". The discussion on the thread was