Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-20 Thread Andres Freund
Hi Robert, Hi All! On Wednesday, June 20, 2012 03:08:48 AM Robert Haas wrote: On Tue, Jun 19, 2012 at 2:23 PM, Andres Freund and...@2ndquadrant.com wrote: Well, the words are fuzzy, but I would define logical replication to be something which is independent of the binary format in which

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-19 Thread Andres Freund
Hi, The most important part, even for people not following my discussion with Robert is at the bottom where the possible wal decoding strategies are laid out. On Tuesday, June 19, 2012 03:20:58 AM Robert Haas wrote: On Sat, Jun 16, 2012 at 7:43 AM, Andres Freund and...@2ndquadrant.com wrote:

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-19 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: The problem is just that to support basically arbitrary decoding requirements you need to provide at least those pieces of information in a transactionally consistent manner: * the data * table names * column names * type information *

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 2:23 PM, Andres Freund and...@2ndquadrant.com wrote: Well, the words are fuzzy, but I would define logical replication to be something which is independent of the binary format in which stuff gets stored on disk.  If it's not independent of the disk format, then you

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-18 Thread Robert Haas
On Sat, Jun 16, 2012 at 3:03 PM, Steve Singer st...@ssinger.info wrote: I feel that in-core support to capture changes and turn them into change records that can be replayed on other databases, without relying on triggers and log tables, would be good to have. I think we want some flexible

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-18 Thread Robert Haas
On Sat, Jun 16, 2012 at 7:43 AM, Andres Freund and...@2ndquadrant.com wrote: Hm. Yes, you could do that. But I have to say I don't really see a point. Maybe the fact that I do envision multimaster systems at some point is clouding my judgement though as its far less easy in that case. Why?  

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-16 Thread Andres Freund
Hi Robert, On Friday, June 15, 2012 10:03:38 PM Robert Haas wrote: On Thu, Jun 14, 2012 at 4:13 PM, Andres Freund and...@2ndquadrant.com wrote: I don't plan to throw in loads of conflict resolution smarts. The aim is to get to the place where all the infrastructure is there so that a MM

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-16 Thread Steve Singer
On 12-06-15 04:03 PM, Robert Haas wrote: On Thu, Jun 14, 2012 at 4:13 PM, Andres Freundand...@2ndquadrant.com wrote: I don't plan to throw in loads of conflict resolution smarts. The aim is to get to the place where all the infrastructure is there so that a MM solution can be built by

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-15 Thread Robert Haas
On Thu, Jun 14, 2012 at 4:13 PM, Andres Freund and...@2ndquadrant.com wrote: I don't plan to throw in loads of conflict resolution smarts. The aim is to get to the place where all the infrastructure is there so that a MM solution can be built by basically plugging in a conflict resolution

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-15 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: So maybe instead of trying to cobble together a set of catalog contents that we can use for decoding any tuple whatsoever, we should instead divide the world into well-behaved types and poorly-behaved types. Well-behaved types are those that can be

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-14 Thread Robert Haas
On Wed, Jun 13, 2012 at 7:27 AM, Andres Freund and...@2ndquadrant.com wrote: === Design goals for logical replication === : - in core - fast - async - robust - multi-master - modular - as unintrusive as possible implementation wise - basis for other technologies (sharding, replication

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-14 Thread Andres Freund
Hi Robert, Thanks for your answer. On Thursday, June 14, 2012 06:17:26 PM Robert Haas wrote: On Wed, Jun 13, 2012 at 7:27 AM, Andres Freund and...@2ndquadrant.com wrote: === Design goals for logical replication === : - in core - fast - async - robust - multi-master - modular -

[HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
Hi everyone, This mail contains the highlevel design description of how our prototype of in-core logical replication works. The individual patches will be posted as replies to this email. I obviously welcome all sorts of productive comments to both the individual patches and the architecture.

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Robert Haas
On Wed, Jun 13, 2012 at 7:27 AM, Andres Freund and...@2ndquadrant.com wrote: Unless somebody objects I will add most of the individual marked as RFC to the current commitfest. I hope that with comments stemming from that round we can get several of the patches into the first or second

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
On Wednesday, June 13, 2012 05:53:31 PM Robert Haas wrote: On Wed, Jun 13, 2012 at 7:27 AM, Andres Freund and...@2ndquadrant.com wrote: Unless somebody objects I will add most of the individual marked as RFC to the current commitfest. I hope that with comments stemming from that round we

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Steve Singer
On 12-06-13 07:27 AM, Andres Freund wrote: Its also available in the 'cabal-rebasing' branch on git.postgresql.org/users/andresfreund/postgres.git . That branch will modify history though. That branch has a merge error in f685a11ce43b9694cbe61ffa42e396c9fbc32b05 gcc -O2 -Wall

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
On Wednesday, June 13, 2012 07:11:40 PM Steve Singer wrote: On 12-06-13 07:27 AM, Andres Freund wrote: Its also available in the 'cabal-rebasing' branch on git.postgresql.org/users/andresfreund/postgres.git . That branch will modify history though. That branch has a merge error in

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
On Wednesday, June 13, 2012 07:11:40 PM Steve Singer wrote: On 12-06-13 07:27 AM, Andres Freund wrote: Its also available in the 'cabal-rebasing' branch on git.postgresql.org/users/andresfreund/postgres.git . That branch will modify history though. That branch has a merge error in

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-13 Thread Andres Freund
Hi, The patch as of yet doesn't contain how you actually can use the prototype... Obviously at this point its not very convenient: I have two config files: Node 1: port = 5501 wal_level = logical max_wal_senders = 10 wal_keep_segments = 200 multimaster_conninfo = 'port=5502 host=/tmp'