RE: Multi-Master Logical Replication

2022-06-14 Thread r.takahash...@fujitsu.com
Hi Kuroda san, > I think even if LRG is implemented as contrib modules or any extensions, > it will deeply depend on the subscription option "origin" proposed in [1]. > So LRG cannot be used for older version, only PG16 or later. Sorry, I misunderstood. I understand now. Regards, Ryohei

RE: Multi-Master Logical Replication

2022-06-14 Thread kuroda.hay...@fujitsu.com
Dear Takahashi-san, Thanks for giving feedbacks! > > I don't know if it requires the kind of code you are thinking but I > > agree that it is worth considering implementing it as an extension. > > I think the other advantage to implement as an extension is that users could > install the

RE: Multi-Master Logical Replication

2022-06-13 Thread r.takahash...@fujitsu.com
Hi, In addition to the use cases mentioned above, some users want to use n-way replication of partial database. The following is the typical use case. * There are several data centers. (ex. Japan and India) * The database in each data center has its unique data. (ex. the database in Japan

Re: Multi-Master Logical Replication

2022-06-10 Thread Amit Kapila
On Fri, Jun 10, 2022 at 12:40 PM Bharath Rupireddy wrote: > > On Fri, Jun 10, 2022 at 9:54 AM Amit Kapila wrote: > > > > > > 1. Are you proposing to use logical replication subscribers to be in > > > sync quorum? In other words, in an N-masters node, M (M >= N)-node > > > configuration, will

Re: Multi-Master Logical Replication

2022-06-10 Thread Bharath Rupireddy
On Fri, Jun 10, 2022 at 9:54 AM Amit Kapila wrote: > > On Thu, Jun 9, 2022 at 6:04 PM Bharath Rupireddy > wrote: > > > > On Thu, Apr 28, 2022 at 5:20 AM Peter Smith wrote: > > > > > > MULTI-MASTER LOGICAL REPLICATION > > > > > > 1.0 BACK

Re: Multi-Master Logical Replication

2022-06-09 Thread Amit Kapila
On Thu, Jun 9, 2022 at 6:04 PM Bharath Rupireddy wrote: > > On Thu, Apr 28, 2022 at 5:20 AM Peter Smith wrote: > > > > MULTI-MASTER LOGICAL REPLICATION > > > > 1.0 BACKGROUND > > > > Let’s assume that a user wishes to set up a multi-master environment

Re: Multi-Master Logical Replication

2022-06-09 Thread Bharath Rupireddy
On Thu, Apr 28, 2022 at 5:20 AM Peter Smith wrote: > > MULTI-MASTER LOGICAL REPLICATION > > 1.0 BACKGROUND > > Let’s assume that a user wishes to set up a multi-master environment > so that a set of PostgreSQL instances (nodes) use logical replication > to share tabl

RE: Multi-Master Logical Replication

2022-06-06 Thread kuroda.hay...@fujitsu.com
Dear hackers, I found another use-case for LRG. It might be helpful for migration. LRG for migration -- LRG may be helpful for machine migration, OS upgrade, or PostgreSQL itself upgrade. Assumes that users want to migrate database to other environment,

Re: Multi-Master Logical Replication

2022-06-04 Thread Amit Kapila
On Fri, Jun 3, 2022 at 7:12 AM Bruce Momjian wrote: > > On Thu, Jun 2, 2022 at 05:12:49PM +1000, Peter Smith wrote: > > On Thu, Jun 2, 2022 at 12:03 AM Bruce Momjian wrote: > > > > > > On Wed, Jun 1, 2022 at 10:27:27AM +0530, Amit Kapila wrote: > > ... > > > > > My big point is that you should

Re: Multi-Master Logical Replication

2022-06-02 Thread Bruce Momjian
On Thu, Jun 2, 2022 at 05:12:49PM +1000, Peter Smith wrote: > On Thu, Jun 2, 2022 at 12:03 AM Bruce Momjian wrote: > > > > On Wed, Jun 1, 2022 at 10:27:27AM +0530, Amit Kapila wrote: > ... > > > My big point is that you should not be showing up with a patch but > > rather have these

Re: Multi-Master Logical Replication

2022-06-02 Thread Peter Smith
On Thu, Jun 2, 2022 at 12:03 AM Bruce Momjian wrote: > > On Wed, Jun 1, 2022 at 10:27:27AM +0530, Amit Kapila wrote: ... > My big point is that you should not be showing up with a patch but > rather have these discussions to get agreement that this is the > direction the community wants to go.

Re: Multi-Master Logical Replication

2022-06-02 Thread Amit Kapila
On Wed, Jun 1, 2022 at 7:33 PM Bruce Momjian wrote: > > On Wed, Jun 1, 2022 at 10:27:27AM +0530, Amit Kapila wrote: > > On Tue, May 31, 2022 at 7:36 PM Bruce Momjian wrote: > > > Uh, thinking some more, why would anyone set things up this way --- > > > having part of a table being primary on

Re: Multi-Master Logical Replication

2022-06-01 Thread Bruce Momjian
On Wed, Jun 1, 2022 at 10:27:27AM +0530, Amit Kapila wrote: > On Tue, May 31, 2022 at 7:36 PM Bruce Momjian wrote: > > Uh, thinking some more, why would anyone set things up this way --- > > having part of a table being primary on one server and a different part > > of the table be a subscriber.

Re: Multi-Master Logical Replication

2022-05-31 Thread Amit Kapila
On Tue, May 31, 2022 at 7:36 PM Bruce Momjian wrote: > > On Wed, May 25, 2022 at 10:32:50PM -0400, Bruce Momjian wrote: > > On Wed, May 25, 2022 at 12:13:17PM +0530, Amit Kapila wrote: > > > > > > It helps with setting up logical replication among two or more nodes > > > (data flows both ways)

Re: Multi-Master Logical Replication

2022-05-31 Thread Bruce Momjian
On Wed, May 25, 2022 at 10:32:50PM -0400, Bruce Momjian wrote: > On Wed, May 25, 2022 at 12:13:17PM +0530, Amit Kapila wrote: > > > You still have not answered my question above. "Without these features, > > > what workload would this help with?" You have only explained how the > > > patch would

Re: Multi-Master Logical Replication

2022-05-25 Thread Bruce Momjian
On Wed, May 25, 2022 at 12:13:17PM +0530, Amit Kapila wrote: > > You still have not answered my question above. "Without these features, > > what workload would this help with?" You have only explained how the > > patch would fix one of the many larger problems. > > > > It helps with setting up

Re: Multi-Master Logical Replication

2022-05-25 Thread Peter Smith
On Wed, May 25, 2022 at 4:43 PM Amit Kapila wrote: > > On Tue, May 24, 2022 at 5:57 PM Bruce Momjian wrote: > > > > On Sat, May 14, 2022 at 12:20:05PM +0530, Amit Kapila wrote: > > > On Sat, May 14, 2022 at 12:33 AM Bruce Momjian wrote: > > > > > > > > Uh, without these features, what workload

Re: Multi-Master Logical Replication

2022-05-25 Thread Amit Kapila
On Tue, May 24, 2022 at 5:57 PM Bruce Momjian wrote: > > On Sat, May 14, 2022 at 12:20:05PM +0530, Amit Kapila wrote: > > On Sat, May 14, 2022 at 12:33 AM Bruce Momjian wrote: > > > > > > Uh, without these features, what workload would this help with? > > > > > > > To allow replication among

Re: Multi-Master Logical Replication

2022-05-24 Thread Bruce Momjian
On Sat, May 14, 2022 at 12:20:05PM +0530, Amit Kapila wrote: > On Sat, May 14, 2022 at 12:33 AM Bruce Momjian wrote: > > > > Uh, without these features, what workload would this help with? > > > > To allow replication among multiple nodes when some of the nodes may > have pre-existing data. This

RE: Multi-Master Logical Replication

2022-05-18 Thread kuroda.hay...@fujitsu.com
Hi hackers, I created a small PoC. Please see the attached patches. REQUIREMENT Before patching them, patches in [1] must also be applied. DIFFERENCES FROM PREVIOUS DESCRIPTIONS * LRG is now implemented as SQL functions, not as a contrib module. * New tables are added as system catalogs.

Re: Multi-Master Logical Replication

2022-05-14 Thread Amit Kapila
On Sat, May 14, 2022 at 12:33 AM Bruce Momjian wrote: > > Uh, without these features, what workload would this help with? > To allow replication among multiple nodes when some of the nodes may have pre-existing data. This work plans to provide simple APIs to achieve that. Now, let me try to

Re: Multi-Master Logical Replication

2022-05-13 Thread Bruce Momjian
On Fri, Apr 29, 2022 at 07:05:11PM +1000, Peter Smith wrote: > This MMLR proposal was mostly just to create an interface making it > easier to use PostgreSQL core logical replication CREATE > PUBLICATION/SUBSCRIPTION for table sharing among a set of nodes. > Otherwise, this is difficult for a user

Re: Multi-Master Logical Replication

2022-05-11 Thread vignesh C
On Thu, Apr 28, 2022 at 5:20 AM Peter Smith wrote: > > MULTI-MASTER LOGICAL REPLICATION > > 1.0 BACKGROUND > > Let’s assume that a user wishes to set up a multi-master environment > so that a set of PostgreSQL instances (nodes) use logical replication > to share tabl

Re: Multi-Master Logical Replication

2022-04-29 Thread vignesh C
On Fri, Apr 29, 2022 at 2:35 PM Peter Smith wrote: > > On Fri, Apr 29, 2022 at 2:16 PM Yura Sokolov wrote: > > > > В Чт, 28/04/2022 в 17:37 +0530, vignesh C пишет: > > > On Thu, Apr 28, 2022 at 4:24 PM Yura Sokolov > > > wrote: > > > > В Чт, 28/04/2022 в 09:49 +1000, Peter Smith пишет: > > > >

Re: Multi-Master Logical Replication

2022-04-29 Thread Peter Smith
On Fri, Apr 29, 2022 at 2:16 PM Yura Sokolov wrote: > > В Чт, 28/04/2022 в 17:37 +0530, vignesh C пишет: > > On Thu, Apr 28, 2022 at 4:24 PM Yura Sokolov > > wrote: > > > В Чт, 28/04/2022 в 09:49 +1000, Peter Smith пишет: > > > > > > > 1.1 ADVANTAGES OF MMLR > > > > > > > > - Increases write

Re: Multi-Master Logical Replication

2022-04-28 Thread Yura Sokolov
В Чт, 28/04/2022 в 17:37 +0530, vignesh C пишет: > On Thu, Apr 28, 2022 at 4:24 PM Yura Sokolov wrote: > > В Чт, 28/04/2022 в 09:49 +1000, Peter Smith пишет: > > > > > 1.1 ADVANTAGES OF MMLR > > > > > > - Increases write scalability (e.g., all nodes can write arbitrary data). > > > > I've

Re: Multi-Master Logical Replication

2022-04-28 Thread vignesh C
On Thu, Apr 28, 2022 at 4:24 PM Yura Sokolov wrote: > > В Чт, 28/04/2022 в 09:49 +1000, Peter Smith пишет: > > > 1.1 ADVANTAGES OF MMLR > > > > - Increases write scalability (e.g., all nodes can write arbitrary data). > > I've never heard how transactional-aware multimaster increases > write

Re: Multi-Master Logical Replication

2022-04-28 Thread Yura Sokolov
В Чт, 28/04/2022 в 09:49 +1000, Peter Smith пишет: > 1.1 ADVANTAGES OF MMLR > > - Increases write scalability (e.g., all nodes can write arbitrary data). I've never heard how transactional-aware multimaster increases write scalability. More over, usually even non-transactional multimaster

RE: Multi-Master Logical Replication

2022-04-28 Thread kuroda.hay...@fujitsu.com
Dear Laurenz, Thank you for your interest in our works! > I am missing a discussion how replication conflicts are handled to > prevent replication from breaking or the databases from drifting apart. Actually we don't have plans for developing the feature that avoids conflict. We think that it

Re: Multi-Master Logical Replication

2022-04-28 Thread Laurenz Albe
On Thu, 2022-04-28 at 09:49 +1000, Peter Smith wrote: > To prevent the circular recursive problem Vignesh is developing a > patch [1] that introduces new SUBSCRIPTION options "local_only" (for > publishing only data originating at the publisher node) and > "copy_data=force". Using this patch, we

Multi-Master Logical Replication

2022-04-27 Thread Peter Smith
MULTI-MASTER LOGICAL REPLICATION 1.0 BACKGROUND Let’s assume that a user wishes to set up a multi-master environment so that a set of PostgreSQL instances (nodes) use logical replication to share tables with every other node in the set. We define this as a multi-master logical replication (MMLR