Re: Synchronizing slots from primary to standby

2023-06-29 Thread Amit Kapila
On Wed, Jun 28, 2023 at 12:19 PM Drouvot, Bertrand wrote: > > On 6/26/23 12:34 PM, Amit Kapila wrote: > > On Mon, Jun 26, 2023 at 11:15 AM Drouvot, Bertrand > > wrote: > >> > >> On 6/20/23 12:22 PM, Amit Kapila wrote: > >>> On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand > >>> wrote: > >> > >>

RE: Synchronizing slots from primary to standby

2023-06-29 Thread Hayato Kuroda (Fujitsu)
Dear Drouvot, Hi, I'm also interested in the feature. Followings are my high-level comments. I did not mention some detailed notations because this patch is not at the stage. And very sorry that I could not follow all of this discussions. 1. I thought that we should not reuse logical replication

Re: Synchronizing slots from primary to standby

2023-06-27 Thread Drouvot, Bertrand
Hi, On 6/26/23 12:34 PM, Amit Kapila wrote: On Mon, Jun 26, 2023 at 11:15 AM Drouvot, Bertrand wrote: On 6/20/23 12:22 PM, Amit Kapila wrote: On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand wrote: In such a case (slot valid on the primary but invalidated on the standby) then I think w

Re: Synchronizing slots from primary to standby

2023-06-26 Thread Amit Kapila
On Mon, Jun 26, 2023 at 11:15 AM Drouvot, Bertrand wrote: > > On 6/20/23 12:22 PM, Amit Kapila wrote: > > On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand > > wrote: > > >> In such a case (slot valid on the primary but invalidated on the standby) > >> then I think we > >> could drop and recreat

Re: Synchronizing slots from primary to standby

2023-06-25 Thread Drouvot, Bertrand
Hi, On 6/20/23 12:22 PM, Amit Kapila wrote: On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand wrote: In such a case (slot valid on the primary but invalidated on the standby) then I think we could drop and recreate the invalidated slot on the standby. Will it be safe? Because after recre

Re: Synchronizing slots from primary to standby

2023-06-20 Thread Amit Kapila
On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand wrote: > > On 6/19/23 12:03 PM, Amit Kapila wrote: > > On Mon, Jun 19, 2023 at 11:34 AM Drouvot, Bertrand > > wrote: > >> > >> Also I think we need to handle the case of invalidated replication > >> slot(s): should > >> we drop/recreate it/them?

Re: Synchronizing slots from primary to standby

2023-06-19 Thread Drouvot, Bertrand
Hi, On 6/19/23 12:03 PM, Amit Kapila wrote: On Mon, Jun 19, 2023 at 11:34 AM Drouvot, Bertrand wrote: Also I think we need to handle the case of invalidated replication slot(s): should we drop/recreate it/them? (as the main goal is to have sync slot(s) on the standby). Do you intend to a

Re: Synchronizing slots from primary to standby

2023-06-19 Thread Amit Kapila
On Mon, Jun 19, 2023 at 11:34 AM Drouvot, Bertrand wrote: > > Also I think we need to handle the case of invalidated replication slot(s): > should > we drop/recreate it/them? (as the main goal is to have sync slot(s) on the > standby). > Do you intend to ask what happens to logical slots invali

Re: Synchronizing slots from primary to standby

2023-06-18 Thread Drouvot, Bertrand
Hi, On 6/16/23 11:56 AM, Amit Kapila wrote: On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand wrote: Please find attached V5 (a rebase of V4 posted up-thread). In addition to the "rebasing" work, the TAP test adds a test about conflict handling (logical slot invalidation) relying on the wor

Re: Synchronizing slots from primary to standby

2023-06-16 Thread Amit Kapila
On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand wrote: > > Please find attached V5 (a rebase of V4 posted up-thread). > > In addition to the "rebasing" work, the TAP test adds a test about conflict > handling (logical slot invalidation) > relying on the work done in the "Minimal logical decodin

Re: Synchronizing slots from primary to standby

2023-04-17 Thread Drouvot, Bertrand
Hi, On 4/14/23 3:22 PM, Drouvot, Bertrand wrote: Now that the "Minimal logical decoding on standby" patch series (mentioned up-thread) has been committed, I think we can resume working on this one ("Synchronizing slots from primary to standby"). I'll work on a rebase and share it once done (u

Re: Synchronizing slots from primary to standby

2023-04-14 Thread Drouvot, Bertrand
Hi, On 11/15/22 10:02 AM, Drouvot, Bertrand wrote: Hi, On 2/11/22 3:26 PM, Peter Eisentraut wrote: On 10.02.22 22:47, Bruce Momjian wrote: On Tue, Feb  8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote: Which means that if e.g. the standby_slot_names GUC differs from synchronize_slot_names

Re: Synchronizing slots from primary to standby

2022-11-15 Thread Drouvot, Bertrand
Hi, On 2/11/22 3:26 PM, Peter Eisentraut wrote: On 10.02.22 22:47, Bruce Momjian wrote: On Tue, Feb  8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote: Which means that if e.g. the standby_slot_names GUC differs from synchronize_slot_names on the physical replica, the slots synchronized on t

Re: Synchronizing slots from primary to standby

2022-03-09 Thread Ashutosh Sharma
Hi, I have spent little time trying to understand the concern raised by Andres and while doing so I could think of a couple of issues which I would like to share here. Although I'm not quite sure how inline these are with the problems seen by Andres. 1) Firstly, what if we come across a situation

Re: Synchronizing slots from primary to standby

2022-02-27 Thread Bharath Rupireddy
On Thu, Feb 24, 2022 at 12:46 AM James Coleman wrote: > I've been working on adding test coverage to prove this out, but I've > encountered the problem reported in [1]. > > My assumption, but Andres please correct me if I'm wrong, that we > should see issues with the following steps (given the pri

Re: Synchronizing slots from primary to standby

2022-02-23 Thread James Coleman
On Fri, Feb 18, 2022 at 5:23 PM Andres Freund wrote: > > Hi, > > On 2022-02-11 15:28:19 +0100, Peter Eisentraut wrote: > > On 05.02.22 20:59, Andres Freund wrote: > > > On 2022-01-03 14:46:52 +0100, Peter Eisentraut wrote: > > > > From ec00dc6ab8bafefc00e9b1c78ac9348b643b8a87 Mon Sep 17 00:00:00

RE: Synchronizing slots from primary to standby

2022-02-20 Thread kato-...@fujitsu.com
Hello, This patch status is already returned with feedback. However, I've applied this patch on 27b77ecf9f and tested so report it. make installcheck-world is passed. However, when I promote the standby server and update on the new primary server, apply worker could not start logical replication

Re: Synchronizing slots from primary to standby

2022-02-18 Thread Andres Freund
Hi, On 2022-02-11 15:28:19 +0100, Peter Eisentraut wrote: > On 05.02.22 20:59, Andres Freund wrote: > > On 2022-01-03 14:46:52 +0100, Peter Eisentraut wrote: > > > From ec00dc6ab8bafefc00e9b1c78ac9348b643b8a87 Mon Sep 17 00:00:00 2001 > > > From: Peter Eisentraut > > > Date: Mon, 3 Jan 2022 14:43

Re: Synchronizing slots from primary to standby

2022-02-18 Thread James Coleman
On Fri, Feb 11, 2022 at 9:26 AM Peter Eisentraut wrote: > > The way I understand it: > > 1. This feature (probably) depends on the "Minimal logical decoding on > standbys" patch. The details there aren't totally clear (to me). That > patch had some activity lately but I don't see it in a state t

Re: Synchronizing slots from primary to standby

2022-02-18 Thread James Coleman
On Fri, Feb 11, 2022 at 9:26 AM Peter Eisentraut wrote: > > On 10.02.22 22:47, Bruce Momjian wrote: > > I would love to see this feature in PG 15. Can someone explain its > > current status? Thanks. > > The way I understand it: > ... Hi Peter, I'm starting to review this patch, and last time I

Re: Synchronizing slots from primary to standby

2022-02-11 Thread Peter Eisentraut
On 05.02.22 20:59, Andres Freund wrote: On 2022-01-03 14:46:52 +0100, Peter Eisentraut wrote: From ec00dc6ab8bafefc00e9b1c78ac9348b643b8a87 Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Mon, 3 Jan 2022 14:43:36 +0100 Subject: [PATCH v3] Synchronize logical replication slots from prima

Re: Synchronizing slots from primary to standby

2022-02-11 Thread Peter Eisentraut
On 10.02.22 22:47, Bruce Momjian wrote: On Tue, Feb 8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote: Which means that if e.g. the standby_slot_names GUC differs from synchronize_slot_names on the physical replica, the slots synchronized on the physical replica are not going to be valid. Or

Re: Synchronizing slots from primary to standby

2022-02-10 Thread Bruce Momjian
On Tue, Feb 8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote: > > Which means that if e.g. the standby_slot_names GUC differs from > > synchronize_slot_names on the physical replica, the slots synchronized on > > the > > physical replica are not going to be valid. Or if the primary drops its >

Re: Synchronizing slots from primary to standby

2022-02-08 Thread Ashutosh Sharma
On Tue, Feb 8, 2022 at 2:02 AM Andres Freund wrote: > > Hi, > > On 2022-02-07 13:38:38 +0530, Ashutosh Sharma wrote: > > Are you talking about this scenario - what if the logical replication > > slot on the publisher is dropped, but is being referenced by the > > standby where the slot is synchron

Re: Synchronizing slots from primary to standby

2022-02-07 Thread Andres Freund
Hi, On 2022-01-03 14:46:52 +0100, Peter Eisentraut wrote: > +static void > +ApplyLauncherStartSlotSync(TimestampTz *last_start_time, long *wait_time) > +{ > [...] > + > + foreach(lc, slots) > + { > + WalRecvReplicationSlotData *slot_data = lfirst(lc); > + LogicalRep

Re: Synchronizing slots from primary to standby

2022-02-07 Thread Andres Freund
Hi, On 2022-02-07 13:38:38 +0530, Ashutosh Sharma wrote: > Are you talking about this scenario - what if the logical replication > slot on the publisher is dropped, but is being referenced by the > standby where the slot is synchronized? It's a bit hard to say, because neither in this thread nor

Re: Synchronizing slots from primary to standby

2022-02-07 Thread Ashutosh Sharma
Hi Andres, Are you talking about this scenario - what if the logical replication slot on the publisher is dropped, but is being referenced by the standby where the slot is synchronized? Should the redo function for the drop replication slot have the capability to drop it on standby and its subscri

Re: Synchronizing slots from primary to standby

2022-02-05 Thread Andres Freund
Hi, On 2022-01-03 14:46:52 +0100, Peter Eisentraut wrote: > From ec00dc6ab8bafefc00e9b1c78ac9348b643b8a87 Mon Sep 17 00:00:00 2001 > From: Peter Eisentraut > Date: Mon, 3 Jan 2022 14:43:36 +0100 > Subject: [PATCH v3] Synchronize logical replication slots from primary to > standby I've just skim

Re: Synchronizing slots from primary to standby

2022-02-04 Thread Ashutosh Sharma
On Sat, Jan 22, 2022 at 4:33 AM Hsu, John wrote: > >> I might be missing something but isn’t it okay even if the new primary >> server is behind the subscribers? IOW, even if two slot's LSNs (i.e., >> restart_lsn and confirm_flush_lsn) are behind the subscriber's remote >> LSN (i.e

Re: Synchronizing slots from primary to standby

2022-01-21 Thread Hsu, John
> I might be missing something but isn’t it okay even if the new primary > server is behind the subscribers? IOW, even if two slot's LSNs (i.e., > restart_lsn and confirm_flush_lsn) are behind the subscriber's remote > LSN (i.e., pg_replication_origin.remote_lsn), the primary sends only

Re: Synchronizing slots from primary to standby

2022-01-20 Thread Masahiko Sawada
On Wed, Dec 15, 2021 at 7:13 AM Peter Eisentraut wrote: > > On 31.10.21 11:08, Peter Eisentraut wrote: > > I want to reactivate $subject. I took Petr Jelinek's patch from [0], > > rebased it, added a bit of testing. It basically works, but as > > mentioned in [0], there are various issues to wor

Re: Synchronizing slots from primary to standby

2022-01-03 Thread Peter Eisentraut
Here is an updated patch to fix some build failures. No feature changes. On 14.12.21 23:12, Peter Eisentraut wrote: On 31.10.21 11:08, Peter Eisentraut wrote: I want to reactivate $subject.  I took Petr Jelinek's patch from [0], rebased it, added a bit of testing.  It basically works, but as

Re: Synchronizing slots from primary to standby

2021-12-15 Thread Hsu, John
Hello, I started taking a brief look at the v2 patch, and it does appear to work for the basic case. Logical slot is synchronized across and I can connect to the promoted standby and stream changes afterwards. It's not clear to me what the correct behavior is when a logical slot that has been

Re: Synchronizing slots from primary to standby

2021-12-14 Thread Peter Eisentraut
On 28.11.21 07:52, Bharath Rupireddy wrote: 1) Instead of a new LIST_SLOT command, can't we use READ_REPLICATION_SLOT (slight modifications needs to be done to make it support logical replication slots and to get more information from the subscriber). I looked at that but didn't see an obvious

Re: Synchronizing slots from primary to standby

2021-12-14 Thread Peter Eisentraut
On 24.11.21 17:25, Dimitri Fontaine wrote: Is there a case to be made about doing the same thing for physical replication slots too? It has been considered. At the moment, I'm not doing it, because it would add more code and complexity and it's not that important. But it could be added in t

Re: Synchronizing slots from primary to standby

2021-12-14 Thread Peter Eisentraut
On 24.11.21 07:11, Masahiko Sawada wrote: I haven’t looked at the patch deeply but regarding 007_sync_rep.pl, the tests seem to fail since the tests rely on the order of the wal sender array on the shared memory. Since a background worker for synchronizing replication slots periodically connects

Re: Synchronizing slots from primary to standby

2021-12-14 Thread Peter Eisentraut
On 31.10.21 11:08, Peter Eisentraut wrote: I want to reactivate $subject.  I took Petr Jelinek's patch from [0], rebased it, added a bit of testing.  It basically works, but as mentioned in [0], there are various issues to work out. The idea is that the standby runs a background worker to peri

Re: Synchronizing slots from primary to standby

2021-11-29 Thread Bharath Rupireddy
On Mon, Nov 29, 2021 at 1:10 PM Dilip Kumar wrote: > > On Mon, Nov 29, 2021 at 12:19 PM Bharath Rupireddy > wrote: > > > > On Mon, Nov 29, 2021 at 11:14 AM Dilip Kumar wrote: > > > > > > On Mon, Nov 29, 2021 at 9:40 AM Bharath Rupireddy > > > wrote: > > > > > > > > On Mon, Nov 29, 2021 at 1:48

Re: Synchronizing slots from primary to standby

2021-11-28 Thread Dilip Kumar
On Mon, Nov 29, 2021 at 12:19 PM Bharath Rupireddy wrote: > > On Mon, Nov 29, 2021 at 11:14 AM Dilip Kumar wrote: > > > > On Mon, Nov 29, 2021 at 9:40 AM Bharath Rupireddy > > wrote: > > > > > > On Mon, Nov 29, 2021 at 1:48 AM SATYANARAYANA NARLAPURAM > > > wrote: > > > > > > > >> 3) Instead of

Re: Synchronizing slots from primary to standby

2021-11-28 Thread Bharath Rupireddy
On Mon, Nov 29, 2021 at 11:14 AM Dilip Kumar wrote: > > On Mon, Nov 29, 2021 at 9:40 AM Bharath Rupireddy > wrote: > > > > On Mon, Nov 29, 2021 at 1:48 AM SATYANARAYANA NARLAPURAM > > wrote: > > > > > >> 3) Instead of the subscriber pulling the slot info, why can't the > > >> publisher (via the

Re: Synchronizing slots from primary to standby

2021-11-28 Thread Dilip Kumar
On Mon, Nov 29, 2021 at 9:40 AM Bharath Rupireddy wrote: > > On Mon, Nov 29, 2021 at 1:48 AM SATYANARAYANA NARLAPURAM > wrote: > > > >> 3) Instead of the subscriber pulling the slot info, why can't the > >> publisher (via the walsender or a new bg worker maybe?) push the > >> latest slot info? I'

Re: Synchronizing slots from primary to standby

2021-11-28 Thread Bharath Rupireddy
On Mon, Nov 29, 2021 at 1:48 AM SATYANARAYANA NARLAPURAM wrote: > >> 3) Instead of the subscriber pulling the slot info, why can't the >> publisher (via the walsender or a new bg worker maybe?) push the >> latest slot info? I'm not sure we want to add more functionality to >> the walsender, if yes

Re: Synchronizing slots from primary to standby

2021-11-28 Thread SATYANARAYANA NARLAPURAM
> 3) Instead of the subscriber pulling the slot info, why can't the > publisher (via the walsender or a new bg worker maybe?) push the > latest slot info? I'm not sure we want to add more functionality to > the walsender, if yes, isn't it going to be much simpler? > Standby pulling the information

Re: Synchronizing slots from primary to standby

2021-11-27 Thread Bharath Rupireddy
On Sun, Oct 31, 2021 at 3:38 PM Peter Eisentraut wrote: > > I want to reactivate $subject. I took Petr Jelinek's patch from [0], > rebased it, added a bit of testing. It basically works, but as > mentioned in [0], there are various issues to work out. > > The idea is that the standby runs a back

Re: Synchronizing slots from primary to standby

2021-11-24 Thread Dimitri Fontaine
Hi all, Peter Eisentraut writes: > I want to reactivate $subject. I took Petr Jelinek's patch from [0], > rebased it, added a bit of testing. It basically works, but as > mentioned in [0], there are various issues to work out. Thanks for working on that topic, I believe it's an important part

Re: Synchronizing slots from primary to standby

2021-11-23 Thread Masahiko Sawada
On Sun, Oct 31, 2021 at 7:08 PM Peter Eisentraut wrote: > > I want to reactivate $subject. I took Petr Jelinek's patch from [0], > rebased it, added a bit of testing. It basically works, but as > mentioned in [0], there are various issues to work out. Thank you for working on this feature! > >

Re: Synchronizing slots from primary to standby

2019-07-08 Thread Thomas Munro
On Mon, Dec 31, 2018 at 10:23 AM Petr Jelinek wrote: > As Andres has mentioned over at minimal decoding on standby thread [1], > that functionality can be used to add simple worker which periodically > synchronizes the slot state from the primary to a standby. > > Attached patch is rough implement

<    4   5   6   7   8   9