Re: Synchronizing slots from primary to standby

2023-11-23 Thread Drouvot, Bertrand
Hi, On 11/23/23 6:13 AM, Amit Kapila wrote: On Tue, Nov 21, 2023 at 4:35 PM Drouvot, Bertrand wrote: On 11/21/23 10:32 AM, shveta malik wrote: On Tue, Nov 21, 2023 at 2:02 PM shveta malik wrote: v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, rebased the patches. PFA

Re: Synchronizing slots from primary to standby

2023-11-23 Thread Amit Kapila
On Wed, Nov 22, 2023 at 10:02 AM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, November 21, 2023 5:33 PM shveta malik > wrote: > > > > > > v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, rebased the > > patches. PFA v37_2 patches. > > Thanks for updating the patches. > > I'd like

Re: Synchronizing slots from primary to standby

2023-11-23 Thread Ajin Cherian
On Tue, Nov 21, 2023 at 8:32 PM shveta malik wrote: > > v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, > rebased the patches. PFA v37_2 patches. Thanks for the patch. Some comments: subscriptioncmds.c: CreateSubscription() and tablesync.c: process_syncing_tables_for_apply()

Re: Synchronizing slots from primary to standby

2023-11-22 Thread Amit Kapila
On Tue, Nov 21, 2023 at 4:35 PM Drouvot, Bertrand wrote: > > On 11/21/23 10:32 AM, shveta malik wrote: > > On Tue, Nov 21, 2023 at 2:02 PM shveta malik wrote: > >> > > > v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, > > rebased the patches. PFA v37_2 patches. > > Thanks! > >

Re: Synchronizing slots from primary to standby

2023-11-21 Thread Peter Smith
In addition to my recent v35-0001 comment not yet addressed [1], here are some review comments for patch v37-0001. == src/backend/replication/walsender.c 1. PhysicalWakeupLogicalWalSnd +/* + * Wake up logical walsenders with failover-enabled slots if the physical slot + * of the current

RE: Synchronizing slots from primary to standby

2023-11-21 Thread Zhijie Hou (Fujitsu)
On Tuesday, November 21, 2023 5:33 PM shveta malik wrote: > > > v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, rebased the > patches. PFA v37_2 patches. Thanks for updating the patches. I'd like to discuss one issue related to the correct handling of failover flag when

Re: Synchronizing slots from primary to standby

2023-11-21 Thread Drouvot, Bertrand
Hi, On 11/21/23 10:32 AM, shveta malik wrote: On Tue, Nov 21, 2023 at 2:02 PM shveta malik wrote: v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, rebased the patches. PFA v37_2 patches. Thanks! Regarding the promotion flow: If the primary is available and reachable I

Re: Synchronizing slots from primary to standby

2023-11-21 Thread shveta malik
On Tue, Nov 21, 2023 at 1:13 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/21/23 6:16 AM, Amit Kapila wrote: > > On Mon, Nov 20, 2023 at 6:51 PM Drouvot, Bertrand > > wrote: > >> As far the 'i' state here, from what I see, it is currently useful for: > >> > >> 1. Cascading standby to not sync

Re: Synchronizing slots from primary to standby

2023-11-21 Thread shveta malik
On Tue, Nov 21, 2023 at 10:02 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, November 17, 2023 7:39 PM Amit Kapila > wrote: > > > > On Thu, Nov 16, 2023 at 5:34 PM shveta malik > > wrote: > > > > > > PFA v35. > > > > > > > Review v35-0002* > > == > > Thanks for the comments. > > >

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Drouvot, Bertrand
Hi, On 11/21/23 6:16 AM, Amit Kapila wrote: On Mon, Nov 20, 2023 at 6:51 PM Drouvot, Bertrand wrote: As far the 'i' state here, from what I see, it is currently useful for: 1. Cascading standby to not sync slots with state = 'i' from the first standby. 2. Easily report Slots that did not

Re: Synchronizing slots from primary to standby

2023-11-20 Thread shveta malik
On Tue, Nov 21, 2023 at 10:02 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, November 17, 2023 7:39 PM Amit Kapila > wrote: > > > > On Thu, Nov 16, 2023 at 5:34 PM shveta malik > > wrote: > > > > > > PFA v35. > > > > > > > Review v35-0002* > > == > > Thanks for the comments. > > >

Re: Synchronizing slots from primary to standby

2023-11-20 Thread shveta malik
On Tue, Nov 21, 2023 at 10:01 AM Zhijie Hou (Fujitsu) wrote: > > On Saturday, November 18, 2023 6:46 PM Amit Kapila > wrote: > > > > On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > > wrote: > > > > > > On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: > > > > On Tuesday, November 14, 2023

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Peter Smith
Here are some review comments for the patch v35-0001. == 0. GENERAL documentation I felt that the documentation gave details of the individual changes (e.g. GUC 'standby_slot_names' and API, CREATE SUBSCRIPTION option, and pg_replication_slots 'failover' attribute etc.) but there is nothing

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 6:51 PM Drouvot, Bertrand wrote: > > On 11/20/23 11:59 AM, Amit Kapila wrote: > > On Mon, Nov 20, 2023 at 3:17 PM Drouvot, Bertrand > > wrote: > >> > >> On 11/18/23 11:45 AM, Amit Kapila wrote: > >>> On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > >>> wrote: > >

RE: Synchronizing slots from primary to standby

2023-11-20 Thread Zhijie Hou (Fujitsu)
On Friday, November 17, 2023 7:39 PM Amit Kapila wrote: > > On Thu, Nov 16, 2023 at 5:34 PM shveta malik > wrote: > > > > PFA v35. > > > > Review v35-0002* > == Thanks for the comments. > 1. > As quoted in the commit message, > > > If a logical slot is invalidated on the

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 4:28 PM Amit Kapila wrote: > > 9. > +synchronize_one_slot(WalReceiverConn *wrconn, RemoteSlot *remote_slot, > + bool *slot_updated) > { > ... > + else > + { > + TransactionId xmin_horizon = InvalidTransactionId; > + ReplicationSlot *slot; > + > +

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Drouvot, Bertrand
On 11/20/23 11:59 AM, Amit Kapila wrote: On Mon, Nov 20, 2023 at 3:17 PM Drouvot, Bertrand wrote: On 11/18/23 11:45 AM, Amit Kapila wrote: On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand wrote: On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: On Tuesday, November 14, 2023 10:27 PM

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 3:17 PM Drouvot, Bertrand wrote: > > On 11/18/23 11:45 AM, Amit Kapila wrote: > > On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > > wrote: > >> > >> On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: > >>> On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand > >>>

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Sat, Nov 18, 2023 at 4:15 PM Amit Kapila wrote: > > On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > wrote: > > More Review for v35-0002* > > More review of v35-0002* 1. +/* + * Helper function to check if local_slot is present in

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Drouvot, Bertrand
Hi, On 11/18/23 11:45 AM, Amit Kapila wrote: On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand wrote: On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand wrote: I feel the WaitForWALToBecomeAvailable may not be the best place to

Re: Synchronizing slots from primary to standby

2023-11-18 Thread Amit Kapila
On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand wrote: > > On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: > > On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand > > wrote: > > > > I feel the WaitForWALToBecomeAvailable may not be the best place to shutdown > > slotsync worker and drop

Re: Synchronizing slots from primary to standby

2023-11-17 Thread Drouvot, Bertrand
Hi, On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand wrote: On 11/13/23 2:57 PM, Zhijie Hou (Fujitsu) wrote: On Friday, November 10, 2023 4:16 PM Drouvot, Bertrand wrote: Yeah good point, agree to just error out in all the case

Re: Synchronizing slots from primary to standby

2023-11-17 Thread Amit Kapila
On Thu, Nov 16, 2023 at 5:34 PM shveta malik wrote: > > PFA v35. > Review v35-0002* == 1. As quoted in the commit message, > If a logical slot is invalidated on the primary, slot on the standby is also invalidated. If a logical slot on the primary is valid but is invalidated on the

Re: Synchronizing slots from primary to standby

2023-11-17 Thread Drouvot, Bertrand
Hi, On 11/16/23 1:03 PM, shveta malik wrote: On Thu, Nov 16, 2023 at 3:43 PM Amit Kapila wrote: PFA v35. It has below changes: Thanks for the update! 6) shutdown the slotsync worker on promotion. + /* +* Shutdown the slot sync workers to prevent potential conflicts between +*

Re: Synchronizing slots from primary to standby

2023-11-16 Thread Ajin Cherian
On Tue, Nov 14, 2023 at 12:57 AM Zhijie Hou (Fujitsu) wrote: > > 2) Raise error if user slot with same name already exists on standby. "ERROR: not synchronizing slot test; it is a user created slot" I just tested this using v35 and to me the error message when this happens is not very good.

RE: Synchronizing slots from primary to standby

2023-11-16 Thread Zhijie Hou (Fujitsu)
On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand wrote: > On 11/13/23 2:57 PM, Zhijie Hou (Fujitsu) wrote: > > On Friday, November 10, 2023 4:16 PM Drouvot, Bertrand > wrote: > >> Yeah good point, agree to just error out in all the case then (if we > >> discard the sync_ reserved

Re: Synchronizing slots from primary to standby

2023-11-16 Thread Amit Kapila
On Wed, Nov 15, 2023 at 5:21 PM shveta malik wrote: > > PFA v34. > Few comments on v34-0001* === 1. + char buf[100]; + + buf[0] = '\0'; + + if (MySubscription->twophasestate == LOGICALREP_TWOPHASE_STATE_PENDING) + strcat(buf, "twophase"); + if (MySubscription->failoverstate

Re: Synchronizing slots from primary to standby

2023-11-15 Thread shveta malik
On Tue, Nov 14, 2023 at 7:56 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/13/23 2:57 PM, Zhijie Hou (Fujitsu) wrote: > > On Friday, November 10, 2023 4:16 PM Drouvot, Bertrand > > wrote: > >> Yeah good point, agree to just error out in all the case then (if we > >> discard the > >> sync_

Re: Synchronizing slots from primary to standby

2023-11-14 Thread Drouvot, Bertrand
Hi, On 11/13/23 2:57 PM, Zhijie Hou (Fujitsu) wrote: On Friday, November 10, 2023 4:16 PM Drouvot, Bertrand wrote: Yeah good point, agree to just error out in all the case then (if we discard the sync_ reserved wording proposal, which seems to be the case as probably not worth the extra

Re: Synchronizing slots from primary to standby

2023-11-13 Thread Drouvot, Bertrand
Hi, On 11/13/23 5:24 AM, shveta malik wrote: On Thu, Nov 9, 2023 at 8:56 AM Amit Kapila wrote: Apart from the above, I would like to discuss the slot sync work distribution strategy of this patch. The current implementation as explained in the commit message [1] works well if the slots

Re: Synchronizing slots from primary to standby

2023-11-13 Thread Ajin Cherian
On Mon, Nov 13, 2023 at 5:38 PM shveta malik wrote: > Okay. Thanks for testing Ajin. I think it needs a fix wherein we set > the local-slot's invalidation status (provided it is not invalidated > already) from the remote slot before this check itself. And if the > slot is invalidated locally

Re: Synchronizing slots from primary to standby

2023-11-12 Thread shveta malik
On Mon, Nov 13, 2023 at 11:02 AM Ajin Cherian wrote: > > On Thu, Nov 9, 2023 at 9:54 PM shveta malik wrote: > > > > PFA v32 patches which has below changes: > Testing with this patch, I see that if the failover enabled slot is > invalidated on the primary, then the corresponding synced slot is

Re: Synchronizing slots from primary to standby

2023-11-12 Thread Ajin Cherian
On Thu, Nov 9, 2023 at 9:54 PM shveta malik wrote: > > PFA v32 patches which has below changes: Testing with this patch, I see that if the failover enabled slot is invalidated on the primary, then the corresponding synced slot is not invalidated on the standby. Instead, I see that it continuously

Re: Synchronizing slots from primary to standby

2023-11-12 Thread shveta malik
On Thu, Nov 9, 2023 at 8:56 AM Amit Kapila wrote: > > On Thu, Nov 9, 2023 at 8:11 AM Amit Kapila wrote: > > > > On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand > > wrote: > > > > > > > Unrelated to above, if there is a user slot on standby with the same > > > > name which the slot-sync worker

Re: Synchronizing slots from primary to standby

2023-11-12 Thread shveta malik
On Mon, Nov 13, 2023 at 6:19 AM Zhijie Hou (Fujitsu) wrote: > > On Thursday, November 9, 2023 6:54 PM shveta malik > wrote: > > > > > > PFA v32 patches which has below changes: > > Thanks for updating the patch. > > Here are few comments: > > > 2. > The check for wal_level < WAL_LEVEL_LOGICAL.

RE: Synchronizing slots from primary to standby

2023-11-12 Thread Zhijie Hou (Fujitsu)
On Thursday, November 9, 2023 6:54 PM shveta malik wrote: > > > PFA v32 patches which has below changes: Thanks for updating the patch. Here are few comments: 1. Do we need to update the slot upgrade code in pg_upgrade to upgrade the slot's failover property as well ? 2. The check for

Re: Synchronizing slots from primary to standby

2023-11-10 Thread Drouvot, Bertrand
Hi, On 11/10/23 4:31 AM, shveta malik wrote: On Thu, Nov 9, 2023 at 9:15 PM Drouvot, Bertrand wrote: Yeah I think so, because there is a time window when one could "use" the slot after the promotion and before it is removed. Producing things like: " 2023-11-09 15:16:50.294 UTC [2580462] LOG:

Re: Synchronizing slots from primary to standby

2023-11-10 Thread Drouvot, Bertrand
Hi, On 11/10/23 8:55 AM, Amit Kapila wrote: On Fri, Nov 10, 2023 at 12:50 PM Drouvot, Bertrand wrote: But even if we ERROR out instead of emitting a WARNING, the user would still need to be notified/monitor such errors. I agree that then probably they will come to know earlier because the

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Amit Kapila
On Thu, Nov 9, 2023 at 9:15 PM Drouvot, Bertrand wrote: > > > You mean here? > > /* > * Check to see if promotion is requested. Note that we do > * this only after failure, so when you promote, we still > * finish replaying as much as we can from archive and > * pg_wal before failover. >

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Amit Kapila
On Fri, Nov 10, 2023 at 12:50 PM Drouvot, Bertrand wrote: > > On 11/10/23 6:41 AM, Amit Kapila wrote: > > On Thu, Nov 9, 2023 at 7:29 PM Drouvot, Bertrand > > wrote: > > > > Are you saying that we change the state of the already existing slot > > on standby? > > Yes. > > > And, such a state

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Drouvot, Bertrand
Hi, On 11/10/23 6:41 AM, Amit Kapila wrote: On Thu, Nov 9, 2023 at 7:29 PM Drouvot, Bertrand wrote: Are you saying that we change the state of the already existing slot on standby? Yes. And, such a state would indicate that we are trying to sync the slot with the same name from the

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Amit Kapila
On Thu, Nov 9, 2023 at 7:29 PM Drouvot, Bertrand wrote: > > On 11/9/23 3:41 AM, Amit Kapila wrote: > > On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand > > wrote: > >> > >>> Unrelated to above, if there is a user slot on standby with the same > >>> name which the slot-sync worker is trying to

Re: Synchronizing slots from primary to standby

2023-11-09 Thread shveta malik
On Thu, Nov 9, 2023 at 9:15 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/9/23 11:54 AM, shveta malik wrote: > > > > PFA v32 patches which has below changes: > > Thanks! > > > 7) Added warning for cases where a user-slot with the same name is > > already present which slot-sync worker is trying

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Drouvot, Bertrand
Hi, On 11/9/23 11:54 AM, shveta malik wrote: PFA v32 patches which has below changes: Thanks! 7) Added warning for cases where a user-slot with the same name is already present which slot-sync worker is trying to create. Sync for such slots is skipped. I'm seeing assertion and segfault

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Drouvot, Bertrand
Hi, On 11/9/23 3:41 AM, Amit Kapila wrote: On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand wrote: Unrelated to above, if there is a user slot on standby with the same name which the slot-sync worker is trying to create, then shall it emit a warning and skip the sync of that slot or shall

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Amit Kapila
On Thu, Nov 9, 2023 at 8:11 AM Amit Kapila wrote: > > On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand > wrote: > > > > > Unrelated to above, if there is a user slot on standby with the same > > > name which the slot-sync worker is trying to create, then shall it > > > emit a warning and skip

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Amit Kapila
On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand wrote: > > > Unrelated to above, if there is a user slot on standby with the same > > name which the slot-sync worker is trying to create, then shall it > > emit a warning and skip the sync of that slot or shall it throw an > > error? > > > > I'd

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Drouvot, Bertrand
Hi, On 11/8/23 12:50 PM, shveta malik wrote: On Wed, Nov 8, 2023 at 3:19 PM Drouvot, Bertrand wrote: Hi, On 11/8/23 9:57 AM, Amit Kapila wrote: On Wed, Nov 8, 2023 at 12:32 PM Drouvot, Bertrand wrote: Hi, On 11/8/23 4:50 AM, Amit Kapila wrote: I think if we want to follow this

Re: Synchronizing slots from primary to standby

2023-11-08 Thread shveta malik
On Wed, Nov 8, 2023 at 3:19 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/8/23 9:57 AM, Amit Kapila wrote: > > On Wed, Nov 8, 2023 at 12:32 PM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 11/8/23 4:50 AM, Amit Kapila wrote: > >> > >>> I think if we want to follow > >>> this approach

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Drouvot, Bertrand
Hi, On 11/8/23 9:57 AM, Amit Kapila wrote: On Wed, Nov 8, 2023 at 12:32 PM Drouvot, Bertrand wrote: Hi, On 11/8/23 4:50 AM, Amit Kapila wrote: I think if we want to follow this approach then we need to also monitor these slots for any change in the consecutive cycles and if we are able to

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Amit Kapila
On Wed, Nov 8, 2023 at 12:32 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/8/23 4:50 AM, Amit Kapila wrote: > > > I think if we want to follow > > this approach then we need to also monitor these slots for any change > > in the consecutive cycles and if we are able to sync them then > >

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Drouvot, Bertrand
Hi, On 11/8/23 4:50 AM, Amit Kapila wrote: On Tue, Nov 7, 2023 at 7:58 PM Drouvot, Bertrand wrote: If we think this window is too short we could: - increase it or - don't drop the slot once created (even if there is no activity on the primary during PrimaryCatchupWaitAttempt attempts) so

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Amit Kapila
On Tue, Nov 7, 2023 at 7:58 PM Drouvot, Bertrand wrote: > > On 11/7/23 11:55 AM, Amit Kapila wrote: > >>> > >>> This is not full proof solution but optimization over first one. Now > >>> in any sync-cycle, we take 2 attempts for slots-creation (if any slots > >>> are available to be created). In

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Drouvot, Bertrand
Hi, On 11/7/23 11:55 AM, Amit Kapila wrote: On Tue, Nov 7, 2023 at 3:51 PM Drouvot, Bertrand wrote: On 10/31/23 10:37 AM, shveta malik wrote: On Fri, Oct 27, 2023 at 8:43 PM Drouvot, Bertrand Agree with your test case, but in my case I was not using pub/sub. I was not clear, so when I

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Amit Kapila
On Tue, Nov 7, 2023 at 3:51 PM Drouvot, Bertrand wrote: > > On 10/31/23 10:37 AM, shveta malik wrote: > > On Fri, Oct 27, 2023 at 8:43 PM Drouvot, Bertrand > >> > >> Agree with your test case, but in my case I was not using pub/sub. > >> > >> I was not clear, so when I said: > >> > - create

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Drouvot, Bertrand
Hi, On 10/31/23 10:37 AM, shveta malik wrote: On Fri, Oct 27, 2023 at 8:43 PM Drouvot, Bertrand Agree with your test case, but in my case I was not using pub/sub. I was not clear, so when I said: - create logical_slot1 on the primary (and don't start using it) I meant don't start

Re: Synchronizing slots from primary to standby

2023-11-06 Thread Amit Kapila
On Mon, Nov 6, 2023 at 1:57 PM Amit Kapila wrote: > > On Mon, Nov 6, 2023 at 7:01 AM Zhijie Hou (Fujitsu) > wrote: > > +static void +WalSndGetStandbySlots(List **standby_slots, bool force) +{ + if (!MyReplicationSlot->data.failover) + return; + + if (standby_slot_names_list == NIL &&

Re: Synchronizing slots from primary to standby

2023-11-06 Thread Amit Kapila
On Mon, Nov 6, 2023 at 7:01 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, November 3, 2023 7:32 PM Amit Kapila > > > > 5. > > @@ -228,6 +230,28 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo > > fcinfo, bool confirm, bool bin > > NameStr(MyReplicationSlot->data.plugin), > >

Re: Synchronizing slots from primary to standby

2023-11-05 Thread shveta malik
On Mon, Nov 6, 2023 at 7:01 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, November 3, 2023 7:32 PM Amit Kapila > > > > On Thu, Nov 2, 2023 at 2:35 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Here is the new version patch set(V29) which addressed Peter > > > comments[1][2] and fixed one doc

Re: Synchronizing slots from primary to standby

2023-11-03 Thread Amit Kapila
On Thu, Nov 2, 2023 at 2:35 PM Zhijie Hou (Fujitsu) wrote: > > Here is the new version patch set(V29) which addressed Peter comments[1][2] > and > fixed one doc compile error. > Few comments: == 1. + +failover (boolean) + + + Specifies

RE: Synchronizing slots from primary to standby

2023-10-31 Thread Zhijie Hou (Fujitsu)
On Tuesday, October 31, 2023 6:45 PM Amit Kapila wrote: > > b) Currently I have restricted 'alter subscription.. refresh > > publication with copy=true' when failover=true (on a similar line of > > two-phase). The reason being, refresh with copy=true will go for > > table-sync again and since

Re: Synchronizing slots from primary to standby

2023-10-31 Thread Amit Kapila
On Fri, Oct 27, 2023 at 4:04 PM shveta malik wrote: > > On Fri, Oct 27, 2023 at 3:26 PM shveta malik wrote: > == > > Open questions regarding change for pt 1 above: > a) I think we should restrict the 'alter-sub set failover' when > failover-state is currently in 'p' (pending) state i.e.

Re: Synchronizing slots from primary to standby

2023-10-31 Thread shveta malik
On Fri, Oct 27, 2023 at 8:43 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/27/23 11:56 AM, shveta malik wrote: > > On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/25/23 5:00 AM, shveta malik wrote: > >>> On Tue, Oct 24, 2023 at 11:54 AM Drouvot,

Re: Synchronizing slots from primary to standby

2023-10-31 Thread shveta malik
On Thu, Oct 26, 2023 at 5:43 PM Amit Kapila wrote: > > On Thu, Oct 26, 2023 at 5:38 PM Drouvot, Bertrand > wrote: > > > > On 10/26/23 10:40 AM, Amit Kapila wrote: > > > On Wed, Oct 25, 2023 at 8:49 PM Drouvot, Bertrand > > > wrote: > > >> > > > > > > Good point, I think we should enhance the

Re: Synchronizing slots from primary to standby

2023-10-30 Thread Amit Kapila
On Tue, Oct 31, 2023 at 7:16 AM Hayato Kuroda (Fujitsu) wrote: > > > > 2. > > > Currently ctx->failover is set only in the pgoutput_startup(), but not > > > sure it is > > OK. > > > Can we change the parameter in CreateDecodingContext() or similar > > > functions? > > > > > > Because IIUC it

RE: Synchronizing slots from primary to standby

2023-10-30 Thread Hayato Kuroda (Fujitsu)
Dear Ajin, Thanks for your reply! > On Thu, Oct 26, 2023 at 6:08 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Dear Shveta, > > > > > PFA v25 patch set. The changes are: > > > > Thanks for making the patch! It seems that there are lots of comments, so > > I can put some high-level comments for

Re: Synchronizing slots from primary to standby

2023-10-30 Thread Ajin Cherian
On Thu, Oct 26, 2023 at 6:08 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Shveta, > > > PFA v25 patch set. The changes are: > > Thanks for making the patch! It seems that there are lots of comments, so > I can put some high-level comments for 0001. > Sorry if there are duplicated comments. > > 1. >

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Amit Kapila
On Fri, Oct 27, 2023 at 9:00 PM Drouvot, Bertrand wrote: > > On 10/27/23 10:35 AM, Amit Kapila wrote: > > On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand > > > > I think even if we provide such an API, we need to have logic to get > > the slots from the primary and create them. > > Yeah, my

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Drouvot, Bertrand
Hi, On 10/27/23 10:35 AM, Amit Kapila wrote: On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand I think even if we provide such an API, we need to have logic to get the slots from the primary and create them. Yeah, my idea was to add an API (in addition to what is already in place). Say,

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Drouvot, Bertrand
Hi, On 10/27/23 10:51 AM, shveta malik wrote: On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: I discussed this with my colleague Hou-San and we think that one possibility could be to somehow accelerate the increment of restart_lsn on primary. This can be achieved by connecting to

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Drouvot, Bertrand
Hi, On 10/27/23 11:56 AM, shveta malik wrote: On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: Hi, On 10/25/23 5:00 AM, shveta malik wrote: On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand wrote: Hi, On 10/23/23 2:56 PM, shveta malik wrote: On Mon, Oct 23, 2023 at 5:52 PM

Re: Synchronizing slots from primary to standby

2023-10-27 Thread shveta malik
On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/25/23 5:00 AM, shveta malik wrote: > > On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/23/23 2:56 PM, shveta malik wrote: > >>> On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand

Re: Synchronizing slots from primary to standby

2023-10-27 Thread shveta malik
On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/25/23 5:00 AM, shveta malik wrote: > > On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/23/23 2:56 PM, shveta malik wrote: > >>> On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Amit Kapila
On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: > > On 10/25/23 5:00 AM, shveta malik wrote: > > On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/23/23 2:56 PM, shveta malik wrote: > >>> On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand > >>>

Re: Synchronizing slots from primary to standby

2023-10-26 Thread Amit Kapila
On Thu, Oct 26, 2023 at 12:38 PM Hayato Kuroda (Fujitsu) wrote: > > > PFA v25 patch set. The changes are: > > Thanks for making the patch! It seems that there are lots of comments, so > I can put some high-level comments for 0001. > Sorry if there are duplicated comments. > > 1. > The patch

Re: Synchronizing slots from primary to standby

2023-10-26 Thread Amit Kapila
On Thu, Oct 26, 2023 at 5:38 PM Drouvot, Bertrand wrote: > > On 10/26/23 10:40 AM, Amit Kapila wrote: > > On Wed, Oct 25, 2023 at 8:49 PM Drouvot, Bertrand > > wrote: > >> > > > > Good point, I think we should enhance the WalSndWait() logic to > > address this case. > > Agree. I think it would

Re: Synchronizing slots from primary to standby

2023-10-26 Thread Drouvot, Bertrand
Hi, On 10/26/23 10:40 AM, Amit Kapila wrote: On Wed, Oct 25, 2023 at 8:49 PM Drouvot, Bertrand wrote: Good point, I think we should enhance the WalSndWait() logic to address this case. Agree. I think it would need to take care of the new CV and probably provide a way for the caller to

Re: Synchronizing slots from primary to standby

2023-10-26 Thread Amit Kapila
On Wed, Oct 25, 2023 at 8:49 PM Drouvot, Bertrand wrote: > > On 10/9/23 12:30 PM, shveta malik wrote: > > PFA v22 patch-set. It has below changes: > > > > patch 001: > > 1) Now physical walsender wakes up logical walsender(s) by using a new > > CV as suggested in [1] > > Thanks! > > I think that

RE: Synchronizing slots from primary to standby

2023-10-26 Thread Hayato Kuroda (Fujitsu)
Dear Shveta, > PFA v25 patch set. The changes are: Thanks for making the patch! It seems that there are lots of comments, so I can put some high-level comments for 0001. Sorry if there are duplicated comments. 1. The patch seemed not to consider the case that failover option between

Re: Synchronizing slots from primary to standby

2023-10-25 Thread Drouvot, Bertrand
Hi, On 10/9/23 12:30 PM, shveta malik wrote: PFA v22 patch-set. It has below changes: patch 001: 1) Now physical walsender wakes up logical walsender(s) by using a new CV as suggested in [1] Thanks! I think that works fine as long as the standby is up and running and catching up. The

Re: Synchronizing slots from primary to standby

2023-10-25 Thread Drouvot, Bertrand
Hi, On 10/25/23 6:57 AM, Amit Kapila wrote: On Tue, Oct 24, 2023 at 3:35 PM Drouvot, Bertrand wrote: On 10/24/23 7:44 AM, Ajin Cherian wrote: On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand wrote: 2) When we create a subscription, another slot is created during the subscription

Re: Synchronizing slots from primary to standby

2023-10-25 Thread Drouvot, Bertrand
Hi, On 10/25/23 5:00 AM, shveta malik wrote: On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand wrote: Hi, On 10/23/23 2:56 PM, shveta malik wrote: On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand wrote: We are waiting for DEFAULT_NAPTIME_PER_CYCLE (3 minutes) before checking if

Re: Synchronizing slots from primary to standby

2023-10-24 Thread Amit Kapila
On Tue, Oct 24, 2023 at 3:35 PM Drouvot, Bertrand wrote: > > On 10/24/23 7:44 AM, Ajin Cherian wrote: > > On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand > > wrote: > > 2) When we create a subscription, another slot is created during the > subscription synchronization, namely > like

Re: Synchronizing slots from primary to standby

2023-10-24 Thread shveta malik
On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand wrote: > > Hi, > > On 10/23/23 2:56 PM, shveta malik wrote: > > On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand > > wrote: > > >> We are waiting for DEFAULT_NAPTIME_PER_CYCLE (3 minutes) before checking > >> if there > >> is new synced

Re: Synchronizing slots from primary to standby

2023-10-24 Thread shveta malik
On Tue, Oct 24, 2023 at 3:35 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/24/23 7:44 AM, Ajin Cherian wrote: > > On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand > > wrote: > >> > >> @@ -602,6 +602,9 @@ CreateDecodingContext(XLogRecPtr start_lsn, > >>

Re: Synchronizing slots from primary to standby

2023-10-24 Thread Drouvot, Bertrand
Hi, On 10/24/23 7:44 AM, Ajin Cherian wrote: On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand wrote: @@ -602,6 +602,9 @@ CreateDecodingContext(XLogRecPtr start_lsn, SnapBuildSetTwoPhaseAt(ctx->snapshot_builder, start_lsn); } + /* set failover in the slot, as requested */

Re: Synchronizing slots from primary to standby

2023-10-24 Thread Peter Smith
Here are some review comments for patch v25-0002 (additional to v25-0002 review comments [1]) == src/backend/catalog/system_views.sql 1. @@ -1003,7 +1003,8 @@ CREATE VIEW pg_replication_slots AS L.safe_wal_size, L.two_phase, L.conflicting, -

Re: Synchronizing slots from primary to standby

2023-10-24 Thread Peter Smith
Here are some review comments for v24-0001 == 1. GENERAL - failover slots terminology There is inconsistent terminology, such as below. Try to use the same wording everywhere. - failover logical slots - failover slots - logical failover slots - logical replication failover slots - etc.

Re: Synchronizing slots from primary to standby

2023-10-24 Thread Drouvot, Bertrand
Hi, On 10/23/23 2:56 PM, shveta malik wrote: On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand wrote: We are waiting for DEFAULT_NAPTIME_PER_CYCLE (3 minutes) before checking if there is new synced slot(s) to be created on the standby. Do we want to keep this behavior for V1? I think

Re: Synchronizing slots from primary to standby

2023-10-23 Thread Ajin Cherian
On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/20/23 5:27 AM, shveta malik wrote: > > On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: > > > > PFA v25 patch set. The changes are: > > > > 1) 'enable_failover' is changed to 'failover' > > 2) Alter subscription

RE: Synchronizing slots from primary to standby

2023-10-23 Thread Zhijie Hou (Fujitsu)
On Friday, October 20, 2023 11:27 AM shveta malik wrote: > > The changes are mostly in patch001 and a very few in patch002. > > Thank You Ajin for working on alter-subscription changes and adding more > TAP-tests for 'failover' Thanks for updating the patch. Here are few things I noticed when

Re: Synchronizing slots from primary to standby

2023-10-23 Thread shveta malik
On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/20/23 5:27 AM, shveta malik wrote: > > On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: > > > > PFA v25 patch set. The changes are: > > > > 1) 'enable_failover' is changed to 'failover' > > 2) Alter subscription changes

Re: Synchronizing slots from primary to standby

2023-10-23 Thread Drouvot, Bertrand
Hi, On 10/20/23 5:27 AM, shveta malik wrote: On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: PFA v25 patch set. The changes are: 1) 'enable_failover' is changed to 'failover' 2) Alter subscription changes to support 'failover' 3) Fixes a bug in patch001 wherein any change in

Re: Synchronizing slots from primary to standby

2023-10-23 Thread Drouvot, Bertrand
Hi, On 10/18/23 6:43 AM, shveta malik wrote: On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand +static void +ProcessRepliesAndTimeOut(void) +{ + CHECK_FOR_INTERRUPTS(); + + /* Process any requests or signals received recently */ + if (ConfigReloadPending) + { +

Re: Synchronizing slots from primary to standby

2023-10-19 Thread shveta malik
On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/13/23 10:35 AM, shveta malik wrote: > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik wrote: > >> > > > > PFA v24 patch set which has below changes: > > > > 1) 'enable_failover' displayed in pg_replication_slots. > > 2)

Re: Synchronizing slots from primary to standby

2023-10-19 Thread shveta malik
On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: > > On Tue, Oct 17, 2023 at 2:01 PM shveta malik wrote: > > > > On Tue, Oct 17, 2023 at 12:44 PM Peter Smith wrote: > > > > > > FYI - the latest patch failed to apply. > > > > > > [postgres@CentOS7-x64 oss_postgres_misc]$ git apply > > >

Re: Synchronizing slots from primary to standby

2023-10-18 Thread Amit Kapila
On Tue, Oct 17, 2023 at 2:01 PM shveta malik wrote: > > On Tue, Oct 17, 2023 at 12:44 PM Peter Smith wrote: > > > > FYI - the latest patch failed to apply. > > > > [postgres@CentOS7-x64 oss_postgres_misc]$ git apply > >

Re: Synchronizing slots from primary to standby

2023-10-18 Thread shveta malik
On Wed, Oct 18, 2023 at 10:20 AM Amit Kapila wrote: > > On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand > wrote: > > > > On 10/13/23 10:35 AM, shveta malik wrote: > > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik > > > wrote: > > >> > > > > Code: > > > > + True if this logical slot is

Re: Synchronizing slots from primary to standby

2023-10-17 Thread Amit Kapila
On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand wrote: > > On 10/13/23 10:35 AM, shveta malik wrote: > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik wrote: > >> > > Code: > > + True if this logical slot is enabled to be synced to the physical > standbys > + so that logical

Re: Synchronizing slots from primary to standby

2023-10-17 Thread shveta malik
On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/13/23 10:35 AM, shveta malik wrote: > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik wrote: > >> > > > > PFA v24 patch set which has below changes: > > > > 1) 'enable_failover' displayed in pg_replication_slots. > > 2)

<    2   3   4   5   6   7   8   9   >