Re: Synchronizing slots from primary to standby

2023-11-23 Thread Drouvot, Bertrand
Hi, On 11/23/23 6:13 AM, Amit Kapila wrote: On Tue, Nov 21, 2023 at 4:35 PM Drouvot, Bertrand wrote: On 11/21/23 10:32 AM, shveta malik wrote: On Tue, Nov 21, 2023 at 2:02 PM shveta malik wrote: v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, rebased the patches. PFA v

Re: Synchronizing slots from primary to standby

2023-11-23 Thread Amit Kapila
On Wed, Nov 22, 2023 at 10:02 AM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, November 21, 2023 5:33 PM shveta malik > wrote: > > > > > > v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, rebased the > > patches. PFA v37_2 patches. > > Thanks for updating the patches. > > I'd like

Re: Synchronizing slots from primary to standby

2023-11-23 Thread Ajin Cherian
On Tue, Nov 21, 2023 at 8:32 PM shveta malik wrote: > > v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, > rebased the patches. PFA v37_2 patches. Thanks for the patch. Some comments: subscriptioncmds.c: CreateSubscription() and tablesync.c: process_syncing_tables_for_apply()

Re: Synchronizing slots from primary to standby

2023-11-22 Thread Amit Kapila
On Tue, Nov 21, 2023 at 4:35 PM Drouvot, Bertrand wrote: > > On 11/21/23 10:32 AM, shveta malik wrote: > > On Tue, Nov 21, 2023 at 2:02 PM shveta malik wrote: > >> > > > v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, > > rebased the patches. PFA v37_2 patches. > > Thanks! > > Re

Re: Synchronizing slots from primary to standby

2023-11-21 Thread Peter Smith
In addition to my recent v35-0001 comment not yet addressed [1], here are some review comments for patch v37-0001. == src/backend/replication/walsender.c 1. PhysicalWakeupLogicalWalSnd +/* + * Wake up logical walsenders with failover-enabled slots if the physical slot + * of the current walse

RE: Synchronizing slots from primary to standby

2023-11-21 Thread Zhijie Hou (Fujitsu)
On Tuesday, November 21, 2023 5:33 PM shveta malik wrote: > > > v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, rebased the > patches. PFA v37_2 patches. Thanks for updating the patches. I'd like to discuss one issue related to the correct handling of failover flag when execu

Re: Synchronizing slots from primary to standby

2023-11-21 Thread Drouvot, Bertrand
Hi, On 11/21/23 10:32 AM, shveta malik wrote: On Tue, Nov 21, 2023 at 2:02 PM shveta malik wrote: v37 fails to apply to HEAD due to a recent commit e83aa9f92fdd, rebased the patches. PFA v37_2 patches. Thanks! Regarding the promotion flow: If the primary is available and reachable I do

Re: Synchronizing slots from primary to standby

2023-11-21 Thread shveta malik
On Tue, Nov 21, 2023 at 1:13 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/21/23 6:16 AM, Amit Kapila wrote: > > On Mon, Nov 20, 2023 at 6:51 PM Drouvot, Bertrand > > wrote: > >> As far the 'i' state here, from what I see, it is currently useful for: > >> > >> 1. Cascading standby to not sync slo

Re: Synchronizing slots from primary to standby

2023-11-21 Thread shveta malik
On Tue, Nov 21, 2023 at 10:02 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, November 17, 2023 7:39 PM Amit Kapila > wrote: > > > > On Thu, Nov 16, 2023 at 5:34 PM shveta malik > > wrote: > > > > > > PFA v35. > > > > > > > Review v35-0002* > > == > > Thanks for the comments. > > > 1.

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Drouvot, Bertrand
Hi, On 11/21/23 6:16 AM, Amit Kapila wrote: On Mon, Nov 20, 2023 at 6:51 PM Drouvot, Bertrand wrote: As far the 'i' state here, from what I see, it is currently useful for: 1. Cascading standby to not sync slots with state = 'i' from the first standby. 2. Easily report Slots that did not catc

Re: Synchronizing slots from primary to standby

2023-11-20 Thread shveta malik
On Tue, Nov 21, 2023 at 10:02 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, November 17, 2023 7:39 PM Amit Kapila > wrote: > > > > On Thu, Nov 16, 2023 at 5:34 PM shveta malik > > wrote: > > > > > > PFA v35. > > > > > > > Review v35-0002* > > == > > Thanks for the comments. > > > 1.

Re: Synchronizing slots from primary to standby

2023-11-20 Thread shveta malik
On Tue, Nov 21, 2023 at 10:01 AM Zhijie Hou (Fujitsu) wrote: > > On Saturday, November 18, 2023 6:46 PM Amit Kapila > wrote: > > > > On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > > wrote: > > > > > > On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: > > > > On Tuesday, November 14, 2023 10

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Peter Smith
Here are some review comments for the patch v35-0001. == 0. GENERAL documentation I felt that the documentation gave details of the individual changes (e.g. GUC 'standby_slot_names' and API, CREATE SUBSCRIPTION option, and pg_replication_slots 'failover' attribute etc.) but there is nothing t

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 6:51 PM Drouvot, Bertrand wrote: > > On 11/20/23 11:59 AM, Amit Kapila wrote: > > On Mon, Nov 20, 2023 at 3:17 PM Drouvot, Bertrand > > wrote: > >> > >> On 11/18/23 11:45 AM, Amit Kapila wrote: > >>> On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > >>> wrote: > >

RE: Synchronizing slots from primary to standby

2023-11-20 Thread Zhijie Hou (Fujitsu)
On Friday, November 17, 2023 7:39 PM Amit Kapila wrote: > > On Thu, Nov 16, 2023 at 5:34 PM shveta malik > wrote: > > > > PFA v35. > > > > Review v35-0002* > == Thanks for the comments. > 1. > As quoted in the commit message, > > > If a logical slot is invalidated on the primary,

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 4:28 PM Amit Kapila wrote: > > 9. > +synchronize_one_slot(WalReceiverConn *wrconn, RemoteSlot *remote_slot, > + bool *slot_updated) > { > ... > + else > + { > + TransactionId xmin_horizon = InvalidTransactionId; > + ReplicationSlot *slot; > + > + ReplicationSlotCreate(remot

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Drouvot, Bertrand
On 11/20/23 11:59 AM, Amit Kapila wrote: On Mon, Nov 20, 2023 at 3:17 PM Drouvot, Bertrand wrote: On 11/18/23 11:45 AM, Amit Kapila wrote: On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand wrote: On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: On Tuesday, November 14, 2023 10:27 PM Dr

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 3:17 PM Drouvot, Bertrand wrote: > > On 11/18/23 11:45 AM, Amit Kapila wrote: > > On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > > wrote: > >> > >> On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: > >>> On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand > >>>

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Sat, Nov 18, 2023 at 4:15 PM Amit Kapila wrote: > > On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > wrote: > > More Review for v35-0002* > > More review of v35-0002* 1. +/* + * Helper function to check if local_slot is present in remote_sl

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Drouvot, Bertrand
Hi, On 11/18/23 11:45 AM, Amit Kapila wrote: On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand wrote: On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand wrote: I feel the WaitForWALToBecomeAvailable may not be the best place to shutdo

Re: Synchronizing slots from primary to standby

2023-11-18 Thread Amit Kapila
On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand wrote: > > On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: > > On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand > > wrote: > > > > I feel the WaitForWALToBecomeAvailable may not be the best place to shutdown > > slotsync worker and drop

Re: Synchronizing slots from primary to standby

2023-11-17 Thread Drouvot, Bertrand
Hi, On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand wrote: On 11/13/23 2:57 PM, Zhijie Hou (Fujitsu) wrote: On Friday, November 10, 2023 4:16 PM Drouvot, Bertrand wrote: Yeah good point, agree to just error out in all the case then

Re: Synchronizing slots from primary to standby

2023-11-17 Thread Amit Kapila
On Thu, Nov 16, 2023 at 5:34 PM shveta malik wrote: > > PFA v35. > Review v35-0002* == 1. As quoted in the commit message, > If a logical slot is invalidated on the primary, slot on the standby is also invalidated. If a logical slot on the primary is valid but is invalidated on the st

Re: Synchronizing slots from primary to standby

2023-11-17 Thread Drouvot, Bertrand
Hi, On 11/16/23 1:03 PM, shveta malik wrote: On Thu, Nov 16, 2023 at 3:43 PM Amit Kapila wrote: PFA v35. It has below changes: Thanks for the update! 6) shutdown the slotsync worker on promotion. + /* +* Shutdown the slot sync workers to prevent potential conflicts between +* u

Re: Synchronizing slots from primary to standby

2023-11-16 Thread Ajin Cherian
On Tue, Nov 14, 2023 at 12:57 AM Zhijie Hou (Fujitsu) wrote: > > 2) Raise error if user slot with same name already exists on standby. "ERROR: not synchronizing slot test; it is a user created slot" I just tested this using v35 and to me the error message when this happens is not very good. Neit

RE: Synchronizing slots from primary to standby

2023-11-16 Thread Zhijie Hou (Fujitsu)
On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand wrote: > On 11/13/23 2:57 PM, Zhijie Hou (Fujitsu) wrote: > > On Friday, November 10, 2023 4:16 PM Drouvot, Bertrand > wrote: > >> Yeah good point, agree to just error out in all the case then (if we > >> discard the sync_ reserved wording

Re: Synchronizing slots from primary to standby

2023-11-16 Thread Amit Kapila
On Wed, Nov 15, 2023 at 5:21 PM shveta malik wrote: > > PFA v34. > Few comments on v34-0001* === 1. + char buf[100]; + + buf[0] = '\0'; + + if (MySubscription->twophasestate == LOGICALREP_TWOPHASE_STATE_PENDING) + strcat(buf, "twophase"); + if (MySubscription->failoverstate ==

Re: Synchronizing slots from primary to standby

2023-11-15 Thread shveta malik
On Tue, Nov 14, 2023 at 7:56 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/13/23 2:57 PM, Zhijie Hou (Fujitsu) wrote: > > On Friday, November 10, 2023 4:16 PM Drouvot, Bertrand > > wrote: > >> Yeah good point, agree to just error out in all the case then (if we > >> discard the > >> sync_ reser

Re: Synchronizing slots from primary to standby

2023-11-14 Thread Drouvot, Bertrand
Hi, On 11/13/23 2:57 PM, Zhijie Hou (Fujitsu) wrote: On Friday, November 10, 2023 4:16 PM Drouvot, Bertrand wrote: Yeah good point, agree to just error out in all the case then (if we discard the sync_ reserved wording proposal, which seems to be the case as probably not worth the extra work)

Re: Synchronizing slots from primary to standby

2023-11-13 Thread Drouvot, Bertrand
Hi, On 11/13/23 5:24 AM, shveta malik wrote: On Thu, Nov 9, 2023 at 8:56 AM Amit Kapila wrote: Apart from the above, I would like to discuss the slot sync work distribution strategy of this patch. The current implementation as explained in the commit message [1] works well if the slots belong

Re: Synchronizing slots from primary to standby

2023-11-13 Thread Ajin Cherian
On Mon, Nov 13, 2023 at 5:38 PM shveta malik wrote: > Okay. Thanks for testing Ajin. I think it needs a fix wherein we set > the local-slot's invalidation status (provided it is not invalidated > already) from the remote slot before this check itself. And if the > slot is invalidated locally (eith

Re: Synchronizing slots from primary to standby

2023-11-12 Thread shveta malik
On Mon, Nov 13, 2023 at 11:02 AM Ajin Cherian wrote: > > On Thu, Nov 9, 2023 at 9:54 PM shveta malik wrote: > > > > PFA v32 patches which has below changes: > Testing with this patch, I see that if the failover enabled slot is > invalidated on the primary, then the corresponding synced slot is no

Re: Synchronizing slots from primary to standby

2023-11-12 Thread Ajin Cherian
On Thu, Nov 9, 2023 at 9:54 PM shveta malik wrote: > > PFA v32 patches which has below changes: Testing with this patch, I see that if the failover enabled slot is invalidated on the primary, then the corresponding synced slot is not invalidated on the standby. Instead, I see that it continuously

Re: Synchronizing slots from primary to standby

2023-11-12 Thread shveta malik
On Thu, Nov 9, 2023 at 8:56 AM Amit Kapila wrote: > > On Thu, Nov 9, 2023 at 8:11 AM Amit Kapila wrote: > > > > On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand > > wrote: > > > > > > > Unrelated to above, if there is a user slot on standby with the same > > > > name which the slot-sync worker i

Re: Synchronizing slots from primary to standby

2023-11-12 Thread shveta malik
On Mon, Nov 13, 2023 at 6:19 AM Zhijie Hou (Fujitsu) wrote: > > On Thursday, November 9, 2023 6:54 PM shveta malik > wrote: > > > > > > PFA v32 patches which has below changes: > > Thanks for updating the patch. > > Here are few comments: > > > 2. > The check for wal_level < WAL_LEVEL_LOGICAL. >

RE: Synchronizing slots from primary to standby

2023-11-12 Thread Zhijie Hou (Fujitsu)
On Thursday, November 9, 2023 6:54 PM shveta malik wrote: > > > PFA v32 patches which has below changes: Thanks for updating the patch. Here are few comments: 1. Do we need to update the slot upgrade code in pg_upgrade to upgrade the slot's failover property as well ? 2. The check for wal_

Re: Synchronizing slots from primary to standby

2023-11-10 Thread Drouvot, Bertrand
Hi, On 11/10/23 4:31 AM, shveta malik wrote: On Thu, Nov 9, 2023 at 9:15 PM Drouvot, Bertrand wrote: Yeah I think so, because there is a time window when one could "use" the slot after the promotion and before it is removed. Producing things like: " 2023-11-09 15:16:50.294 UTC [2580462] LOG:

Re: Synchronizing slots from primary to standby

2023-11-10 Thread Drouvot, Bertrand
Hi, On 11/10/23 8:55 AM, Amit Kapila wrote: On Fri, Nov 10, 2023 at 12:50 PM Drouvot, Bertrand wrote: But even if we ERROR out instead of emitting a WARNING, the user would still need to be notified/monitor such errors. I agree that then probably they will come to know earlier because the slo

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Amit Kapila
On Thu, Nov 9, 2023 at 9:15 PM Drouvot, Bertrand wrote: > > > You mean here? > > /* > * Check to see if promotion is requested. Note that we do > * this only after failure, so when you promote, we still > * finish replaying as much as we can from archive and > * pg_wal before failover. >

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Amit Kapila
On Fri, Nov 10, 2023 at 12:50 PM Drouvot, Bertrand wrote: > > On 11/10/23 6:41 AM, Amit Kapila wrote: > > On Thu, Nov 9, 2023 at 7:29 PM Drouvot, Bertrand > > wrote: > > > > Are you saying that we change the state of the already existing slot > > on standby? > > Yes. > > > And, such a state would

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Drouvot, Bertrand
Hi, On 11/10/23 6:41 AM, Amit Kapila wrote: On Thu, Nov 9, 2023 at 7:29 PM Drouvot, Bertrand wrote: Are you saying that we change the state of the already existing slot on standby? Yes. And, such a state would indicate that we are trying to sync the slot with the same name from the primar

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Amit Kapila
On Thu, Nov 9, 2023 at 7:29 PM Drouvot, Bertrand wrote: > > On 11/9/23 3:41 AM, Amit Kapila wrote: > > On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand > > wrote: > >> > >>> Unrelated to above, if there is a user slot on standby with the same > >>> name which the slot-sync worker is trying to cre

Re: Synchronizing slots from primary to standby

2023-11-09 Thread shveta malik
On Thu, Nov 9, 2023 at 9:15 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/9/23 11:54 AM, shveta malik wrote: > > > > PFA v32 patches which has below changes: > > Thanks! > > > 7) Added warning for cases where a user-slot with the same name is > > already present which slot-sync worker is trying to

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Drouvot, Bertrand
Hi, On 11/9/23 11:54 AM, shveta malik wrote: PFA v32 patches which has below changes: Thanks! 7) Added warning for cases where a user-slot with the same name is already present which slot-sync worker is trying to create. Sync for such slots is skipped. I'm seeing assertion and segfault

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Drouvot, Bertrand
Hi, On 11/9/23 3:41 AM, Amit Kapila wrote: On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand wrote: Unrelated to above, if there is a user slot on standby with the same name which the slot-sync worker is trying to create, then shall it emit a warning and skip the sync of that slot or shall it

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Amit Kapila
On Thu, Nov 9, 2023 at 8:11 AM Amit Kapila wrote: > > On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand > wrote: > > > > > Unrelated to above, if there is a user slot on standby with the same > > > name which the slot-sync worker is trying to create, then shall it > > > emit a warning and skip the

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Amit Kapila
On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand wrote: > > > Unrelated to above, if there is a user slot on standby with the same > > name which the slot-sync worker is trying to create, then shall it > > emit a warning and skip the sync of that slot or shall it throw an > > error? > > > > I'd vo

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Drouvot, Bertrand
Hi, On 11/8/23 12:50 PM, shveta malik wrote: On Wed, Nov 8, 2023 at 3:19 PM Drouvot, Bertrand wrote: Hi, On 11/8/23 9:57 AM, Amit Kapila wrote: On Wed, Nov 8, 2023 at 12:32 PM Drouvot, Bertrand wrote: Hi, On 11/8/23 4:50 AM, Amit Kapila wrote: I think if we want to follow this approac

Re: Synchronizing slots from primary to standby

2023-11-08 Thread shveta malik
On Wed, Nov 8, 2023 at 3:19 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/8/23 9:57 AM, Amit Kapila wrote: > > On Wed, Nov 8, 2023 at 12:32 PM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 11/8/23 4:50 AM, Amit Kapila wrote: > >> > >>> I think if we want to follow > >>> this approach th

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Drouvot, Bertrand
Hi, On 11/8/23 9:57 AM, Amit Kapila wrote: On Wed, Nov 8, 2023 at 12:32 PM Drouvot, Bertrand wrote: Hi, On 11/8/23 4:50 AM, Amit Kapila wrote: I think if we want to follow this approach then we need to also monitor these slots for any change in the consecutive cycles and if we are able to

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Amit Kapila
On Wed, Nov 8, 2023 at 12:32 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/8/23 4:50 AM, Amit Kapila wrote: > > > I think if we want to follow > > this approach then we need to also monitor these slots for any change > > in the consecutive cycles and if we are able to sync them then > > accordingl

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Drouvot, Bertrand
Hi, On 11/8/23 4:50 AM, Amit Kapila wrote: On Tue, Nov 7, 2023 at 7:58 PM Drouvot, Bertrand wrote: If we think this window is too short we could: - increase it or - don't drop the slot once created (even if there is no activity on the primary during PrimaryCatchupWaitAttempt attempts) so tha

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Amit Kapila
On Tue, Nov 7, 2023 at 7:58 PM Drouvot, Bertrand wrote: > > On 11/7/23 11:55 AM, Amit Kapila wrote: > >>> > >>> This is not full proof solution but optimization over first one. Now > >>> in any sync-cycle, we take 2 attempts for slots-creation (if any slots > >>> are available to be created). In f

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Drouvot, Bertrand
Hi, On 11/7/23 11:55 AM, Amit Kapila wrote: On Tue, Nov 7, 2023 at 3:51 PM Drouvot, Bertrand wrote: On 10/31/23 10:37 AM, shveta malik wrote: On Fri, Oct 27, 2023 at 8:43 PM Drouvot, Bertrand Agree with your test case, but in my case I was not using pub/sub. I was not clear, so when I sai

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Amit Kapila
On Tue, Nov 7, 2023 at 3:51 PM Drouvot, Bertrand wrote: > > On 10/31/23 10:37 AM, shveta malik wrote: > > On Fri, Oct 27, 2023 at 8:43 PM Drouvot, Bertrand > >> > >> Agree with your test case, but in my case I was not using pub/sub. > >> > >> I was not clear, so when I said: > >> > - create l

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Drouvot, Bertrand
Hi, On 10/31/23 10:37 AM, shveta malik wrote: On Fri, Oct 27, 2023 at 8:43 PM Drouvot, Bertrand Agree with your test case, but in my case I was not using pub/sub. I was not clear, so when I said: - create logical_slot1 on the primary (and don't start using it) I meant don't start decoding

Re: Synchronizing slots from primary to standby

2023-11-06 Thread Amit Kapila
On Mon, Nov 6, 2023 at 1:57 PM Amit Kapila wrote: > > On Mon, Nov 6, 2023 at 7:01 AM Zhijie Hou (Fujitsu) > wrote: > > +static void +WalSndGetStandbySlots(List **standby_slots, bool force) +{ + if (!MyReplicationSlot->data.failover) + return; + + if (standby_slot_names_list == NIL && strcmp(stan

Re: Synchronizing slots from primary to standby

2023-11-06 Thread Amit Kapila
On Mon, Nov 6, 2023 at 7:01 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, November 3, 2023 7:32 PM Amit Kapila > > > > 5. > > @@ -228,6 +230,28 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo > > fcinfo, bool confirm, bool bin > > NameStr(MyReplicationSlot->data.plugin), > > format_proce

Re: Synchronizing slots from primary to standby

2023-11-05 Thread shveta malik
On Mon, Nov 6, 2023 at 7:01 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, November 3, 2023 7:32 PM Amit Kapila > > > > On Thu, Nov 2, 2023 at 2:35 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Here is the new version patch set(V29) which addressed Peter > > > comments[1][2] and fixed one doc co

Re: Synchronizing slots from primary to standby

2023-11-03 Thread Amit Kapila
On Thu, Nov 2, 2023 at 2:35 PM Zhijie Hou (Fujitsu) wrote: > > Here is the new version patch set(V29) which addressed Peter comments[1][2] > and > fixed one doc compile error. > Few comments: == 1. + +failover (boolean) + + + Specifies whether

RE: Synchronizing slots from primary to standby

2023-10-31 Thread Zhijie Hou (Fujitsu)
On Tuesday, October 31, 2023 6:45 PM Amit Kapila wrote: > > b) Currently I have restricted 'alter subscription.. refresh > > publication with copy=true' when failover=true (on a similar line of > > two-phase). The reason being, refresh with copy=true will go for > > table-sync again and since fa

Re: Synchronizing slots from primary to standby

2023-10-31 Thread Amit Kapila
On Fri, Oct 27, 2023 at 4:04 PM shveta malik wrote: > > On Fri, Oct 27, 2023 at 3:26 PM shveta malik wrote: > == > > Open questions regarding change for pt 1 above: > a) I think we should restrict the 'alter-sub set failover' when > failover-state is currently in 'p' (pending) state i.e.

Re: Synchronizing slots from primary to standby

2023-10-31 Thread shveta malik
On Fri, Oct 27, 2023 at 8:43 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/27/23 11:56 AM, shveta malik wrote: > > On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/25/23 5:00 AM, shveta malik wrote: > >>> On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand

Re: Synchronizing slots from primary to standby

2023-10-31 Thread shveta malik
On Thu, Oct 26, 2023 at 5:43 PM Amit Kapila wrote: > > On Thu, Oct 26, 2023 at 5:38 PM Drouvot, Bertrand > wrote: > > > > On 10/26/23 10:40 AM, Amit Kapila wrote: > > > On Wed, Oct 25, 2023 at 8:49 PM Drouvot, Bertrand > > > wrote: > > >> > > > > > > Good point, I think we should enhance the Wal

Re: Synchronizing slots from primary to standby

2023-10-30 Thread Amit Kapila
On Tue, Oct 31, 2023 at 7:16 AM Hayato Kuroda (Fujitsu) wrote: > > > > 2. > > > Currently ctx->failover is set only in the pgoutput_startup(), but not > > > sure it is > > OK. > > > Can we change the parameter in CreateDecodingContext() or similar > > > functions? > > > > > > Because IIUC it mea

RE: Synchronizing slots from primary to standby

2023-10-30 Thread Hayato Kuroda (Fujitsu)
Dear Ajin, Thanks for your reply! > On Thu, Oct 26, 2023 at 6:08 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Dear Shveta, > > > > > PFA v25 patch set. The changes are: > > > > Thanks for making the patch! It seems that there are lots of comments, so > > I can put some high-level comments for 000

Re: Synchronizing slots from primary to standby

2023-10-30 Thread Ajin Cherian
On Thu, Oct 26, 2023 at 6:08 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Shveta, > > > PFA v25 patch set. The changes are: > > Thanks for making the patch! It seems that there are lots of comments, so > I can put some high-level comments for 0001. > Sorry if there are duplicated comments. > > 1. >

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Amit Kapila
On Fri, Oct 27, 2023 at 9:00 PM Drouvot, Bertrand wrote: > > On 10/27/23 10:35 AM, Amit Kapila wrote: > > On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand > > > > I think even if we provide such an API, we need to have logic to get > > the slots from the primary and create them. > > Yeah, my idea

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Drouvot, Bertrand
Hi, On 10/27/23 10:35 AM, Amit Kapila wrote: On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand I think even if we provide such an API, we need to have logic to get the slots from the primary and create them. Yeah, my idea was to add an API (in addition to what is already in place). Say, eve

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Drouvot, Bertrand
Hi, On 10/27/23 10:51 AM, shveta malik wrote: On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: I discussed this with my colleague Hou-San and we think that one possibility could be to somehow accelerate the increment of restart_lsn on primary. This can be achieved by connecting to the

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Drouvot, Bertrand
Hi, On 10/27/23 11:56 AM, shveta malik wrote: On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: Hi, On 10/25/23 5:00 AM, shveta malik wrote: On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand wrote: Hi, On 10/23/23 2:56 PM, shveta malik wrote: On Mon, Oct 23, 2023 at 5:52 PM Dro

Re: Synchronizing slots from primary to standby

2023-10-27 Thread shveta malik
On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/25/23 5:00 AM, shveta malik wrote: > > On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/23/23 2:56 PM, shveta malik wrote: > >>> On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand

Re: Synchronizing slots from primary to standby

2023-10-27 Thread shveta malik
On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/25/23 5:00 AM, shveta malik wrote: > > On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/23/23 2:56 PM, shveta malik wrote: > >>> On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand

Re: Synchronizing slots from primary to standby

2023-10-27 Thread Amit Kapila
On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: > > On 10/25/23 5:00 AM, shveta malik wrote: > > On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/23/23 2:56 PM, shveta malik wrote: > >>> On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand > >>> w

Re: Synchronizing slots from primary to standby

2023-10-26 Thread Amit Kapila
On Thu, Oct 26, 2023 at 12:38 PM Hayato Kuroda (Fujitsu) wrote: > > > PFA v25 patch set. The changes are: > > Thanks for making the patch! It seems that there are lots of comments, so > I can put some high-level comments for 0001. > Sorry if there are duplicated comments. > > 1. > The patch seemed

Re: Synchronizing slots from primary to standby

2023-10-26 Thread Amit Kapila
On Thu, Oct 26, 2023 at 5:38 PM Drouvot, Bertrand wrote: > > On 10/26/23 10:40 AM, Amit Kapila wrote: > > On Wed, Oct 25, 2023 at 8:49 PM Drouvot, Bertrand > > wrote: > >> > > > > Good point, I think we should enhance the WalSndWait() logic to > > address this case. > > Agree. I think it would ne

Re: Synchronizing slots from primary to standby

2023-10-26 Thread Drouvot, Bertrand
Hi, On 10/26/23 10:40 AM, Amit Kapila wrote: On Wed, Oct 25, 2023 at 8:49 PM Drouvot, Bertrand wrote: Good point, I think we should enhance the WalSndWait() logic to address this case. Agree. I think it would need to take care of the new CV and probably provide a way for the caller to de

Re: Synchronizing slots from primary to standby

2023-10-26 Thread Amit Kapila
On Wed, Oct 25, 2023 at 8:49 PM Drouvot, Bertrand wrote: > > On 10/9/23 12:30 PM, shveta malik wrote: > > PFA v22 patch-set. It has below changes: > > > > patch 001: > > 1) Now physical walsender wakes up logical walsender(s) by using a new > > CV as suggested in [1] > > Thanks! > > I think that w

RE: Synchronizing slots from primary to standby

2023-10-26 Thread Hayato Kuroda (Fujitsu)
Dear Shveta, > PFA v25 patch set. The changes are: Thanks for making the patch! It seems that there are lots of comments, so I can put some high-level comments for 0001. Sorry if there are duplicated comments. 1. The patch seemed not to consider the case that failover option between replication

Re: Synchronizing slots from primary to standby

2023-10-25 Thread Drouvot, Bertrand
Hi, On 10/9/23 12:30 PM, shveta malik wrote: PFA v22 patch-set. It has below changes: patch 001: 1) Now physical walsender wakes up logical walsender(s) by using a new CV as suggested in [1] Thanks! I think that works fine as long as the standby is up and running and catching up. The probl

Re: Synchronizing slots from primary to standby

2023-10-25 Thread Drouvot, Bertrand
Hi, On 10/25/23 6:57 AM, Amit Kapila wrote: On Tue, Oct 24, 2023 at 3:35 PM Drouvot, Bertrand wrote: On 10/24/23 7:44 AM, Ajin Cherian wrote: On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand wrote: 2) When we create a subscription, another slot is created during the subscription synchr

Re: Synchronizing slots from primary to standby

2023-10-25 Thread Drouvot, Bertrand
Hi, On 10/25/23 5:00 AM, shveta malik wrote: On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand wrote: Hi, On 10/23/23 2:56 PM, shveta malik wrote: On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand wrote: We are waiting for DEFAULT_NAPTIME_PER_CYCLE (3 minutes) before checking if there

Re: Synchronizing slots from primary to standby

2023-10-24 Thread Amit Kapila
On Tue, Oct 24, 2023 at 3:35 PM Drouvot, Bertrand wrote: > > On 10/24/23 7:44 AM, Ajin Cherian wrote: > > On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand > > wrote: > > 2) When we create a subscription, another slot is created during the > subscription synchronization, namely > like "pg_16397

Re: Synchronizing slots from primary to standby

2023-10-24 Thread shveta malik
On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand wrote: > > Hi, > > On 10/23/23 2:56 PM, shveta malik wrote: > > On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand > > wrote: > > >> We are waiting for DEFAULT_NAPTIME_PER_CYCLE (3 minutes) before checking > >> if there > >> is new synced slot(s)

Re: Synchronizing slots from primary to standby

2023-10-24 Thread shveta malik
On Tue, Oct 24, 2023 at 3:35 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/24/23 7:44 AM, Ajin Cherian wrote: > > On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand > > wrote: > >> > >> @@ -602,6 +602,9 @@ CreateDecodingContext(XLogRecPtr start_lsn, > >> SnapBuildSetTwoPhaseAt(ctx->snap

Re: Synchronizing slots from primary to standby

2023-10-24 Thread Drouvot, Bertrand
Hi, On 10/24/23 7:44 AM, Ajin Cherian wrote: On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand wrote: @@ -602,6 +602,9 @@ CreateDecodingContext(XLogRecPtr start_lsn, SnapBuildSetTwoPhaseAt(ctx->snapshot_builder, start_lsn); } + /* set failover in the slot, as requested */

Re: Synchronizing slots from primary to standby

2023-10-24 Thread Peter Smith
Here are some review comments for patch v25-0002 (additional to v25-0002 review comments [1]) == src/backend/catalog/system_views.sql 1. @@ -1003,7 +1003,8 @@ CREATE VIEW pg_replication_slots AS L.safe_wal_size, L.two_phase, L.conflicting, -

Re: Synchronizing slots from primary to standby

2023-10-24 Thread Peter Smith
Here are some review comments for v24-0001 == 1. GENERAL - failover slots terminology There is inconsistent terminology, such as below. Try to use the same wording everywhere. - failover logical slots - failover slots - logical failover slots - logical replication failover slots - etc. These

Re: Synchronizing slots from primary to standby

2023-10-23 Thread Drouvot, Bertrand
Hi, On 10/23/23 2:56 PM, shveta malik wrote: On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand wrote: We are waiting for DEFAULT_NAPTIME_PER_CYCLE (3 minutes) before checking if there is new synced slot(s) to be created on the standby. Do we want to keep this behavior for V1? I think fo

Re: Synchronizing slots from primary to standby

2023-10-23 Thread Ajin Cherian
On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/20/23 5:27 AM, shveta malik wrote: > > On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: > > > > PFA v25 patch set. The changes are: > > > > 1) 'enable_failover' is changed to 'failover' > > 2) Alter subscription changes

RE: Synchronizing slots from primary to standby

2023-10-23 Thread Zhijie Hou (Fujitsu)
On Friday, October 20, 2023 11:27 AM shveta malik wrote: > > The changes are mostly in patch001 and a very few in patch002. > > Thank You Ajin for working on alter-subscription changes and adding more > TAP-tests for 'failover' Thanks for updating the patch. Here are few things I noticed when

Re: Synchronizing slots from primary to standby

2023-10-23 Thread shveta malik
On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/20/23 5:27 AM, shveta malik wrote: > > On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: > > > > PFA v25 patch set. The changes are: > > > > 1) 'enable_failover' is changed to 'failover' > > 2) Alter subscription changes

Re: Synchronizing slots from primary to standby

2023-10-23 Thread Drouvot, Bertrand
Hi, On 10/20/23 5:27 AM, shveta malik wrote: On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: PFA v25 patch set. The changes are: 1) 'enable_failover' is changed to 'failover' 2) Alter subscription changes to support 'failover' 3) Fixes a bug in patch001 wherein any change in standby_slot_n

Re: Synchronizing slots from primary to standby

2023-10-22 Thread Drouvot, Bertrand
Hi, On 10/18/23 6:43 AM, shveta malik wrote: On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand +static void +ProcessRepliesAndTimeOut(void) +{ + CHECK_FOR_INTERRUPTS(); + + /* Process any requests or signals received recently */ + if (ConfigReloadPending) + { +

Re: Synchronizing slots from primary to standby

2023-10-19 Thread shveta malik
On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/13/23 10:35 AM, shveta malik wrote: > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik wrote: > >> > > > > PFA v24 patch set which has below changes: > > > > 1) 'enable_failover' displayed in pg_replication_slots. > > 2) Sup

Re: Synchronizing slots from primary to standby

2023-10-19 Thread shveta malik
On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: > > On Tue, Oct 17, 2023 at 2:01 PM shveta malik wrote: > > > > On Tue, Oct 17, 2023 at 12:44 PM Peter Smith wrote: > > > > > > FYI - the latest patch failed to apply. > > > > > > [postgres@CentOS7-x64 oss_postgres_misc]$ git apply > > > ../patc

Re: Synchronizing slots from primary to standby

2023-10-18 Thread Amit Kapila
On Tue, Oct 17, 2023 at 2:01 PM shveta malik wrote: > > On Tue, Oct 17, 2023 at 12:44 PM Peter Smith wrote: > > > > FYI - the latest patch failed to apply. > > > > [postgres@CentOS7-x64 oss_postgres_misc]$ git apply > > ../patches_misc/v24-0001-Allow-logical-walsenders-to-wait-for-the-physica.pat

Re: Synchronizing slots from primary to standby

2023-10-18 Thread shveta malik
On Wed, Oct 18, 2023 at 10:20 AM Amit Kapila wrote: > > On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand > wrote: > > > > On 10/13/23 10:35 AM, shveta malik wrote: > > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik > > > wrote: > > >> > > > > Code: > > > > + True if this logical slot is

Re: Synchronizing slots from primary to standby

2023-10-17 Thread Amit Kapila
On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand wrote: > > On 10/13/23 10:35 AM, shveta malik wrote: > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik wrote: > >> > > Code: > > + True if this logical slot is enabled to be synced to the physical > standbys > + so that logical replica

Re: Synchronizing slots from primary to standby

2023-10-17 Thread shveta malik
On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/13/23 10:35 AM, shveta malik wrote: > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik wrote: > >> > > > > PFA v24 patch set which has below changes: > > > > 1) 'enable_failover' displayed in pg_replication_slots. > > 2) Sup

<    2   3   4   5   6   7   8   9   >