Re: logical decoding and replication of sequences, take 2

2023-11-30 Thread Amit Kapila
der->committed.xcnt > $4 = 11532 > - > > And with each iteration it grows by 1. > Can we somehow avoid this either by keeping DDL-related xacts open or aborting them? Also, will it make any difference to use setval as do_setval() seems to be logging each time? If possible, can you share the scripts? Kuroda-San has access to the performance machine, he may be able to try it as well. -- With Regards, Amit Kapila.

Re: [Proposal] global sequence implemented by snowflake ID

2023-11-30 Thread Amit Kapila
On Thu, Nov 30, 2023 at 6:48 AM Michael Paquier wrote: > > On Tue, Nov 28, 2023 at 02:23:44PM +0530, Amit Kapila wrote: > > It is interesting to see you want to work towards globally distributed > > sequences. I think it would be important to discuss how and what we &

Re: Is this a problem in GenericXLogFinish()?

2023-11-30 Thread Amit Kapila
fferent > (1077 ms -> 1125 ms). > In my environment, it increased from 375ms to 385ms (median of five runs). I think it is acceptable especially if it increases code coverage. Can you once check that? -- With Regards, Amit Kapila.

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-11-30 Thread Amit Kapila
message (which would tell the actual problem) to the user in this case also has a value. OTOH, as you said, this case won't happen in practical scenarios, so I am fine either way with a slight tilt toward retaining a better error message (aka the current way). Sawada-San/Bharath, do you have any suggestions on this? -- With Regards, Amit Kapila.

Re: pg_upgrade and logical replication

2023-11-30 Thread Amit Kapila
On Wed, Nov 29, 2023 at 3:02 PM Amit Kapila wrote: > In general, the test cases are a bit complex to understand, so, it will be difficult to enhance these later. The complexity comes from the fact that one upgrade test is trying to test multiple things (a) Enabled/Disabled subscriptions;

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-11-29 Thread Amit Kapila
On Thu, Nov 30, 2023 at 8:40 AM Amit Kapila wrote: > > On Wed, Nov 29, 2023 at 2:56 PM Hayato Kuroda (Fujitsu) > wrote: > > > > > > > > > > > > Pushed! > > > > > > > > Hi all, the CF entry for this is marked RfC, and CI is tr

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-11-29 Thread Amit Kapila
able the autovacuum on old publisher. PSA the patch > file. > Agreed, for now, we should change the test as you proposed. I'll take care of that. However, I wonder, if we should also ensure that autovacuum or any other worker is shut down before walsender processes the last set of WAL before shutdown. We can analyze more on this and probably start a separate thread to discuss this point. -- With Regards, Amit Kapila.

Re: pg_upgrade and logical replication

2023-11-29 Thread Amit Kapila
arate patch. [1] - https://www.postgresql.org/docs/devel/app-pgdump.html -- With Regards, Amit Kapila.

Re: logical decoding and replication of sequences, take 2

2023-11-29 Thread Amit Kapila
? How do we see this work w.r.t to some sort of global sequences? There is some recent discussion where I have raised a similar point [1]. [1] - https://www.postgresql.org/message-id/CAA4eK1JF%3D4_Eoq7FFjHSe98-_ooJ5QWd0s2_pj8gR%2B_dvwKxvA%40mail.gmail.com -- With Regards, Amit Kapila.

Re: pg_upgrade and logical replication

2023-11-29 Thread Amit Kapila
e, I have modified a few comments and messages in the attached. Kindly review and include the changes if you are fine with those. -- With Regards, Amit Kapila. diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 4a4bafba11..a4e723f922 100644 --- a/src/bin/pg_dump/pg

Re: Synchronizing slots from primary to standby

2023-11-28 Thread Amit Kapila
On Tue, Nov 28, 2023 at 2:17 PM Drouvot, Bertrand wrote: > > On 11/2/23 1:27 AM, Zhijie Hou (Fujitsu) wrote: > > On Tuesday, October 31, 2023 6:45 PM Amit Kapila > > wrote: > >> We have create_replication_slot and drop_replication_slot in repl_gram.y. >

Re: logical decoding and replication of sequences, take 2

2023-11-28 Thread Amit Kapila
t; 0006 in the attached patch series does this, and it seems to fix the TAP > test failure. I left it at the end, to make it easier to run tests > without the patch applied. > Offhand, I don't have any better idea than what you have suggested for the problem but this needs some thoughts including the questions asked by you. I'll spend some time on it and respond back. -- With Regards, Amit Kapila.

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-11-28 Thread Amit Kapila
ogicalfuncs.c? > I am not sure if either of those is better than logical.c. IIRC, I thought it was okay to keep in logical.c as others primarily deal with exposed SQL functions and I felt it somewhat matches with the intent of logical.c ("The goal is to encapsulate most of the internal complexity for consumers of logical decoding, so they can create and consume a changestream with a low amount of code.."). -- With Regards, Amit Kapila.

Re: [Proposal] global sequence implemented by snowflake ID

2023-11-28 Thread Amit Kapila
ghts? [1] - https://commitfest.postgresql.org/45/3823/ -- With Regards, Amit Kapila.

Re: pg_upgrade and logical replication

2023-11-27 Thread Amit Kapila
On Mon, Nov 27, 2023 at 3:18 PM vignesh C wrote: > > On Sat, 25 Nov 2023 at 17:50, Amit Kapila wrote: > > > > On Sat, Nov 25, 2023 at 7:21 AM vignesh C wrote: > > > > > > > Few comments on v19: > > == > > 1. > > +

Re: logical decoding and replication of sequences, take 2

2023-11-27 Thread Amit Kapila
On Mon, Nov 27, 2023 at 4:41 PM Amit Kapila wrote: > > On Mon, Nov 27, 2023 at 4:17 PM Tomas Vondra > wrote: > > > > > FWIW I think one of the earlier patch versions did something like this, > > by adding a "created" flag in the xlog record. And we concl

Re: logical decoding and replication of sequences, take 2

2023-11-27 Thread Amit Kapila
On Mon, Nov 27, 2023 at 4:17 PM Tomas Vondra wrote: > > On 11/27/23 11:13, Amit Kapila wrote: > > On Mon, Nov 27, 2023 at 11:34 AM Amit Kapila > > wrote: > >> > >> On Mon, Nov 27, 2023 at 6:41 AM Tomas Vondra > >> wrote: > >>

Re: Synchronizing slots from primary to standby

2023-11-27 Thread Amit Kapila
in the sense of whether launcher is the right candidate for the same but now with the single slot-sync worker, it doesn't seem worth having it. What do you think? -- With Regards, Amit Kapila.

Re: logical decoding and replication of sequences, take 2

2023-11-27 Thread Amit Kapila
On Mon, Nov 27, 2023 at 11:34 AM Amit Kapila wrote: > > On Mon, Nov 27, 2023 at 6:41 AM Tomas Vondra > wrote: > > > > While going over 0001, I realized there might be an optimization for > > ReorderBufferSequenceIsTransactional. As coded in 0001, it always > &g

Re: pgoutput incorrectly replaces missing values with NULL since PostgreSQL 15

2023-11-26 Thread Amit Kapila
On Fri, Nov 24, 2023 at 7:23 PM Nikhil Benesch wrote: > > Thank you both for reviewing. The updated patch set LGTM. > Pushed! -- With Regards, Amit Kapila.

Re: logical decoding and replication of sequences, take 2

2023-11-26 Thread Amit Kapila
f other stuff. > > commit 29d0a77fa6606f9c01ba17311fc452dabd3f793d > Author: Amit Kapila > Date: Thu Oct 26 06:54:16 2023 +0530 > > Migrate logical slots to the new node during an upgrade. > ... > > I fixed that, but perhaps someone might want to double check ... >

Re: pg_upgrade and logical replication

2023-11-25 Thread Amit Kapila
lem with restoring stale originLSN in such cases as we won't be able to distinguish during the upgrade but I think we should document it in the comments somewhere in the patch. -- With Regards, Amit Kapila.

Re: undetected deadlock in ALTER SUBSCRIPTION ... REFRESH PUBLICATION

2023-11-25 Thread Amit Kapila
iscovered in this thread. I think you can compare the timing of regression tests in subscription, with and without the patch to show there is no regression. And probably some tests with a large number of tables for sync with very little data. -- With Regards, Amit Kapila.

Re: pgoutput incorrectly replaces missing values with NULL since PostgreSQL 15

2023-11-24 Thread Amit Kapila
On Thu, Nov 23, 2023 at 2:33 PM Amit Kapila wrote: > > On Thu, Nov 23, 2023 at 1:10 PM Nikhil Benesch > wrote: > > > > While working on Materialize's streaming logical replication from Postgres > > [0], > > my colleagues Sean Loiselle and Petros Angelatos

Re: Synchronizing slots from primary to standby

2023-11-24 Thread Amit Kapila
> in mind as improvement for later? > Agreed, we can think of improving it in the future but there is no correctness issue with the current implementation (not trying to do the last fetch after the promotion signal). -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-23 Thread Amit Kapila
ponsibility to match the > failover > property in case it changes the slot_name. > Personally, I think we should document this behavior instead of complicating the patch and the user anyway has a way to achieve it. -- With Regards, Amit Kapila.

Re: undetected deadlock in ALTER SUBSCRIPTION ... REFRESH PUBLICATION

2023-11-23 Thread Amit Kapila
On Wed, Nov 22, 2023 at 4:51 PM Tomas Vondra wrote: > > On 11/22/23 11:38, Amit Kapila wrote: > > > > Okay. IIUC, what's going on here is that the apply worker acquires > > AccessShareLock on pg_subscription to update rel state for one of the > > tables say tbl-1,

Re: pgoutput incorrectly replaces missing values with NULL since PostgreSQL 15

2023-11-23 Thread Amit Kapila
ncludes a test. I've > verified that the test fails against the current master and passes against > the patched version. > > I'm relatively unfamiliar with the project norms here, but assuming the patch > is > acceptable, this strikes me as important enough to warrant a backport to both > v15 and v16. > Right. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-22 Thread Amit Kapila
on is slightly behind what could be requested by subscribers which means the corresponding data will be available on the new primary. Do you have something in mind that can create any problem if we don't attempt additional fetching round after the promotion signal is received? -- With Regards, Amit Kapila.

Re: undetected deadlock in ALTER SUBSCRIPTION ... REFRESH PUBLICATION

2023-11-22 Thread Amit Kapila
On Tue, Nov 21, 2023 at 6:56 PM Tomas Vondra wrote: > > On 11/21/23 14:16, Amit Kapila wrote: > > On Tue, Nov 21, 2023 at 5:17 PM Tomas Vondra > > wrote: > >> > > > > It seems there is some inconsistency in what you have written for > > clie

Re: Lockless exit path for ReplicationOriginExitCleanup

2023-11-22 Thread Amit Kapila
et(). Why can't there be a lockless exit path > something like [1] similar to > replorigin_session_reset() which checks session_replication_state == > NULL without a lock? > I don't see any problem with such a check but not sure of the benefit of doing so either. -- With Regards, Amit Kapila.

Re: Stop the search once replication origin is found

2023-11-22 Thread Amit Kapila
On Mon, Nov 20, 2023 at 4:36 PM Amit Kapila wrote: > > On Mon, Nov 20, 2023 at 2:36 PM Antonin Houska wrote: > > > > Although it's not performance-critical, I think it just makes sense to break > > the loop in replorigin_session_setup() as soon as we've found the origi

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-11-22 Thread Amit Kapila
On Wed, Nov 22, 2023 at 1:30 PM John Naylor wrote: > > On Thu, Nov 9, 2023 at 5:07 PM Amit Kapila wrote: > > > > On Wed, Nov 8, 2023 at 11:05 PM vignesh C wrote: > > > > > > On Wed, 8 Nov 2023 at 08:43, vignesh C wrote: > > > > > &

Re: undetected deadlock in ALTER SUBSCRIPTION ... REFRESH PUBLICATION

2023-11-21 Thread Amit Kapila
nt backend and tablesync worker seems to be blocked for a lock on pg_subscription_rel. > The tablesync workers can't proceed because their lock request is stuck > behind the AccessExclusiveLock request. > > And the apply worker can't proceed, because it's waiting for status > update from the tablesync workers. > This part is not clear to me because wait_for_relation_state_change()->GetSubscriptionRelState() seems to be releasing the lock while closing the relation. Am, I missing something? -- With Regards, Amit Kapila.

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-11-21 Thread Amit Kapila
On Sat, Nov 18, 2023 at 4:54 PM Amit Kapila wrote: > > On Thu, Nov 16, 2023 at 6:09 PM Bharath Rupireddy > wrote: > > PSA v18 patch. > > > > LGTM. I'll push this early next week unless there are further > suggestions or comments. > Pushed. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 6:51 PM Drouvot, Bertrand wrote: > > On 11/20/23 11:59 AM, Amit Kapila wrote: > > On Mon, Nov 20, 2023 at 3:17 PM Drouvot, Bertrand > > wrote: > >> > >> On 11/18/23 11:45 AM, Amit Kapila wrote: > >>> On Fri, Nov 17

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 4:28 PM Amit Kapila wrote: > > 9. > +synchronize_one_slot(WalReceiverConn *wrconn, RemoteSlot *remote_slot, > + bool *slot_updated) > { > ... > + else > + { > + TransactionId xmin_horizon = InvalidTransactionId; > + ReplicationSlot *slot;

Re: Stop the search once replication origin is found

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 2:36 PM Antonin Houska wrote: > > Although it's not performance-critical, I think it just makes sense to break > the loop in replorigin_session_setup() as soon as we've found the origin. > Your proposal sounds reasonable to me. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Mon, Nov 20, 2023 at 3:17 PM Drouvot, Bertrand wrote: > > On 11/18/23 11:45 AM, Amit Kapila wrote: > > On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > > wrote: > >> > >> On 11/17/23 2:46 AM, Zhijie Hou (Fujitsu) wrote: > >>> On Tuesd

Re: Synchronizing slots from primary to standby

2023-11-20 Thread Amit Kapila
On Sat, Nov 18, 2023 at 4:15 PM Amit Kapila wrote: > > On Fri, Nov 17, 2023 at 5:18 PM Drouvot, Bertrand > wrote: > > More Review for v35-0002* > > More review of v35-0002* 1. +/* + * Helper function to check if

Re: pg_upgrade and logical replication

2023-11-19 Thread Amit Kapila
se origins once the system reaches the SYNCDONE state but I am not sure it is a good idea to have such a dependency and that upgrade assuming such things doesn't seems ideal to me. Personally, I think allowing an upgrade in 'i' (initialize) state or 'r' (ready) state seems safe because in those states either slots/origins don't exist or are dropped. What do you think? -- With Regards, Amit Kapila.

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-11-18 Thread Amit Kapila
better than putting > > the ternary in the outer scope. > > I retained wrapping messages in errmsg("..."). > > > You can verify this by doing "make update-po" and then searching for the > > messages in postgres.pot. > > Translation gives me [1] with v18 patch > > PSA v18 patch. > LGTM. I'll push this early next week unless there are further suggestions or comments. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-18 Thread Amit Kapila
it and attempt the wait and sync in next + * sync-cycle. + */ + if (MyReplicationSlot->data.persistency != RS_PERSISTENT) + { + ReplicationSlotPersist(); + *slot_updated = true; + } I think the reason to persist in this case is because next time local restart_lsn can be ahead than the current location and it can take more time to create such a slot. We can probably mention the same in the comments. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-17 Thread Amit Kapila
tatic bool +validate_dbname(const char *dbname) I don't know how important it is to validate the dbname before launching the sync slot worker because anyway after launching, it will give an error while initializing the connection if the dbname is invalid. But, if we think it is really required, did you consider using GetDatabaseTuple()? -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-16 Thread Amit Kapila
atement: GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subbinary, substream, subtwophasestate, subdisableonerr, subpasswordrequired, subrunasowner, subslotname, subsynccommit, subpublications, suborigin) 13. + ConditionVariable wal_confirm_rcv_cv; + WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER]; } WalSndCtlData; It is better to add a comment for this new variable explaining its use. -- With Regards, Amit Kapila.

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-11-15 Thread Amit Kapila
g %s to recalcitrant children", send_abort_for_kill ? "SIGABRT" : "SIGKILL"))); Do here words SIGABRT/SIGKILL remain untranslated due to the translator's comment? I thought this was similar to the message being proposed but seems like this message construction follows translation rules better. -- With Regards, Amit Kapila.

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-11-15 Thread Amit Kapila
: "acquired physical replication slot \"%s\"", >NameStr(s->data.name))); > > ~~~ > Personally, I prefer the way Bharath had in his patch. Did you see any preferred way in the existing code? -- With Regards, Amit Kapila.

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-11-15 Thread Amit Kapila
On Wed, Nov 15, 2023 at 11:00 AM Bharath Rupireddy wrote: > > PSA v15 patch. > The patch looks good to me. I have slightly modified the translator message and commit message in the attached. I'll push this tomorrow unless there are any comments. -- With Regards, Amit Kapila. v16

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-11-14 Thread Amit Kapila
quot;SIGKILL"))); Another minor comment: +Causes each replication command and walsender +process replication slot acquisition/release to be logged in the server +log. Isn't it better to use process's instead of process in the above sentence? [1] -https://www.postgresql.

Re: A recent message added to pg_upgade

2023-11-14 Thread Amit Kapila
On Mon, Nov 13, 2023 at 10:19 AM Michael Paquier wrote: > > On Mon, Nov 13, 2023 at 08:45:12AM +0530, Amit Kapila wrote: > > I think we can be specific about logical replication stuff. I have not > > done any study on autovacuum behavior related to this, so we can > > up

Re: pg_upgrade and logical replication[

2023-11-13 Thread Amit Kapila
On Tue, Nov 14, 2023 at 5:52 AM Michael Paquier wrote: > > On Mon, Nov 13, 2023 at 04:02:27PM +0530, Amit Kapila wrote: > > On Mon, Nov 13, 2023 at 1:52 PM Michael Paquier wrote: > >> It seems to me that INIT cannot be relied on for a similar reason. > >> Thi

Re: Is this a problem in GenericXLogFinish()?

2023-11-13 Thread Amit Kapila
onse doesn't address your concern, can you please explain a bit more about the scenario you have in mind? [1] - /* * If prevpage and the writepage (block in which we are moving tuples * from overflow) are same, then no need to separately register * prevpage. During replay, we can directly update the nextblock in * writepage. */ -- With Regards, Amit Kapila.

Re: pg_upgrade and logical replication

2023-11-13 Thread Amit Kapila
On Mon, Nov 13, 2023 at 5:01 PM Amit Kapila wrote: > > On Fri, Nov 10, 2023 at 7:26 PM vignesh C wrote: > > > > Thanks for the comments, the attached v13 version patch has the > > changes for the same. > > > > + > + ReplicationOriginNameForLogicalRep(

Re: pg_upgrade and logical replication

2023-11-13 Thread Amit Kapila
ation_origin something the patch is doing for some other system table. -- With Regards, Amit Kapila.

Re: pg_upgrade and logical replication

2023-11-13 Thread Amit Kapila
ot committed *and* a slot related to it > lying around. > I don't think this can happen otherwise this could be a problem even without an upgrade after restart. -- With Regards, Amit Kapila.

Re: Is this a problem in GenericXLogFinish()?

2023-11-12 Thread Amit Kapila
at we can acquire a cleanup lock on it if it is the + * same as primary bucket buffer or update the nextblkno if it is same + * as the previous bucket buffer. + */ + else if (xlrec.is_prim_bucket_same_wrt || xlrec.is_prev_bucket_same_wrt) + { + uint8 wbuf_flags; + + Assert(xlrec.ntups == 0); Can we move this comment inside else if, just before Assert? -- With Regards, Amit Kapila.

Re: A recent message added to pg_upgade

2023-11-12 Thread Amit Kapila
On Sat, Nov 11, 2023 at 5:46 AM Michael Paquier wrote: > > On Fri, Nov 10, 2023 at 03:27:25PM +0530, Amit Kapila wrote: > > I don't think this comment is correct because there won't be any apply > > activity on the new cluster as after restoration subscriptions should > >

Re: A recent message added to pg_upgade

2023-11-10 Thread Amit Kapila
AL during the upgrade. Also, I am not completely sure about using the word 'corruption' in this context. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Amit Kapila
n syncslot workers as well on promotion similar to walreceiver? Because we don't want them to even attempt once to sync after promotion. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Amit Kapila
On Fri, Nov 10, 2023 at 12:50 PM Drouvot, Bertrand wrote: > > On 11/10/23 6:41 AM, Amit Kapila wrote: > > On Thu, Nov 9, 2023 at 7:29 PM Drouvot, Bertrand > > wrote: > > > > Are you saying that we change the state of the already existing slot > > on stand

Re: Synchronizing slots from primary to standby

2023-11-09 Thread Amit Kapila
On Thu, Nov 9, 2023 at 7:29 PM Drouvot, Bertrand wrote: > > On 11/9/23 3:41 AM, Amit Kapila wrote: > > On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand > > wrote: > >> > >>> Unrelated to above, if there is a user slot on standby with the same > >

Re: A recent message added to pg_upgade

2023-11-09 Thread Amit Kapila
check callback ensures the value is > not overridden by the user/ > These comments appear mostly repetitive to what is already mentioned in start_postmaster(). So, I have changed those referred to already written comments, and slightly adjusted the comments at another place. See attached. Personally, I don't see the need for a test for this, so removed the same but can add it back if you or others think so. -- With Regards, Amit Kapila. inhibit_m_s_w_k_s_during_upgrade_6.patch Description: Binary data

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-11-09 Thread Amit Kapila
he same. > Pushed! -- With Regards, Amit Kapila.

Re: A recent message added to pg_upgade

2023-11-08 Thread Amit Kapila
take care of this unless we see some opposition to this idea. [1] - https://www.postgresql.org/message-id/20231102.115834.1012152975995247837.horikyota.ntt%40gmail.com -- With Regards, Amit Kapila.

Re: A recent message added to pg_upgade

2023-11-08 Thread Amit Kapila
On Thu, Nov 9, 2023 at 11:40 AM Kyotaro Horiguchi wrote: > > At Thu, 9 Nov 2023 09:53:07 +0530, Amit Kapila > wrote in > > Michael, Horiguchi-San, and others, do you have any thoughts on what > > is the best way to proceed? > > As I previously mentioned,

Re: pg_upgrade and logical replication

2023-11-08 Thread Amit Kapila
hanges be synchronized till all the new tables are created and synced before step 7? -- With Regards, Amit Kapila.

Re: A recent message added to pg_upgade

2023-11-08 Thread Amit Kapila
On Tue, Nov 7, 2023 at 4:16 PM Amit Kapila wrote: > > On Tue, Nov 7, 2023 at 8:12 AM Michael Paquier wrote: > > > > On Tue, Nov 07, 2023 at 07:59:46AM +0530, Amit Kapila wrote: > > > Do you mean to say that if 'IsBinaryUpgrade' is true then let's not > > &g

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Amit Kapila
On Thu, Nov 9, 2023 at 8:11 AM Amit Kapila wrote: > > On Wed, Nov 8, 2023 at 8:09 PM Drouvot, Bertrand > wrote: > > > > > Unrelated to above, if there is a user slot on standby with the same > > > name which the slot-sync worker is trying to create, then shall

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Amit Kapila
ot sync mechanism would be stopped. Do you have reasons to prefer giving a WARNING and skipping creating such slots? I expect this WARNING to keep getting repeated in LOGs because the consecutive sync tries will again generate a WARNING. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-08 Thread Amit Kapila
On Wed, Nov 8, 2023 at 12:32 PM Drouvot, Bertrand wrote: > > Hi, > > On 11/8/23 4:50 AM, Amit Kapila wrote: > > > I think if we want to follow > > this approach then we need to also monitor these slots for any change > > in the consecutive cycle

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-11-08 Thread Amit Kapila
the whole of > dbinfos by using pg_malloc0 instead of pg_malloc which will ensure > that the slot information is set to 0. > I would prefer this fix instead of initializing the slot array at multiple places. I'll push this tomorrow unless someone thinks otherwise. -- With Regards, Amit Kapila.

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-11-07 Thread Amit Kapila
00839a in check_new_cluster () at check.c:215 > #9 0x55563010 in main (argc=13, argv=0x7fffdf08) at > pg_upgrade.c:136 > > This issue occurs because we are accessing uninitialized slot array > information. > Thanks for the report. I'll review your proposed fix. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Amit Kapila
On Tue, Nov 7, 2023 at 7:58 PM Drouvot, Bertrand wrote: > > On 11/7/23 11:55 AM, Amit Kapila wrote: > >>> > >>> This is not full proof solution but optimization over first one. Now > >>> in any sync-cycle, we take 2 attempts for slots-creation (if a

Re: Synchronizing slots from primary to standby

2023-11-07 Thread Amit Kapila
itted > (and that > specially true/worst for non active slots). > > Maybe something like "newly locally created slot XXX has been synced..."? > > While at it, would that make sense to move > > + slot->data.failover = true; > > once we stop waiting for this slot? I think that would avoid confusion if one > query pg_replication_slots while we are still waiting for this slot to be > synced, > thoughts? (currently we can see pg_replication_slots.synced_slot set to true > while we are still waiting). > The failover property of the slot is different from whether the slot has been synced yet, so we can't change the location of marking it but we can try to improve when to show that slot has been synced. -- With Regards, Amit Kapila.

Re: A recent message added to pg_upgade

2023-11-07 Thread Amit Kapila
On Tue, Nov 7, 2023 at 8:12 AM Michael Paquier wrote: > > On Tue, Nov 07, 2023 at 07:59:46AM +0530, Amit Kapila wrote: > > Do you mean to say that if 'IsBinaryUpgrade' is true then let's not > > allow to launch launcher or apply worker? If so, I guess this won't be > > any

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-11-06 Thread Amit Kapila
directory: > ./build/testrun/pg_upgrade/003_logical_slots/data/delete_old_cluster.sh > Thanks for the patch and verification. Pushed the fix. -- With Regards, Amit Kapila.

Re: A recent message added to pg_upgade

2023-11-06 Thread Amit Kapila
On Sun, Nov 5, 2023 at 5:33 AM Michael Paquier wrote: > > On Fri, Nov 03, 2023 at 01:33:26PM +1100, Peter Smith wrote: > > On Fri, Nov 3, 2023 at 1:11 PM Amit Kapila wrote: > >> Now, that Michael also committed another similar change in commit > >> 7021d3b176,

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-11-06 Thread Amit Kapila
On Tue, Nov 7, 2023 at 3:56 AM David Rowley wrote: > > On Thu, 2 Nov 2023 at 22:42, Amit Kapila wrote: > > The other two look good to me. > > Thanks for looking. > > I spent some time trying to see if the performance changes much with > either of these cases. For t

Re: Synchronizing slots from primary to standby

2023-11-06 Thread Amit Kapila
On Mon, Nov 6, 2023 at 1:57 PM Amit Kapila wrote: > > On Mon, Nov 6, 2023 at 7:01 AM Zhijie Hou (Fujitsu) > wrote: > > +static void +WalSndGetStandbySlots(List **standby_slots, bool force) +{ + if (!MyReplicationSlot->data.failover) + return; + + if (standby_slot_names_lis

Re: Synchronizing slots from primary to standby

2023-11-06 Thread Amit Kapila
On Mon, Nov 6, 2023 at 7:01 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, November 3, 2023 7:32 PM Amit Kapila > > > > 5. > > @@ -228,6 +230,28 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo > > fcinfo, bool confirm, bool bin > > Na

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-11-05 Thread Amit Kapila
er 0/1522730, reading WAL from 0/15226F8. 2023-11-06 08:41:58.015 IST [22332] STATEMENT: START_REPLICATION SLOT "sub1" LOGICAL 0/0 (proto_version '4', origin 'any', publication_names '"pub1"') We can get the PID from the log line as for other logs and I don't see the process name printed anywhere else. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-11-03 Thread Amit Kapila
failover is not enabled for the slot. 6. +/* ALTER_REPLICATION_SLOT slot */ +alter_replication_slot: + K_ALTER_REPLICATION_SLOT IDENT '(' generic_option_list ')' I think you need to update the docs for this new command. See existing docs [1]. [1] - https://www.postgresql.org/docs/devel/protocol-replication.html -- With Regards, Amit Kapila.

Re: A recent message added to pg_upgade

2023-11-02 Thread Amit Kapila
On Thu, Nov 2, 2023 at 2:36 PM Amit Kapila wrote: > > On Thu, Nov 2, 2023 at 11:32 AM Michael Paquier wrote: > > > > On Thu, Nov 02, 2023 at 02:32:07PM +1100, Peter Smith wrote: > > > On Thu, Nov 2, 2023 at 2:25 PM Peter Smith wrote: > > >> Checking thi

Re: pg_upgrade and logical replication

2023-11-02 Thread Amit Kapila
atch was that here unlike the publication's slot information, we can't ensure with origin's remote_lsn that all the WAL is received and applied before allowing the upgrade. I can't think of any problem at the moment due to this but still a point worth giving a thought. -- With Regards, Amit Kapila.

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-11-02 Thread Amit Kapila
On Tue, Oct 31, 2023 at 2:25 AM David Rowley wrote: > > On Mon, 30 Oct 2023 at 23:48, Amit Kapila wrote: > > > > On Fri, Oct 27, 2023 at 3:23 AM David Rowley wrote: > > > * parallel.c in HandleParallelMessages(): > > > * applyparallelworker.c in HandleP

Re: A recent message added to pg_upgade

2023-11-02 Thread Amit Kapila
can't imagine all possible scenarios. As this setting is invalid or can cause problems, it seems people favor preventing it. Alvaro also voted in favor of preventing it, so we are considering to proceed with it unless more people think otherwise. -- With Regards, Amit Kapila.

Re: pg_upgrade and logical replication

2023-11-02 Thread Amit Kapila
On Wed, Nov 1, 2023 at 8:33 AM Michael Paquier wrote: > > On Fri, Oct 27, 2023 at 05:05:39PM +0530, Amit Kapila wrote: > > I was analyzing this part and it seems it could be tricky to upgrade > > in FINISHEDCOPY state. Because the system would expect that subscriber >

Re: Is this a problem in GenericXLogFinish()?

2023-11-02 Thread Amit Kapila
On Wed, Nov 1, 2023 at 12:54 PM Michael Paquier wrote: > > On Mon, Oct 30, 2023 at 03:54:39PM +0530, Amit Kapila wrote: > > On Sat, Oct 28, 2023 at 4:30 PM Michael Paquier wrote: > >> On Sat, Oct 28, 2023 at 03:45:13PM +0530, Amit Kapila wrote: > >> > Yes, we

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-10-31 Thread Amit Kapila
ents whether we need to LOG it or not. I guess at some place like atop LogReplicationSlotAcquire() we should document in a bit more specific way as to when is this expected to be called. -- With Regards, Amit Kapila.

Re: Intermittent failure with t/003_logical_slots.pl test on windows

2023-10-31 Thread Amit Kapila
t; > Failure, exiting > [16:24:21.144](6.275s) not ok 10 - run of pg_upgrade of old cluster > > If the same command is run manually, it succeeds - > Can you add some LOGs in pg_resetwal to find out if the command has performed appropriately? -- With Regards, Amit Kapila.

Re: A recent message added to pg_upgade

2023-10-31 Thread Amit Kapila
soleteSlot or > make it an Assert and say do not override max_slot_wal_keep_size in > docs? Even if someone did override, let the pg_upgrade report the slot > as invalidated and let the user delete the slot or decide what to do > with it. > The problem is this can happen in the background so it can happen at the time of shutdown when all the upgrade is complete. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-10-31 Thread Amit Kapila
and changing slot property in between could be tricky. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-10-30 Thread Amit Kapila
lover flag of the slot, > > while plugins that do not support this will not set this and the > > failover flag of the created slot will remain. > > What do you think? > > May be OK, but I came up with a corner case that external plugins have a > streaming > option 'failover'. What should be? Has the option been reserved? > Sorry, your question is not clear to me. Did you intend to say that the value of the existing streaming option could be 'failover'? -- With Regards, Amit Kapila.

Re: Is this a problem in GenericXLogFinish()?

2023-10-30 Thread Amit Kapila
On Mon, Oct 30, 2023 at 7:13 PM Robert Haas wrote: > > On Sat, Oct 28, 2023 at 6:15 AM Amit Kapila wrote: > > > Hmm. So my question is: do we need the cleanup lock on the write > > > buffer even if there are no tuples, and even if primary bucket and the > &g

Re: Open a streamed block for transactional messages during decoding

2023-10-30 Thread Amit Kapila
On Mon, Oct 30, 2023 at 2:17 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, October 30, 2023 12:20 PM Amit Kapila > wrote: > > > > On Thu, Oct 26, 2023 at 2:01 PM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > On Thursda

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-10-30 Thread Amit Kapila
orkers as you have also noticed. The comment atop initReadOnlyStringInfo() clearly states that it is used in the performance-critical path. So, is it worth changing these places? In the future, this may pose the risk of this API being used inconsistently. -- With Regards, Amit Kapila.

Re: Is this a problem in GenericXLogFinish()?

2023-10-30 Thread Amit Kapila
On Sat, Oct 28, 2023 at 4:30 PM Michael Paquier wrote: > > On Sat, Oct 28, 2023 at 03:45:13PM +0530, Amit Kapila wrote: > > Yes, we need it to exclude any concurrent in-progress scans that could > > return incorrect tuples during bucket squeeze operation. > > Thanks. So

Re: PGDOCS - add more links in the pub/sub reference pages

2023-10-30 Thread Amit Kapila
d your patch. -- With Regards, Amit Kapila.

Re: Open a streamed block for transactional messages during decoding

2023-10-29 Thread Amit Kapila
On Thu, Oct 26, 2023 at 2:01 PM Zhijie Hou (Fujitsu) wrote: > > On Thursday, October 26, 2023 12:42 PM Amit Kapila > wrote: > > > > On Tue, Oct 24, 2023 at 5:27 PM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > While re

Re: A recent message added to pg_upgade

2023-10-29 Thread Amit Kapila
On Mon, Oct 30, 2023 at 7:58 AM Kyotaro Horiguchi wrote: > > At Fri, 27 Oct 2023 14:57:10 +0530, Amit Kapila > wrote in > > On Fri, Oct 27, 2023 at 2:02 PM Alvaro Herrera > > wrote: > > > > > > On 2023-Oct-27, Kyotaro Horiguchi

<    2   3   4   5   6   7   8   9   10   11   >