Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-02 Thread Amit Kapila
e comments as to why we didn't support wal_level to be changed from minimal to logical? It will be helpful for future readers/authors to understand what it would require to further extend this functionality. -- With Regards, Amit Kapila.

Re: Fix use of variable after pfree

2025-09-02 Thread Amit Kapila
t and accepting TCP/IP connections? > > I have moved the pfree(app_data.name) after its usage. > > This change was introduced in PG_18. > The patch applies in the HEAD and REL_18_STABLE branches. > Thanks for the patch. It looks good to me. I'll take care of it. -- With Regards, Amit Kapila.

Re: How can end users know the cause of LR slot sync delays?

2025-08-28 Thread Amit Kapila
On Thu, Aug 28, 2025 at 3:29 PM Kirill Reshke wrote: > > On Thu, 28 Aug 2025 at 14:56, Amit Kapila wrote: > > > > On Thu, Aug 28, 2025 at 11:07 AM Ashutosh Sharma > > wrote: > > > > > > We have seen cases where slot synchronization gets delayed, for

Re: Avoid retaining conflict-related data when no tables are subscribed

2025-08-28 Thread Amit Kapila
e serves the purpose but I find it makes the API complex to understand because now it needs to make decisions based on different states depending on the boolean parameter passed. Can we introduce a new API for the empty subscription case? -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-28 Thread Amit Kapila
On Thu, Aug 28, 2025 at 4:39 PM Amit Kapila wrote: > > On Thu, Aug 28, 2025 at 8:02 AM Zhijie Hou (Fujitsu) > wrote: > > > > I noticed that Cfbot failed to compile the document due to a typo after > > renaming > > the subscription option. Here are the upda

Re: Conflict detection for update_deleted in logical replication

2025-08-28 Thread Amit Kapila
a number of cosmetic and comment changes in the attached atop 0001 patch. Kindly include in next version, if you think they are good to include. -- With Regards, Amit Kapila. diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 29d0c9a6e45..3df1828

Re: How can end users know the cause of LR slot sync delays?

2025-08-28 Thread Amit Kapila
ad_remote_LSN. I think ideally users can compare standby's slot LSN/XMIN with remote_slot being synced. Do you have any better ideas? -- With Regards, Amit Kapila.

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-26 Thread Amit Kapila
On Mon, Aug 25, 2025 at 7:02 PM Mihail Nikalayeu wrote: > > Amit Kapila : > > > > What if the new insert happens in a page prior to the current page? I > > mean that the scan won't encounter the page where Insert happens. > > Hmm Yes - if the TID la

Re: Conflict detection for update_deleted in logical replication

2025-08-25 Thread Amit Kapila
On Mon, Aug 25, 2025 at 5:05 PM Amit Kapila wrote: > > A few comments on 0001: > Some more comments: 1. + /* + * Return false if the leader apply worker has stopped retaining + * information for detecting conflicts. This implies that update_deleted + * can no longer be reliably

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-25 Thread Amit Kapila
earby code unless I am missing something. > > The case of logical replication giving wrong results > > [0] is the behavior from the beginning of logical replication. > > Logical replication was mainly focused on replication without any > concurrent updates on the subscriber side. So, I think this is why the > issue was overlooked. > The other possibility is that as this is a rare scenario so we didn't consider it. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-25 Thread Amit Kapila
_info_retention). I don't see resume_conflict_info_retention in 0001, so I couldn't make sense of this part of the comment. -- With Regards, Amit Kapila.

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-24 Thread Amit Kapila
s - this can occur during a single UPDATE, as well as a > DELETE followed by an INSERT of the same key within the same > transaction (which is effectively equivalent to an UPDATE). > BTW, then isn't it possible that INSERT happens on a different page? -- With Regards, Amit Kapila.

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-24 Thread Amit Kapila
because of an unrelated title) then I suggest you start a separate email thread to discuss just that case and see what others think. [0]: https://www.postgresql.org/message-id/flat/CADzfLwWC49oanFSGPTf%3D6FJoTw-kAnpPZV8nVqAyR5KL68LrHQ%40mail.gmail.com#5f6b3be849f8d95c166decfae541df09 -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-22 Thread Amit Kapila
On Mon, Aug 4, 2025 at 3:11 PM Amit Kapila wrote: > > On Mon, Aug 4, 2025 at 11:46 AM shveta malik wrote: > > > > 7) > > Shall we rename 'max_conflict_retention_duration' to > > 'max_conflict_info_retention_duration' as the latter one is more >

Re: Conflict detection for update_deleted in logical replication

2025-08-22 Thread Amit Kapila
nflict_retention(true); Can't we combine these checks by passing both parameters to CheckSubDeadTupleRetention() and let that function handle all inappropriate value cases? BTW, even for other places, see if you can reduce the length of the function name notify_ineffective_max_conflict_retention. -- With Regards, Amit Kapila.

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-22 Thread Amit Kapila
strategy implemented, it is the subscriber operation that will win. So, the current behavior shouldn't cause any problem. -- With Regards, Amit Kapila.

Re: Logical Replication of sequences

2025-08-21 Thread Amit Kapila
On Thu, Aug 21, 2025 at 10:52 PM Masahiko Sawada wrote: > > On Wed, Aug 20, 2025 at 9:04 PM Amit Kapila wrote: > > > > On Wed, Aug 20, 2025 at 11:00 PM Masahiko Sawada > > wrote: > > > > > > On Tue, Aug 19, 2025 at 9:14 PM Amit Kapila > > >

Re: memory leak in logical WAL sender with pgoutput's cachectx

2025-08-21 Thread Amit Kapila
ehow we can decode the DROP TABLE WAL record, say delete of relid from pg_class then we can use that to remove the corresponding entry from RelationSyncCache. -- With Regards, Amit Kapila.

Re: memory leak in logical WAL sender with pgoutput's cachectx

2025-08-21 Thread Amit Kapila
considering after we have some tests to show the high memory usage. BTW, I think it would be better to run some algorithm to evict entries when we reach the threshold limit as we do for shared buffers, see StrategyGetBuffer. Otherwise, we may be paying the cost to maintain such a list when in practice it may be required only very few times. -- With Regards, Amit Kapila.

Re: doc patch: correct function name for slot synchronization.

2025-08-21 Thread Amit Kapila
ddress the issue. The typo > exists > till PG17. > Thanks for noticing this and providing a patch. I'll take care of this. -- With Regards, Amit Kapila.

Re: Logical Replication of sequences

2025-08-20 Thread Amit Kapila
On Wed, Aug 20, 2025 at 11:00 PM Masahiko Sawada wrote: > > On Tue, Aug 19, 2025 at 9:14 PM Amit Kapila wrote: > > > > If so, I don't think we can do much with the design > > choice we made. During DDL replication of sequences, we need to > > consider it as a c

Re: Proposal: Conflict log history table for Logical Replication

2025-08-20 Thread Amit Kapila
On Wed, Aug 20, 2025 at 11:47 AM Dilip Kumar wrote: > > On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila wrote: > > > > > One idea to keep things simple for the first version is that we allow > > users to specify the table_name for storing conflicts but the table > >

Re: Logical Replication of sequences

2025-08-20 Thread Amit Kapila
E_INIT also doesn't sound intuitive, even though it serves the purpose. I feel it is better to use SUBREL_STATE_DATASYNC state as that indicates data is being synchronized, and let the LSN value be the same as the previous. [1]: https://www.postgresql.org/message-id/CAA4eK1LcBoPBCKa9yFOQnvpBv3a2ejf_EWC%3DZKksGcvqW7e0Zg%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Logical Replication of sequences

2025-08-19 Thread Amit Kapila
EFRESH before the ALTER SEQUENCE command; otherwise, the ALTER SEQUENCE won't be replicated, right? If so, I don't think we can do much with the design choice we made. During DDL replication of sequences, we need to consider it as a conflict. BTW, note that the same situation can happen even when the user manually changed the sequence value on the subscriber in some way. So, we can't prevent that. -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-08-18 Thread Amit Kapila
ests are attached. > > > > Next, I plan to extend the testing to larger workloads by running > > pgbench for 20–30 minutes. > > We will also benchmark performance across different workload types to > > evaluate the improvements once the patch has matured further. > > > > -- > > Thanks, > > Nisha > > > I also did some benchmarking of the proposed parallel apply patch and > compare it with my prewarming approach. > And parallel apply is significantly more efficient than prefetch (it is > expected). > Thanks to you and Nisha for doing some preliminary performance testing, the results are really encouraging (more than 3 to 4 times improvement in multiple workloads). I hope we keep making progress on this patch and make it ready for the next release. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-18 Thread Amit Kapila
On Mon, Aug 18, 2025 at 5:05 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, August 18, 2025 2:32 PM Dilip Kumar wrote: > > > > On Mon, Aug 18, 2025 at 10:36 AM Amit Kapila > > wrote: > > > > > > > --- > > > > Even

Re: Proposal: Conflict log history table for Logical Replication

2025-08-17 Thread Amit Kapila
ow storing conflicts in pre-created tables with more checks about its schema. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-17 Thread Amit Kapila
ans that it is possible that before restart one would deduce that the update_deleted conflict won't be reliably detected for a particular subscription but after restart it could lead to the opposite conclusion. But note that to make it behave similarly we need to store this value persistently in pg_subscription unless you have better ideas for this. Theoretically, there are two places where we can persist this information, one is with pg_subscription, and other in origin. I find it is closer to pg_subscription. -- With Regards, Amit Kapila.

Re: Proposal: Conflict log history table for Logical Replication

2025-08-14 Thread Amit Kapila
On Thu, Aug 14, 2025 at 4:26 PM Alastair Turner wrote: > > On Wed, 13 Aug 2025 at 11:09, Amit Kapila wrote: >> >> On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: >> > >> > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: >> > > >>

Re: Parallel Apply

2025-08-14 Thread Amit Kapila
On Wed, Aug 13, 2025 at 8:57 PM Bruce Momjian wrote: > > On Wed, Aug 13, 2025 at 09:50:27AM +0530, Amit Kapila wrote: > > On Tue, Aug 12, 2025 at 10:40 PM Bruce Momjian wrote: > > > > Currently, PostgreSQL supports parallel apply only for large streaming > > > &

Re: Conflict detection for update_deleted in logical replication

2025-08-14 Thread Amit Kapila
On Thu, Aug 14, 2025 at 9:02 AM shveta malik wrote: > > On Wed, Aug 13, 2025 at 5:42 PM Amit Kapila wrote: > > > > > > > > 4) > > > For the DETAIL part of resume and stop messages, how about these: > > > > > > The retention duration for

Re: Excessive LOG messages from replication slot sync worker

2025-08-14 Thread Amit Kapila
On Thu, Aug 14, 2025 at 9:43 AM Fujii Masao wrote: > > On Wed, Aug 13, 2025 at 1:00 PM Amit Kapila wrote: > > > > On Tue, Aug 12, 2025 at 4:28 PM Fujii Masao wrote: > > > > > > I'm not sure these messages are useful for end users, and LOG might not be &

Re: Conflict detection for update_deleted in logical replication

2025-08-13 Thread Amit Kapila
now indefinite. > Similar to the previous point, will it be better to keep it short by using "conflict detection info", for example, it will lead to message like "The retention duration for conflict detection info is now indefinite."? -- With Regards, Amit Kapila.

Re: Proposal: Conflict log history table for Logical Replication

2025-08-13 Thread Amit Kapila
qjxnA2mGt5O%3DDht7sw%40mail.gmail.com [5] - https://www.postgresql.org/message-id/CANhcyEW%2BuJB_bvQLEaZCgoRTc1%3Di%2BQnrPPHxZ2%3D0SBSCyj9pkg%40mail.gmail.com -- With Regards, Amit Kapila.

Re: PG 18 release notes draft committed

2025-08-12 Thread Amit Kapila
xplicitly. -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-08-12 Thread Amit Kapila
> This is a point to investigate if we observe so. But till now in our internal testing parallel apply gives good improvement in pgbench kind of workload. -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-08-12 Thread Amit Kapila
On Tue, Aug 12, 2025 at 10:40 PM Bruce Momjian wrote: > > On Mon, Aug 11, 2025 at 10:15:41AM +0530, Amit Kapila wrote: > > Hi, > > > > Background and Motivation > > - > > In high-throughput systems, where hundreds of session

Re: Excessive LOG messages from replication slot sync worker

2025-08-12 Thread Amit Kapila
ce, I can tell that these messages have been helpful in finding BF failures and debugging bugs from user reports, so there is value in keeping them at LOG level. -- With Regards, Amit Kapila.

Re: Excessive LOG messages from replication slot sync worker

2025-08-12 Thread Amit Kapila
imply applying Shveta's patch to revert the incorrect format change > to fix the problem, and then discussing LSN-based filename standardization > in a separate thread. > +1. -- With Regards, Amit Kapila.

Re: [Patch] add new parameter to pg_replication_origin_session_setup

2025-08-12 Thread Amit Kapila
On Mon, Aug 11, 2025 at 10:41 PM Doruk Yilmaz wrote: > > On Mon, Aug 11, 2025 at 9:44 AM Amit Kapila wrote: > > How do you advance the origin? Did you use > > > pg_replication_origin_advance()? If so, you should be aware that it > > can be used for initial setup; s

Re: Parallel Apply

2025-08-12 Thread Amit Kapila
On Mon, Aug 11, 2025 at 3:00 PM Kirill Reshke wrote: > > On Mon, 11 Aug 2025 at 13:45, Amit Kapila wrote: > > > > I am not sure if that is directly applicable because this work > > proposes to track dependencies based on logical WAL contents. However, > > if you

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread Amit Kapila
E: created replication slot "sub2" on publisher > CREATE SUBSCRIPTION > > Shall we give notice that max_conflict_retention_duration is ignored > as retain_dead_tuples is false. > How about disallowing this combination? -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-08-11 Thread Amit Kapila
On Tue, Aug 12, 2025 at 12:04 PM Andrei Lepikhov wrote: > > On 11/8/2025 06:45, Amit Kapila wrote: > > The core idea is that the leader apply worker ensures the following: > > a. Identifies dependencies between transactions. b. Coordinates > > parallel workers to apply

Re: Parallel Apply

2025-08-11 Thread Amit Kapila
al replication speedup projects like[0] > I am not sure if that is directly applicable because this work proposes to track dependencies based on logical WAL contents. However, if you can point me to README on the overall design of the work you are pointing to then I can check it once. -- With Regards, Amit Kapila.

Re: [Patch] add new parameter to pg_replication_origin_session_setup

2025-08-10 Thread Amit Kapila
On Wed, Jul 30, 2025 at 12:00 AM Doruk Yilmaz wrote: > > On Mon, Jul 29, 2025 at 8:13 AM Amit Kapila wrote: > > That is true but I still feel there has to be some mechanism where we > > can catch and give an ERROR to the user, if it doesn't follow th

Parallel Apply

2025-08-10 Thread Amit Kapila
nt to maintain commit order because they don't explicitly annotate FK, PK for columns but maintain the integrity via application. So, in such cases as we won't be able to detect transaction dependencies, it would be better to allow out-of-order commits optionally. Thoughts? -- With Regards, Amit Kapila.

Re: Dropping publication breaks logical replication

2025-08-08 Thread Amit Kapila
On Fri, Aug 8, 2025 at 5:19 PM Ashutosh Bapat wrote: > > On Tue, Aug 5, 2025 at 9:50 AM Amit Kapila wrote: > > > > On Fri, Aug 1, 2025 at 5:06 PM Ashutosh Bapat > > wrote: > > > > > > On Fri, Aug 1, 2025 at 4:03 PM Amit Kapila > > > wrote: &g

Re: CREATE PUBLICATION with 'publish_generated_columns' parameter specified but unassigned

2025-08-04 Thread Amit Kapila
oo, otherwise the message is not very helpful. > +CREATE PUBLICATION testpub_xxx WITH (publish_generated_columns); +ERROR: invalid value for publication parameter "publish_generated_columns": "" +DETAIL: Valid values are "none" and "stored". I find this message clearer and helpful for users. So, +1 for retaining it. -- With Regards, Amit Kapila.

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-04 Thread Amit Kapila
On Tue, Aug 5, 2025 at 9:28 AM shveta malik wrote: > > On Mon, Aug 4, 2025 at 3:41 PM Amit Kapila wrote: > > > > On Mon, Aug 4, 2025 at 12:19 PM shveta malik wrote: > > If we want to avoid continuously syncing newly added slots in later > cycles and instead focus only

Re: Dropping publication breaks logical replication

2025-08-04 Thread Amit Kapila
On Fri, Aug 1, 2025 at 5:06 PM Ashutosh Bapat wrote: > > On Fri, Aug 1, 2025 at 4:03 PM Amit Kapila wrote: > > > > Now, seeing this report, it seems the customer(s) are probably okay to > > skip a missing publication and let replication continue. So, we should > &

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-04 Thread Amit Kapila
On Mon, Aug 4, 2025 at 12:19 PM shveta malik wrote: > > On Mon, Aug 4, 2025 at 11:31 AM Amit Kapila wrote: > > > > On Fri, Aug 1, 2025 at 2:50 PM shveta malik wrote: > > > > > > 5) > > > I tried a test where there were 4 slots on the publisher, wher

Re: Conflict detection for update_deleted in logical replication

2025-08-04 Thread Amit Kapila
jejQxBkyo9kop2HMw85wSJqpB%3DJapsSE%2BKw_iRg%40mail.gmail.com [2] - https://www.postgresql.org/docs/devel/sql-createsubscription.html -- With Regards, Amit Kapila.

Re: CREATE PUBLICATION with 'publish_generated_columns' parameter specified but unassigned

2025-08-04 Thread Amit Kapila
of shorthand specification (having > no assigned value) mentioned anywhere in the PG docs [1]. > > OTOH, I did see this is being tested/commented in 'publication.sql' > [2], so perhaps that means it is deliberate? > > ~~~ > > What is the verdict for this syntax -- ok or not? > I think we should give ERROR for this. -- With Regards, Amit Kapila.

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-03 Thread Amit Kapila
SLOTSYNC_PRIMARY_CATCHUP "Waiting for the primary to catch-up." Can't we reuse existing waitevent REPLICATION_SLOTSYNC_MAIN? We may want to change the description. Is there a specific reason to add a new wait_event for this API? -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Amit Kapila
On Fri, Aug 1, 2025 at 4:20 PM Amit Kapila wrote: > > On Fri, Aug 1, 2025 at 3:58 PM Dilip Kumar wrote: > > > > 4. > > + /* > > +* Instead of invoking GetOldestNonRemovableTransactionId() for conflict > > +* detection, we use the conflict detectio

Re: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Amit Kapila
vable_xid value of apply worker and ideally would be better than slot's xmin because it could give update_deleted in fewer cases, however, we can't use that because of reasons mentioned in the comments. Do you think this comment needs improvement for clarity and if so, do you have any proposal? -- With Regards, Amit Kapila.

Re: Dropping publication breaks logical replication

2025-08-01 Thread Amit Kapila
ication and let replication continue. So, we should consider backpatching this change but it would be better if few more people can share their opinion on this matter. [1] - https://www.postgresql.org/message-id/dc08add3-10a8-738b-983a-191c7406707b%40enterprisedb.com -- With Regards, Amit Kapila.

Re: 024_add_drop_pub.pl might fail due to deadlock

2025-08-01 Thread Amit Kapila
On Thu, Jul 31, 2025 at 4:22 PM Ajin Cherian wrote: > > I've fixed the patches accordingly for all branches. > Pushed. -- With Regards, Amit Kapila.

Re: 024_add_drop_pub.pl might fail due to deadlock

2025-07-31 Thread Amit Kapila
nId, subid, 0); > Assert(LockHeldByMe(&tag, AccessShareLock, true)); > #endif > > rel = table_open(SubscriptionRelRelationId, NoLock); > } > Your suggested change looks better to me. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-07-31 Thread Amit Kapila
r change for FindRecentlyDeletedTupleInfoSeq as well. Apart from above, please find a number of comment edits and other cosmetic changes in the attached. -- With Regards, Amit Kapila. diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 61d51939b55..1bf8fac7bea

Re: 024_add_drop_pub.pl might fail due to deadlock

2025-07-29 Thread Amit Kapila
On Tue, Jul 29, 2025 at 4:26 PM vignesh C wrote: > > On Tue, 29 Jul 2025 at 14:46, Ajin Cherian wrote: > > > > On Tue, Jul 29, 2025 at 1:13 PM Amit Kapila wrote: > > > > > > Yes, that makes sense to me. For HEAD and PG18, we can still add a new > > >

Re: [Patch] add new parameter to pg_replication_origin_session_setup

2025-07-28 Thread Amit Kapila
On Tue, Jul 29, 2025 at 2:43 AM Doruk Yilmaz wrote: > > On Mon, Mar 3, 2025 at 6:39 AM Amit Kapila wrote: > > > > To use replication_origin by multiple processes, one must maintain the > > commit order as we do internally by allowing the leader process to > > w

Re: 024_add_drop_pub.pl might fail due to deadlock

2025-07-28 Thread Amit Kapila
assertion check when the assertion is enabled in this build. > Thought? > Yes, that makes sense to me. For HEAD and PG18, we can still add a new argument to the API. For other bank branches, it is better to use a new Ex function as suggested by Kuroda-San. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-07-28 Thread Amit Kapila
n't do any such thing in RelationFindReplTupleSeq(), so, if we do something differently here, it should be explained in the comments. -- With Regards, Amit Kapila.

Re: Enhance pg_createsubscriber to create required standby.

2025-07-25 Thread Amit Kapila
On Wed, Jun 18, 2025 at 10:43 AM David G. Johnston wrote: > > On Tue, Jun 17, 2025 at 9:22 PM Amit Kapila wrote: >> >> >> As shown in Vignesh's email [1] (point 4), there could be multiple >> additional parameters required for the first option suggested b

Re: Conflict detection for update_deleted in logical replication

2025-07-25 Thread Amit Kapila
he may not be interested in eventual consistency as some of the data may not be replicated, so a conflict detection followed by any resolution may not be helpful. The other point is that if we report update_delete in such cases, it won't be reliable, sometimes it can be update_missing as vacuum would have removed the row, OTOH, if we report update_missing, it will always be the same conflict, and we can document it. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-07-24 Thread Amit Kapila
his can happen then still in most cases the index scan will happen. -- With Regards, Amit Kapila.

Re: 024_add_drop_pub.pl might fail due to deadlock

2025-07-23 Thread Amit Kapila
w about > "lock_shared_object"? > I think if we lock in a caller, we don't need to use any lock during table_open. We can use the parameter name as already_locked as we do at some other places in the code. -- With Regards, Amit Kapila.

Re: 024_add_drop_pub.pl might fail due to deadlock

2025-07-22 Thread Amit Kapila
cessExclusiveLock on SubscriptionRelRelationId in DropSubscription. Kindly test again after fixing the first comment above. -- With Regards, Amit Kapila.

Re: Question on any plans to use the User Server/User Mapping to provide Logical Replication Subscriptions the user/password in an encrypted manner

2025-07-22 Thread Amit Kapila
e13de9333f2c09.camel%40j-davis.com -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-07-22 Thread Amit Kapila
the ideas for future optimization and or special notes for some not so obvious design decisions. -- With Regards, Amit Kapila.

Re: Question on any plans to use the User Server/User Mapping to provide Logical Replication Subscriptions the user/password in an encrypted manner

2025-07-22 Thread Amit Kapila
totally hidden from prying eyes. > Can you check the work being discussed in thread [1] and see if that addresses your requirement? [1] - https://www.postgresql.org/message-id/149ff9264db27cdf724b65709fbbaee4bf316835.camel%40j-davis.com -- With Regards, Amit Kapila.

Re: Issues with hash and GiST LP_DEAD setting for kill_prior_tuple

2025-07-22 Thread Amit Kapila
e of the interlocking provided by vacuum scan. See comments atop _hash_kill_items() [There are never any scans active in a bucket at the time VACUUM begins ...]. I think we only need to worry about vacuum processing the page after we have collected the killed_items which shouldn't happen as explained in comments atop _hash_kill_items(). -- With Regards, Amit Kapila.

Re: Update Examples in Logical Replication Docs

2025-07-21 Thread Amit Kapila
lumns' was added in "\dRp+" by commit [3], so I think we should > update the docs to reflect the same. I have attached a patch with the > changes. > The changes apply to both the branches REL_18_STABLE and HEAD. > Thanks for the report and patch. I'll take care of it. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-07-21 Thread Amit Kapila
On Mon, Jul 21, 2025 at 11:27 PM Masahiko Sawada wrote: > > On Sun, Jul 20, 2025 at 9:00 PM Amit Kapila wrote: > > > > > If so, I agree > > with you, we don't need XIDs of other databases as logical WALSender > > will anyway won't process transaction

Re: Logical Replication of sequences

2025-07-21 Thread Amit Kapila
hould we do anything more here? > It would be better if we can give ERROR for options that are not specific to sequences. -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-07-20 Thread Amit Kapila
There could be multiple ways to preserve the slot, one is users always create a special slot on the primary for this purpose or we can provide a slot_option which users can specify/alter so that they get ERROR/WARNING on the last such slot being dropped. I feel we should choose the simplest option and rely on users to use the feature appropriately. We can always enhance the feature in future versions based on feedback from the field. -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-07-20 Thread Amit Kapila
On Mon, Jul 21, 2025 at 10:48 AM shveta malik wrote: > > On Fri, Jul 18, 2025 at 3:03 PM Amit Kapila wrote: > > > > > > > > One concern I have is regarding the default setting of > > > 'force_slot_drop' . I assume the default value of this new D

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-07-20 Thread Amit Kapila
On Mon, Jul 21, 2025 at 10:08 AM shveta malik wrote: > > On Sat, Jul 19, 2025 at 5:10 PM Amit Kapila wrote: > > > > On Fri, Jul 18, 2025 at 11:31 AM Dilip Kumar wrote: > > > > > > On Fri, Jul 18, 2025 at 11:25 AM shveta malik > > > wrote: > &g

Re: Document slot's restart_lsn can go backward

2025-07-20 Thread Amit Kapila
uroda proposed a patch [5], which improves > comments by describing this behavior. As proposed by Amit Kapila [6], > I'm starting a new thread about this comments patch. > > Any feedback on this patch? > I have tried to slightly improve the comments. You can include the suggesti

Re: Conflict detection for update_deleted in logical replication

2025-07-20 Thread Amit Kapila
On Sat, Jul 19, 2025 at 10:32 AM Amit Kapila wrote: > > On Sat, Jul 19, 2025 at 3:01 AM Masahiko Sawada wrote: > > > > On Fri, Jul 18, 2025 at 5:03 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > Here are some review comments and questions:

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-07-19 Thread Amit Kapila
romotion or other reason like timeout. However, at this stage it is not very clear whether it will be useful to provide additional timeout parameter. But we can consider retruning true/false depending on whether we are successful in syncing the slots or not. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-07-18 Thread Amit Kapila
or_conflict > Among these, the first option is better but I think it is better to name it just retain_dead_tuples. The explanation of the option will explain its use. It is similar to other options like binary or streaming. We are not naming them like request_data_binary_format to make the meaning apparent. There is a value in keeping names succinct. -- With Regards, Amit Kapila.

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-07-18 Thread Amit Kapila
dea to add comments in this area. But as this is for pre-existing cases, won't it be better to start a new thread explaining the cases and a patch? We may get feedback from others as well. -- With Regards, Amit Kapila.

Re: Logical Replication of sequences

2025-07-18 Thread Amit Kapila
)); [1]: https://www.postgresql.org/message-id/CAA4eK1%2B6L%2BAoGS3LHdnYnCE%3DnRHergSQyhyO7Y%3D-sOp7isGVMw%40mail.gmail.com [2]: https://www.postgresql.org/message-id/CABdArM52CSDuYsfTAEp4ZSWe%2BGFBvxgnPFgkG%2Bid9T88DUE%2B1Q%40mail.gmail.com -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-07-18 Thread Amit Kapila
On Fri, Jul 18, 2025 at 2:27 PM shveta malik wrote: > > On Thu, Jul 17, 2025 at 3:06 PM Amit Kapila wrote: > > > > On Wed, Jun 18, 2025 at 3:23 PM shveta malik wrote: > > > > > > On Wed, Jun 18, 2025 at 2:39 PM Bertrand Drouvot > > > wrote: >

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-07-17 Thread Amit Kapila
On Fri, Jul 18, 2025 at 12:26 AM Masahiko Sawada wrote: > > On Thu, Jul 17, 2025 at 2:36 AM Amit Kapila wrote: > > > > What if we make DROP SUBSCRIPTION fail if it can lead to removal of > > the last slot on publisher and allow DROP to succeed when the > > subscri

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-07-17 Thread Amit Kapila
On Thu, Jul 17, 2025 at 3:06 PM Amit Kapila wrote: > > On Wed, Jun 18, 2025 at 3:23 PM shveta malik wrote: > > > > On Wed, Jun 18, 2025 at 2:39 PM Bertrand Drouvot > > wrote: > > > > > > Hi, > > > > > > On Tue, Jun 10, 2025 at 02:

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-07-17 Thread Amit Kapila
iption option) is set? Now, users can still be allowed to Drop the subscription, if it disassociates the subscription from the slot by using method explained in docs [1] (See Notes section). Similarly when a user is trying to drop the last logical slot via pg_drop_replication_slot, we will allow it only with the force option. This should ensure that the user is aware of the consequences of dropping the last slot. -- With Regards, Amit Kapila.

Re: 024_add_drop_pub.pl might fail due to deadlock

2025-07-16 Thread Amit Kapila
usiveLock on ReplicationOriginRelationId via replorigin_drop_by_name() and then RowExclusiveLock on SubscriptionRelRelationId via UpdateSubscriptionRelState(). Won't such a scenario taking conflicting locks in reverse order can lead to deadlock at least in PG15? -- With Regards, Amit Kapila.

Re: Logical replication prefetch

2025-07-15 Thread Amit Kapila
On Mon, Jul 14, 2025 at 3:13 PM Amit Kapila wrote: > > On Sun, Jul 13, 2025 at 6:06 PM Konstantin Knizhnik > wrote: > > > > On 13/07/2025 1:28 pm, Amit Kapila wrote: > > > On Tue, Jul 8, 2025 at 12:06 PM Konstantin Knizhnik > > > wrote: > > >>

Re: failover logical replication slots

2025-07-15 Thread Amit Kapila
ll slots on new standby after failover to avoid excessive resource usage. -- With Regards, Amit Kapila.

Re: Report bytes and transactions actually sent downtream

2025-07-14 Thread Amit Kapila
On Mon, Jul 14, 2025 at 10:55 AM Ashutosh Bapat wrote: > > On Sun, Jul 13, 2025 at 4:34 PM Amit Kapila wrote: > > > > > > I think we don't want to make it mandatory for plugins to implement > > these stats, so instead of throwing ERROR, the view should show

Re: Logical replication prefetch

2025-07-14 Thread Amit Kapila
and try to come up with a good way to perform pre-fetch or parallelization of short transactions. -- With Regards, Amit Kapila.

Re: Logical replication prefetch

2025-07-14 Thread Amit Kapila
On Sun, Jul 13, 2025 at 6:06 PM Konstantin Knizhnik wrote: > > On 13/07/2025 1:28 pm, Amit Kapila wrote: > > On Tue, Jul 8, 2025 at 12:06 PM Konstantin Knizhnik > > wrote: > >> There is well known Postgres problem that logical replication subscriber > >> c

Re: failover logical replication slots

2025-07-13 Thread Amit Kapila
n implementing that API as well. Note: I suggest starting a new thread with the concrete proposal for the new API or GUC, stating how it will be helpful. It might help in getting suggestions from others as well. -- With Regards, Amit Kapila.

Re: Report bytes and transactions actually sent downtream

2025-07-13 Thread Amit Kapila
On Tue, Jul 1, 2025 at 7:35 PM Ashutosh Bapat wrote: > > On Tue, Jul 1, 2025 at 4:23 PM Amit Kapila wrote: > > > > On Mon, Jun 30, 2025 at 3:24 PM Ashutosh Bapat > > wrote: > > > > > > Hi All, > > > In a recent logical replication issue, t

Re: Logical replication prefetch

2025-07-13 Thread Amit Kapila
e pub-sub pairs by using row filters. -- With Regards, Amit Kapila.

Re: Logical replication prefetch

2025-07-12 Thread Amit Kapila
On Fri, Jul 11, 2025 at 7:49 PM Konstantin Knizhnik wrote: > > On 08/07/2025 2:51 pm, Amit Kapila wrote: > > On Tue, Jul 8, 2025 at 12:06 PM Konstantin Knizhnik > > wrote: > >> There is well known Postgres problem that logical replication subscriber > >> c

Re: Conflict detection for update_deleted in logical replication

2025-07-11 Thread Amit Kapila
On Thu, Jul 10, 2025 at 6:46 PM Masahiko Sawada wrote: > > On Wed, Jul 9, 2025 at 9:09 PM Amit Kapila wrote: > > > > > > I think that even with retain_conflict_info = off, there is probably a > > > point at which the subscriber can no longer keep up with the >

  1   2   3   4   5   6   7   8   9   10   >