Re: Conflict Detection and Resolution

2024-10-20 Thread shveta malik
On Fri, Oct 18, 2024 at 4:30 PM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, October 9, 2024 2:34 PM shveta malik > wrote: > > > > On Wed, Oct 9, 2024 at 8:58 AM shveta malik > > wrote: > > > > > > On Tue, Oct 8, 2024 at 3:12 PM Nisha Moond >

Re: Conflict Detection and Resolution

2024-10-08 Thread shveta malik
On Wed, Oct 9, 2024 at 8:58 AM shveta malik wrote: > > On Tue, Oct 8, 2024 at 3:12 PM Nisha Moond wrote: > > > Please find few comments on v14-patch004: patch004: 1) GetConflictResolver currently errors out when the resolver is last_update_wins and track_commit_timestamp is dis

Re: Conflict Detection and Resolution

2024-10-08 Thread shveta malik
On Tue, Oct 8, 2024 at 3:12 PM Nisha Moond wrote: > I have not started reviewing v15 yet, but here are few comments for v14-patch003: 1) In apply_handle_update_internal(), I see that FindReplTupleInLocalRel() used to lock the row to be updated in exclusive mode, but now since we are avoiding thi

Re: Make default subscription streaming option as Parallel

2024-10-08 Thread shveta malik
On Tue, Oct 8, 2024 at 3:38 PM Amit Kapila wrote: > > On Tue, Oct 8, 2024 at 2:25 PM shveta malik wrote: > > > > On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: > > > > > > > With parallel streaming as default, do you think there is

Re: Make default subscription streaming option as Parallel

2024-10-08 Thread shveta malik
On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: > With parallel streaming as default, do you think there is a need to increase the default for 'max_logical_replication_workers' as IIUC parallel workers are taken from the same pool. thanks Shveta

Re: Logical Replication of sequences

2024-10-04 Thread shveta malik
On Sun, Sep 29, 2024 at 12:34 PM vignesh C wrote: > > On Thu, 26 Sept 2024 at 11:07, shveta malik wrote: > > > > On Fri, Sep 20, 2024 at 9:36 AM vignesh C wrote: > > > > > > On Wed, 21 Aug 2024 at 11:54, vignesh C wrote: > > > > > >

Re: Conflict Detection and Resolution

2024-10-01 Thread shveta malik
On Tue, Oct 1, 2024 at 9:54 AM shveta malik wrote: > > On Tue, Oct 1, 2024 at 9:48 AM shveta malik wrote: > > > > I have started reviewing patch002, it is WIP, but please find initial > set of comments: > Please find second set of comments for patch002: 9) can_create_fu

Re: Conflict Detection and Resolution

2024-09-30 Thread shveta malik
On Tue, Oct 1, 2024 at 9:48 AM shveta malik wrote: > I have started reviewing patch002, it is WIP, but please find initial set of comments: 1. ExecSimpleRelationInsert(): + /* Check for conflict and return to caller for resolution if found */ + if (resolver != CR_ER

Re: Conflict Detection and Resolution

2024-09-30 Thread shveta malik
On Mon, Sep 30, 2024 at 2:55 PM Peter Smith wrote: > > On Mon, Sep 30, 2024 at 4:29 PM shveta malik wrote: > > > > On Mon, Sep 30, 2024 at 11:04 AM Peter Smith wrote: > > > > > > On Mon, Sep 30, 2024 at 2:27 PM shveta malik > > > wrote: > >

Re: Conflict Detection and Resolution

2024-09-30 Thread shveta malik
On Fri, Sep 27, 2024 at 2:33 PM shveta malik wrote: > > On Fri, Sep 27, 2024 at 10:44 AM shveta malik wrote: > > > > > > > > > > Thanks for the review. > > > > Here is the v14 patch-set fixing review comments in [1] and [2]. > > > > >

Re: Conflict Detection and Resolution

2024-09-29 Thread shveta malik
On Mon, Sep 30, 2024 at 11:04 AM Peter Smith wrote: > > On Mon, Sep 30, 2024 at 2:27 PM shveta malik wrote: > > > > On Fri, Sep 27, 2024 at 1:00 PM Peter Smith wrote: > ... > > > > > > 13. General - ordering of conflict_type. > > > > > >

Re: Conflict Detection and Resolution

2024-09-29 Thread shveta malik
On Fri, Sep 27, 2024 at 1:00 PM Peter Smith wrote: > > Here are some review comments for v14-0001. > > This is a WIP, but here are my comments for all the SGML parts. > > (There will be some overlap here with comments already posted by Shveta) > > == > 1. file modes after applying the patch >

Re: Conflict Detection and Resolution

2024-09-27 Thread shveta malik
On Fri, Sep 27, 2024 at 10:44 AM shveta malik wrote: > > > > > > > Thanks for the review. > > > Here is the v14 patch-set fixing review comments in [1] and [2]. > > > > > > > Thanks for the patches. I am reviewing patch001, it is WIP, but ple

Re: Conflict Detection and Resolution

2024-09-26 Thread shveta malik
On Thu, Sep 26, 2024 at 2:57 PM shveta malik wrote: > > On Fri, Sep 20, 2024 at 8:40 AM Nisha Moond wrote: > > > > Thanks for the review. > > Here is the v14 patch-set fixing review comments in [1] and [2]. > > > > Thanks for the patches. I am reviewing pa

Re: Conflict Detection and Resolution

2024-09-26 Thread shveta malik
On Fri, Sep 20, 2024 at 8:40 AM Nisha Moond wrote: > > Thanks for the review. > Here is the v14 patch-set fixing review comments in [1] and [2]. > Thanks for the patches. I am reviewing patch001, it is WIP, but please find initial set of comments: 1) Please see these 2 errors: postgres=# create

Re: Logical Replication of sequences

2024-09-25 Thread shveta malik
On Fri, Sep 20, 2024 at 9:36 AM vignesh C wrote: > > On Wed, 21 Aug 2024 at 11:54, vignesh C wrote: > > > > On Wed, 21 Aug 2024 at 08:33, Peter Smith wrote: > > > > > > Hi Vignesh, Here are my only review comments for the latest patch set. > > > > Thanks, these issues have been addressed in the

Re: Add contrib/pg_logicalsnapinspect

2024-09-25 Thread shveta malik
On Wed, Sep 25, 2024 at 11:01 PM Bertrand Drouvot wrote: > > Hi, > > On Wed, Sep 25, 2024 at 11:23:17AM +0530, shveta malik wrote: > > + OUT catchange_xip xid[] > > > > One question, what is xid datatype, is it too int8? Sorry, could not > > find the correct

Re: Add contrib/pg_logicalsnapinspect

2024-09-24 Thread shveta malik
On Tue, Sep 24, 2024 at 10:23 PM Bertrand Drouvot wrote: > > Hi, > > On Tue, Sep 24, 2024 at 09:15:31AM +0530, shveta malik wrote: > > On Fri, Sep 20, 2024 at 12:22 PM Bertrand Drouvot > > wrote: > > > > > > > > > Please find attached v8, that:

Re: Add contrib/pg_logicalsnapinspect

2024-09-23 Thread shveta malik
On Mon, Sep 23, 2024 at 7:57 AM Peter Smith wrote: > > My review comments for v8-0001 > > == > contrib/pg_logicalinspect/pg_logicalinspect.c > > 1. > +/* > + * Lookup table for SnapBuildState. > + */ > + > +#define SNAPBUILD_STATE_INCR 1 > + > +static const char *const SnapBuildStateDesc[] = {

Re: Add contrib/pg_logicalsnapinspect

2024-09-23 Thread shveta malik
On Fri, Sep 20, 2024 at 12:22 PM Bertrand Drouvot wrote: > > > Please find attached v8, that: > Thank You for the patch. In one of my tests, I noticed that I got negative checksum: postgres=# SELECT * FROM pg_get_logical_snapshot_meta('0/3481F20'); magic| checksum | version

Re: Allow logical failover slots to wait on synchronous replication

2024-09-20 Thread shveta malik
On Thu, Sep 19, 2024 at 12:02 PM Amit Kapila wrote: > > On Tue, Sep 17, 2024 at 9:08 AM shveta malik wrote: > > > > On Mon, Sep 16, 2024 at 4:04 PM Amit Kapila wrote: > > > > > > On Mon, Sep 16, 2024 at 2:55 PM shveta malik > > > wrote: > >

Re: Conflict Detection and Resolution

2024-09-19 Thread shveta malik
On Thu, Sep 19, 2024 at 5:43 PM vignesh C wrote: > > > > > I was reviewing the CONFLICT RESOLVER (insert_exists='apply_remote') > and found that one conflict remains unresolved in the following > scenario: Thanks for the review and testing. > Pub: > CREATE TABLE circles(c1 CIRCLE, c2 text, EXCLU

Re: Add contrib/pg_logicalsnapinspect

2024-09-18 Thread shveta malik
On Thu, Sep 19, 2024 at 12:17 AM Bertrand Drouvot wrote: > > Hi, > > Thanks for the review! > Thanks for the patch. Should we include in the document who can execute these functions and the required access permissions, similar to how it's done for pgwalinspect, pg_ls_logicalmapdir(), and other s

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-18 Thread shveta malik
On Wed, Sep 18, 2024 at 3:31 PM shveta malik wrote: > > On Wed, Sep 18, 2024 at 2:49 PM shveta malik wrote: > > > > > > Please find the attached v46 patch having changes for the above review > > > > comments and your test review comments and Shveta's re

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-18 Thread shveta malik
On Wed, Sep 18, 2024 at 2:49 PM shveta malik wrote: > > > > Please find the attached v46 patch having changes for the above review > > > comments and your test review comments and Shveta's review comments. > > > When the synced slot is marked as 'inactive

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-18 Thread shveta malik
On Wed, Sep 18, 2024 at 12:21 PM shveta malik wrote: > > On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy > wrote: > > > > Hi, > > > > > > Please find the attached v46 patch having changes for the above review > > comments and your test

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-17 Thread shveta malik
On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy wrote: > > Hi, > > > Please find the attached v46 patch having changes for the above review > comments and your test review comments and Shveta's review comments. > Thanks for addressing comments. Is there a reason that we don't support this inva

Re: Add contrib/pg_logicalsnapinspect

2024-09-17 Thread shveta malik
On Tue, Sep 17, 2024 at 12:44 PM Bertrand Drouvot wrote: > > Hi, > > On Tue, Sep 17, 2024 at 10:18:35AM +0530, shveta malik wrote: > > Thanks for addressing the comments. I have not started reviewing v4 > > yet, but here are few more comments on v3: > > > >

Re: Add contrib/pg_logicalsnapinspect

2024-09-17 Thread shveta malik
On Tue, Sep 17, 2024 at 10:46 AM David G. Johnston wrote: > > > > On Monday, September 16, 2024, shveta malik wrote: >> >> On Tue, Sep 17, 2024 at 10:18 AM shveta malik wrote: >> > >> > Thanks for addressing the comments. I have not started reviewing v

Re: Add contrib/pg_logicalsnapinspect

2024-09-16 Thread shveta malik
On Tue, Sep 17, 2024 at 10:18 AM shveta malik wrote: > > Thanks for addressing the comments. I have not started reviewing v4 > yet, but here are few more comments on v3: > I just noticed that when we pass NULL input, both the new functions give 1 row as output, all cols as NULL: new

Re: Add contrib/pg_logicalsnapinspect

2024-09-16 Thread shveta malik
On Mon, Sep 16, 2024 at 8:03 PM Bertrand Drouvot wrote: > > Hi, > > On Mon, Sep 16, 2024 at 04:02:51PM +0530, shveta malik wrote: > > On Wed, Sep 11, 2024 at 4:21 PM Bertrand Drouvot > > wrote: > > > > > > > > > Yeah, good idea. Done that way

Re: Allow logical failover slots to wait on synchronous replication

2024-09-16 Thread shveta malik
On Mon, Sep 16, 2024 at 4:04 PM Amit Kapila wrote: > > On Mon, Sep 16, 2024 at 2:55 PM shveta malik wrote: > > > > On Mon, Sep 16, 2024 at 11:13 AM Amit Kapila > > wrote: > > > > > > > > Another question aside from the above point, what if so

Re: Add contrib/pg_logicalsnapinspect

2024-09-16 Thread shveta malik
On Wed, Sep 11, 2024 at 4:21 PM Bertrand Drouvot wrote: > > > Yeah, good idea. Done that way in v3 attached. > Thanks for the patch. +1 on the patch's idea. I have started reviewing/testing it. It is WIP but please find few initial comments: src/backend/replication/logical/snapbuild.c: 1) + fsy

Re: Allow logical failover slots to wait on synchronous replication

2024-09-16 Thread shveta malik
On Mon, Sep 16, 2024 at 11:13 AM Amit Kapila wrote: > > On Fri, Sep 13, 2024 at 3:13 PM shveta malik wrote: > > > > On Thu, Sep 12, 2024 at 3:04 PM shveta malik wrote: > > > > > > On Wed, Sep 11, 2024 at 2:40 AM John H wrote: > > > > > > &g

Re: Allow logical failover slots to wait on synchronous replication

2024-09-13 Thread shveta malik
On Thu, Sep 12, 2024 at 3:04 PM shveta malik wrote: > > On Wed, Sep 11, 2024 at 2:40 AM John H wrote: > > > > Hi Shveta, > > > > On Sun, Sep 8, 2024 at 11:16 PM shveta malik wrote: > > > > > > > > I was trying to have a look at the patch a

Re: Conflict detection for update_deleted in logical replication

2024-09-13 Thread shveta malik
On Fri, Sep 13, 2024 at 11:38 AM Amit Kapila wrote: > > > > > > > So in brief, this solution is only for bidrectional setup? For > > > non-bidirectional, > > > feedback_slots is non-configurable and thus irrelevant. > > > > Right. > > > > One possible idea to address the non-bidirectional case ra

Re: Allow logical failover slots to wait on synchronous replication

2024-09-12 Thread shveta malik
On Wed, Sep 11, 2024 at 2:40 AM John H wrote: > > Hi Shveta, > > On Sun, Sep 8, 2024 at 11:16 PM shveta malik wrote: > > > > > I was trying to have a look at the patch again, it doesn't apply on > > the head, needs rebase. > > > > Reba

Re: Conflict detection for update_deleted in logical replication

2024-09-10 Thread shveta malik
On Wed, Sep 11, 2024 at 10:15 AM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, September 11, 2024 12:18 PM shveta malik > wrote: > > > > On Tue, Sep 10, 2024 at 4:30 PM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > On Tuesday,

Re: Conflict detection for update_deleted in logical replication

2024-09-10 Thread shveta malik
On Tue, Sep 10, 2024 at 4:30 PM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 10, 2024 5:56 PM shveta malik > wrote: > > > > On Tue, Sep 10, 2024 at 1:40 PM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > On Tuesday,

Re: Disallow altering invalidated replication slots

2024-09-10 Thread shveta malik
On Tue, Sep 10, 2024 at 11:24 PM Bharath Rupireddy wrote: > > > Please find the attached v2 patch also having Shveta's review comments > addressed. The v2 patch looks good to me. thanks Shveta

Re: Conflict detection for update_deleted in logical replication

2024-09-10 Thread shveta malik
On Tue, Sep 10, 2024 at 1:40 PM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 10, 2024 2:45 PM shveta malik > wrote: > > > --- > > > THE DESIGN > > > --- > > > > > > To achieve the above, we plan to allow the logical walsender to &

Re: Conflict detection for update_deleted in logical replication

2024-09-09 Thread shveta malik
On Thu, Sep 5, 2024 at 5:07 PM Zhijie Hou (Fujitsu) wrote: > > > Hi hackers, > > I am starting a new thread to discuss and propose the conflict detection for > update_deleted scenarios during logical replication. This conflict occurs when > the apply worker cannot find the target tuple to be updat

Re: Disallow altering invalidated replication slots

2024-09-09 Thread shveta malik
On Tue, Sep 10, 2024 at 12:11 AM Bharath Rupireddy wrote: > > Hi, > > ALTER_REPLICATION_SLOT on invalidated replication slots is unnecessary > as there is no way to get the invalidated (logical) slot to work. > Please find the patch to add an error in such cases. Relevant > discussion is at [1]. >

Re: Conflict Detection and Resolution

2024-09-09 Thread shveta malik
On Mon, Sep 9, 2024 at 2:58 PM shveta malik wrote: > > On Fri, Sep 6, 2024 at 2:05 PM Ajin Cherian wrote: > > > > > > Thank you for your feedback, Shveta. I've addressed both sets of comments > > you provided. > > Thanks for the patches. I am reviewing v

Re: Conflict Detection and Resolution

2024-09-09 Thread shveta malik
On Fri, Sep 6, 2024 at 2:05 PM Ajin Cherian wrote: > > > Thank you for your feedback, Shveta. I've addressed both sets of comments you > provided. Thanks for the patches. I am reviewing v12-patch001, it is WIP. But please find first set of comments: 1) src/sgml/logical-replication.sgml: + Users

Re: Allow logical failover slots to wait on synchronous replication

2024-09-08 Thread shveta malik
On Fri, Aug 30, 2024 at 12:56 AM John H wrote: > > Hi Shveta, > > Thanks for reviewing it so quickly. > > On Thu, Aug 29, 2024 at 2:35 AM shveta malik wrote: > > > > Thanks for the patch. Few comments and queries: > > > > 1) > > + static XLogRecP

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-08 Thread shveta malik
On Sun, Sep 8, 2024 at 5:25 PM Bharath Rupireddy wrote: > > > Please find the v45 patch. Addressed above and Shveta's review comments [1]. > Thanks for the patch. Please find my comments: 1) src/sgml/config.sgml: + Synced slots are always considered to be inactive because they don't perform lo

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-08 Thread shveta malik
On Mon, Sep 9, 2024 at 10:26 AM Bharath Rupireddy wrote: > > Hi, > > On Mon, Sep 9, 2024 at 9:17 AM shveta malik wrote: > > > > > We should not allow the invalid replication slot to be altered > > > irrespective of the reason unless there is any benefit. &

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-08 Thread shveta malik
On Thu, Sep 5, 2024 at 9:30 AM Amit Kapila wrote: > > On Wed, Sep 4, 2024 at 9:17 AM shveta malik wrote: > > > > On Tue, Sep 3, 2024 at 3:01 PM shveta malik wrote: > > > > > > > > > 1) > > > I see that ReplicationSlotAlter() will error out i

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-04 Thread shveta malik
On Wed, Sep 4, 2024 at 9:17 AM shveta malik wrote: > > On Tue, Sep 3, 2024 at 3:01 PM shveta malik wrote: > > > > 1) It is related to one of my previous comments (pt 3 in [1]) where I stated that inactive_since should not keep on changing once a slot is invalidated. Below is

Commit Timestamp and LSN Inversion issue

2024-09-03 Thread shveta malik
Hello hackers, (Cc people involved in the earlier discussion) I would like to discuss the $Subject. While discussing Logical Replication's Conflict Detection and Resolution (CDR) design in [1] , it came to our notice that the commit LSN and timestamp may not correlate perfectly i.e. commits may

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-03 Thread shveta malik
On Tue, Sep 3, 2024 at 3:01 PM shveta malik wrote: > > > 1) > I see that ReplicationSlotAlter() will error out if the slot is > invalidated due to timeout. I have not tested it myself, but do you > know if slot-alter errors out for other invalidation causes as well? > Just w

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-09-03 Thread shveta malik
On Sat, Aug 31, 2024 at 1:45 PM Bharath Rupireddy wrote: > > Hi, > > > Please find the attached v44 patch with the above changes. I will > include the 0002 xid_age based invalidation patch later. > Thanks for the patch Bharath. My review and testing is WIP, but please find few comments and querie

Re: DOCS - pg_replication_slot . Fix the 'inactive_since' description

2024-09-02 Thread shveta malik
On Tue, Sep 3, 2024 at 10:43 AM Amit Kapila wrote: > > > > > > > To summarize, the current description wrongly describes the field as a > > > time duration: > > > "The time since the slot has become inactive." > > > > > > I suggest replacing it with: > > > "The slot has been inactive since this ti

Re: DOCS - pg_replication_slot . Fix the 'inactive_since' description

2024-09-01 Thread shveta malik
On Mon, Sep 2, 2024 at 5:47 AM Peter Smith wrote: > > Hi hackers. While reviewing another thread I had cause to look at the > docs for the pg_replication_slot 'inactive_since' field [1] for the > first time. > > I was confused by the description, which is as follows: > > inactive_since timest

Re: Collect statistics about conflicts in logical replication

2024-09-01 Thread shveta malik
On Mon, Sep 2, 2024 at 4:20 AM Peter Smith wrote: > > On Fri, Aug 30, 2024 at 4:24 PM shveta malik wrote: > > > > On Fri, Aug 30, 2024 at 10:53 AM Peter Smith wrote: > > > > ... > > > 2. Arrange all the counts into an intuitive/natural order > > &

Re: Collect statistics about conflicts in logical replication

2024-08-30 Thread shveta malik
On Fri, Aug 30, 2024 at 12:15 PM Zhijie Hou (Fujitsu) wrote: > > > Here is V5 patch which addressed above and Shveta's[1] comments. > The patch looks good to me. thanks Shveta

Re: Conflict Detection and Resolution

2024-08-29 Thread shveta malik
On Fri, Aug 30, 2024 at 12:13 PM Amit Kapila wrote: > > On Wed, Aug 28, 2024 at 10:58 AM shveta malik wrote: > > > > On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian wrote: > > > > > >> 2) > > >> Currently pg_dump is dumping even the default reso

Re: Collect statistics about conflicts in logical replication

2024-08-29 Thread shveta malik
On Fri, Aug 30, 2024 at 10:53 AM Peter Smith wrote: > > Hi Hou-San. Here are my review comments for v4-0001. > > == > > 1. Add links in the docs > > IMO it would be good for all these confl_* descriptions (in > doc/src/sgml/monitoring.sgml) to include links back to where each of > those confli

Re: Collect statistics about conflicts in logical replication

2024-08-29 Thread shveta malik
On Fri, Aug 30, 2024 at 9:40 AM shveta malik wrote: > > On Thu, Aug 29, 2024 at 11:06 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > Agreed. Here is new version patch which change the names as suggested. I > > also > > rebased the patch based on another r

Re: Collect statistics about conflicts in logical replication

2024-08-29 Thread shveta malik
On Thu, Aug 29, 2024 at 11:06 AM Zhijie Hou (Fujitsu) wrote: > > > Agreed. Here is new version patch which change the names as suggested. I also > rebased the patch based on another renaming commit 640178c9. > Thanks for the patch. Few minor things: 1) conflict.h: * This enum is used in statist

Re: Allow logical failover slots to wait on synchronous replication

2024-08-29 Thread shveta malik
On Thu, Aug 29, 2024 at 2:31 AM John H wrote: > > Hi Amit, > > On Mon, Aug 26, 2024 at 11:00 PM Amit Kapila wrote: > > I wanted a simple test where in the first case you use > > synchronous_standby_names = 'ANY 3 (A,B,C,D,E)' and in the second case > > use standby_slot_names = A_slot, B_slot, C_s

Re: Conflict Detection and Resolution

2024-08-28 Thread shveta malik
On Wed, Aug 28, 2024 at 4:07 PM shveta malik wrote: > > > On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian wrote: > > > > > The review is WIP. Please find a few comments on patch001. > More comments on ptach001 in continuation of previous comments: 6) SetDefaultRes

Re: Collect statistics about conflicts in logical replication

2024-08-28 Thread shveta malik
On Thu, Aug 29, 2024 at 4:59 AM Peter Smith wrote: > > On Wed, Aug 28, 2024 at 9:19 PM Amit Kapila wrote: > > > > On Wed, Aug 28, 2024 at 11:43 AM shveta malik > > wrote: > > > > > > Thanks for the patch. Just thinking out loud, since w

Re: Conflict Detection and Resolution

2024-08-28 Thread shveta malik
> On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian wrote: > > The review is WIP. Please find a few comments on patch001. 1) logical-repliction.sgmlL + Additional logging is triggered for specific conflict_resolvers. Users can also configure conflict_types while creating the subscription. Refer to

Re: Collect statistics about conflicts in logical replication

2024-08-27 Thread shveta malik
On Tue, Aug 27, 2024 at 3:21 PM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, August 27, 2024 10:59 AM Peter Smith > wrote: > > > > ~~~ > > > > 3. > > +# Enable track_commit_timestamp to detect origin-differ conflicts in > > +logical # replication. Reduce wal_retrieve_retry_interval to 1ms to > >

Re: Conflict detection and logging in logical replication

2024-08-27 Thread shveta malik
On Wed, Aug 28, 2024 at 9:44 AM Zhijie Hou (Fujitsu) wrote: > > > > +1 on 'update_origin_differs' instead of 'update_origins_differ' as > > > the former is somewhat similar to other conflict names 'insert_exists' > > > and 'update_exists'. > > > > Since we reached a consensus on this, I am attachi

Re: Conflict Detection and Resolution

2024-08-27 Thread shveta malik
On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian wrote: > >> 2) >> Currently pg_dump is dumping even the default resolvers configuration. >> As an example if I have not changed default configuration for say >> sub1, it still dumps all: >> >> CREATE SUBSCRIPTION sub1 CONNECTION '..' PUBLICATION pub1 W

Re: Conflict Detection and Resolution

2024-08-27 Thread shveta malik
On Tue, Aug 27, 2024 at 1:51 PM Nisha Moond wrote: > > Please find v10 patch-set. Changes are: > > 1) patch-001: > - Corrected a patch application warning. > - Added support for pg_dump. > - As suggested in pt.4 of [1]: added a warning during CREATE and > ALTER subscription when track_commit_ti

Re: Conflict detection and logging in logical replication

2024-08-27 Thread shveta malik
On Tue, Aug 27, 2024 at 4:37 AM Peter Smith wrote: > > On Mon, Aug 26, 2024 at 7:52 PM Amit Kapila wrote: > > > > On Thu, Aug 22, 2024 at 2:21 PM Amit Kapila wrote: > > > > > > On Thu, Aug 22, 2024 at 1:33 PM Peter Smith wrote: > > > > > > > > Do you think the documentation for the 'column_valu

Re: Allow logical failover slots to wait on synchronous replication

2024-08-27 Thread shveta malik
On Tue, Aug 27, 2024 at 12:56 AM John H wrote: > > Hi Shveta, > > On Sun, Jul 21, 2024 at 8:42 PM shveta malik wrote: > > > > Ah that's a gap. Let me add some logging/warning in a similar fashion. > > > Although I think I'd have the warning be relativ

Re: Conflict detection and logging in logical replication

2024-08-26 Thread shveta malik
On Mon, Aug 26, 2024 at 3:22 PM Amit Kapila wrote: > > On Thu, Aug 22, 2024 at 2:21 PM Amit Kapila wrote: > > > > On Thu, Aug 22, 2024 at 1:33 PM Peter Smith wrote: > > > > > > Do you think the documentation for the 'column_value' parameter of the > > > conflict logging should say that the displ

Re: Conflict Detection and Resolution

2024-08-26 Thread shveta malik
On Mon, Aug 26, 2024 at 2:23 PM Amit Kapila wrote: > > On Thu, Aug 22, 2024 at 3:45 PM shveta malik wrote: > > > > On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond > > wrote: > > > > > > The patches have been rebased on the latest pgHead following the merg

Re: Conflict Detection and Resolution

2024-08-25 Thread shveta malik
On Mon, Aug 26, 2024 at 7:28 AM Peter Smith wrote: > > On Thu, Aug 22, 2024 at 8:15 PM shveta malik wrote: > > > > On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond > > wrote: > > > > > > The patches have been rebased on the latest pgHead following the merg

Re: Conflict Detection and Resolution

2024-08-25 Thread shveta malik
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond wrote: > > The patches have been rebased on the latest pgHead following the merge > of the conflict detection patch [1]. The detect_conflict option has > been removed, and conflict detection is now enabled by default. This > change required the following

Re: Conflict Detection and Resolution

2024-08-22 Thread shveta malik
On Thu, Aug 22, 2024 at 3:44 PM shveta malik wrote: > > > For clock-skew and timestamp based resolution, if needed, I will post > another email for the design items where suggestions are needed. > Please find issues which need some thoughts and approval for time-based resolution

Re: Conflict Detection and Resolution

2024-08-22 Thread shveta malik
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond wrote: > > The patches have been rebased on the latest pgHead following the merge > of the conflict detection patch [1]. Thanks for working on patches. Summarizing the issues which need some suggestions/thoughts. 1) For subscription based resolvers, c

Re: Conflict detection and logging in logical replication

2024-08-21 Thread shveta malik
On Wed, Aug 21, 2024 at 3:04 PM Zhijie Hou (Fujitsu) wrote: > > > Attach the V20 patch set which addressed above, Shveta[1][2] and > Kuroda-san's[3] > comments. > Thank You for the patch. Few comments: 1) + The key section includes the key values of the local tuple that violated a unique constr

Re: Conflict detection and logging in logical replication

2024-08-20 Thread shveta malik
On Tue, Aug 20, 2024 at 4:45 PM Zhijie Hou (Fujitsu) wrote: > > Here are the remaining patches. > > 0001 adds additional doc to explain the log format. > 0002 collects statistics about conflicts in logical replication. > 0002 has not changed since I last reviewed it. It seems all my old comments

Re: Conflict detection and logging in logical replication

2024-08-20 Thread shveta malik
On Tue, Aug 20, 2024 at 4:45 PM Zhijie Hou (Fujitsu) wrote: > > Here are the remaining patches. > > 0001 adds additional doc to explain the log format. Thanks for the patch. Please find few comments on 001: 1) +Key (column_name, ...)=(column_name, ...); existing local tuple (column_name, ...)=(c

Re: [bug fix] prepared transaction might be lost when max_prepared_transactions is zero on the subscriber

2024-08-20 Thread shveta malik
On Tue, Aug 20, 2024 at 11:36 AM Amit Kapila wrote: > > On Wed, Aug 14, 2024 at 10:26 AM shveta malik wrote: > > > > > > > > Thanks for the detailed analysis. I agree with your analysis that we > > > need to reset the origin information for the shutdow

Re: Conflict detection and logging in logical replication

2024-08-19 Thread shveta malik
On Mon, Aug 19, 2024 at 12:32 PM Zhijie Hou (Fujitsu) wrote: > > > Thanks for reporting the bug. I have fixed it and ran pgindent in V17 patch. > I also adjusted few comments and fixed a typo. > Thanks for the patch. Re-tested it, all scenarios seem to work well now. I see that this version has

Re: Conflict detection and logging in logical replication

2024-08-19 Thread shveta malik
On Mon, Aug 19, 2024 at 12:09 PM Amit Kapila wrote: > > On Mon, Aug 19, 2024 at 11:54 AM shveta malik wrote: > > > > On Mon, Aug 19, 2024 at 11:37 AM Amit Kapila > > wrote: > > > > > > On Mon, Aug 19, 2024 at 9:08 AM shveta malik > > > wro

Re: Conflict detection and logging in logical replication

2024-08-18 Thread shveta malik
On Mon, Aug 19, 2024 at 11:37 AM Amit Kapila wrote: > > On Mon, Aug 19, 2024 at 9:08 AM shveta malik wrote: > > > > On Sun, Aug 18, 2024 at 2:27 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Attach the V16 patch which addressed the comments we agreed o

Re: Conflict detection and logging in logical replication

2024-08-18 Thread shveta malik
On Mon, Aug 19, 2024 at 9:07 AM shveta malik wrote: > > On Sun, Aug 18, 2024 at 2:27 PM Zhijie Hou (Fujitsu) > wrote: > > > > Attach the V16 patch which addressed the comments we agreed on. > > I will add a doc patch to explain the log format after the 0001 is

Re: Conflict detection and logging in logical replication

2024-08-18 Thread shveta malik
On Sun, Aug 18, 2024 at 2:27 PM Zhijie Hou (Fujitsu) wrote: > > Attach the V16 patch which addressed the comments we agreed on. > I will add a doc patch to explain the log format after the 0001 is RFC. > Thank You for addressing comments. Please see this scenario: create table tab1(pk int primar

Re: Conflict detection and logging in logical replication

2024-08-16 Thread shveta malik
On Fri, Aug 16, 2024 at 12:19 PM Amit Kapila wrote: > > On Fri, Aug 16, 2024 at 11:48 AM shveta malik wrote: > > > > On Fri, Aug 16, 2024 at 10:46 AM shveta malik > > wrote: > > > > > > 3) > > > For update_exists(), we dump: >

Re: Conflict detection and logging in logical replication

2024-08-15 Thread shveta malik
On Fri, Aug 16, 2024 at 10:46 AM shveta malik wrote: > > 3) > For update_exists(), we dump: > Key (a, b)=(2, 1) > > For delete_missing, update_missing, update_differ, we dump: > Replica identity (a, b)=(2, 1). > > For update_exists as well, shouldn't we dump '

Re: Conflict detection and logging in logical replication

2024-08-15 Thread shveta malik
On Thu, Aug 15, 2024 at 12:47 PM Zhijie Hou (Fujitsu) wrote: > > Thanks. I have checked and merged the changes. Here is the V15 patch > which addressed above comments. Thanks for the patch. Please find few comments and queries: 1) For various conflicts , we have these in Logs: Replica identity (

Re: [bug fix] prepared transaction might be lost when max_prepared_transactions is zero on the subscriber

2024-08-14 Thread shveta malik
On Tue, Aug 13, 2024 at 9:48 AM Amit Kapila wrote: > > BTW, this needs to be backpatched till 16 when it has been introduced > by the parallel apply feature as part of commit 216a7848. So, can we > test this patch in back-branches as well? > I was able to reproduce the problem on REL_16_STABLE an

Re: [bug fix] prepared transaction might be lost when max_prepared_transactions is zero on the subscriber

2024-08-13 Thread shveta malik
On Tue, Aug 13, 2024 at 9:48 AM Amit Kapila wrote: > > On Mon, Aug 12, 2024 at 3:37 PM shveta malik wrote: > > > > On Fri, Aug 9, 2024 at 2:39 PM Hayato Kuroda (Fujitsu) > > wrote: > > > > > > > > > > + /* > > > > + *

Re: [bug fix] prepared transaction might be lost when max_prepared_transactions is zero on the subscriber

2024-08-12 Thread shveta malik
On Mon, Aug 12, 2024 at 3:36 PM shveta malik wrote: > > On Fri, Aug 9, 2024 at 2:39 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Dear Amit, Shveta, Hou, > > > > Thanks for giving many comments! I've updated the patch. > > > > > @@ -4409

Re: [bug fix] prepared transaction might be lost when max_prepared_transactions is zero on the subscriber

2024-08-12 Thread shveta malik
On Fri, Aug 9, 2024 at 2:39 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Amit, Shveta, Hou, > > Thanks for giving many comments! I've updated the patch. > > > @@ -4409,6 +4409,14 @@ start_apply(XLogRecPtr origin_startpos) > > } > > PG_CATCH(); > > { > > + /* > > + * Reset the origin data to pr

Re: Logical Replication of sequences

2024-08-08 Thread shveta malik
On Wed, Aug 7, 2024 at 2:00 PM vignesh C wrote: > > On Wed, 7 Aug 2024 at 08:09, Amit Kapila wrote: > > > > On Tue, Aug 6, 2024 at 5:13 PM vignesh C wrote: > > > > > > On Mon, 5 Aug 2024 at 18:05, shveta malik wrote: > > > > > >

Re: [bug fix] prepared transaction might be lost when max_prepared_transactions is zero on the subscriber

2024-08-08 Thread shveta malik
On Thu, Aug 8, 2024 at 6:08 PM Amit Kapila wrote: > > On Thu, Aug 8, 2024 at 3:41 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Thursday, August 8, 2024 6:01 PM shveta malik > > wrote: > > > > > > On Thu, Aug 8, 2024 at 12:03 PM Amit Kapila > >

Re: Found issues related with logical replication and 2PC

2024-08-08 Thread shveta malik
On Thu, Aug 8, 2024 at 5:53 PM Amit Kapila wrote: > > On Thu, Aug 8, 2024 at 2:37 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Thanks for discussing! > > > > I reported the issue because 1) I feared the risk of data loss and 2) simply > > because the coding looked incorrect. However, per discussio

Re: [bug fix] prepared transaction might be lost when max_prepared_transactions is zero on the subscriber

2024-08-08 Thread shveta malik
On Thu, Aug 8, 2024 at 12:03 PM Amit Kapila wrote: > > On Thu, Aug 8, 2024 at 10:37 AM Hayato Kuroda (Fujitsu) > wrote: > > > ... > > > > An easiest fix is to reset session replication origin before calling the > > RecordTransactionAbort(). I think this can happen when 1) > > LogicalRepApplyLoop

Re: Found issues related with logical replication and 2PC

2024-08-08 Thread shveta malik
On Thu, Aug 8, 2024 at 9:53 AM Amit Kapila wrote: > > On Thu, Aug 8, 2024 at 8:54 AM shveta malik wrote: > > > > On Wed, Aug 7, 2024 at 5:43 PM Amit Kapila wrote: > > > > > So, if my > > > analysis is correct, this shouldn't be a bug and i

Re: Logical Replication of sequences

2024-08-07 Thread shveta malik
On Wed, Aug 7, 2024 at 1:45 PM vignesh C wrote: > > > The remaining comments have been addressed, and the changes are > included in the attached v20240807 version patch. Thanks for addressing the comment. Please find few comments for v20240807 : patch002: 1) create_publication.sgml: --I think i

Re: Found issues related with logical replication and 2PC

2024-08-07 Thread shveta malik
On Wed, Aug 7, 2024 at 5:43 PM Amit Kapila wrote: > > On Wed, Aug 7, 2024 at 3:32 PM Amit Kapila wrote: > > > > I also think so. Additionally, I feel a test case (or some description > > of the bug that can arise) should be provided for issue-1. > > > > IIUC, the problem could be that we would en

  1   2   3   4   5   >