Re: Conflict detection and logging in logical replication

2024-08-02 Thread Nisha Moond
Performance tests done on the v8-0001 and v8-0002 patches, available at [1]. The purpose of the performance tests is to measure the impact on logical replication with track_commit_timestamp enabled, as this involves fetching the commit_ts data to determine delete_differ/update_differ conflicts.

Re: Conflict detection and logging in logical replication

2024-08-02 Thread Amit Kapila
On Thu, Aug 1, 2024 at 5:23 PM Amit Kapila wrote: > > On Thu, Aug 1, 2024 at 2:26 PM Hayato Kuroda (Fujitsu) > wrote: > > > > 04. general > > > > According to the documentation [1], there is another constraint "exclude", > > which > > can cause another type of conflict. But this pattern cannot

Re: Conflict detection and logging in logical replication

2024-08-01 Thread Amit Kapila
On Thu, Aug 1, 2024 at 2:26 PM Hayato Kuroda (Fujitsu) wrote: > > 04. general > > According to the documentation [1], there is another constraint "exclude", > which > can cause another type of conflict. But this pattern cannot be logged in > detail. > As per docs, "exclusion constraints can

RE: Conflict detection and logging in logical replication

2024-08-01 Thread Hayato Kuroda (Fujitsu)
Dear Hou, Let me contribute the great feature. I read only the 0001 patch and here are initial comments. 01. logical-replication.sgml track_commit_timestamp must be specified only on the subscriber, but it is not clarified. Can you write down that? 02. logical-replication.sgml I felt that

RE: Conflict detection and logging in logical replication

2024-07-31 Thread Zhijie Hou (Fujitsu)
On Wednesday, July 31, 2024 1:36 PM shveta malik wrote: > > On Wed, Jul 31, 2024 at 7:40 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > > 2) > > > apply_handle_delete_internal() > > > > > > --Do we need to check "(!edata->mtstate || > > > edata->mtstate->operation != CMD_UPDATE)" in the

Re: Conflict detection and logging in logical replication

2024-07-31 Thread Amit Kapila
On Wed, Jul 31, 2024 at 7:40 AM Zhijie Hou (Fujitsu) wrote: > > Here is the V8 patch set. It includes the following changes: > A few more comments: 1. I think in FindConflictTuple() the patch is locking the tuple so that after finding a conflict if there is a concurrent delete, it can retry to

Re: Conflict detection and logging in logical replication

2024-07-30 Thread shveta malik
On Wed, Jul 31, 2024 at 7:40 AM Zhijie Hou (Fujitsu) wrote: > > > > > 2) > > apply_handle_delete_internal() > > > > --Do we need to check "(!edata->mtstate || edata->mtstate->operation != > > CMD_UPDATE)" in the else part as well? Can there be a scenario where during > > update flow, it is trying

Re: Conflict detection and logging in logical replication

2024-07-30 Thread shveta malik
On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu) wrote: > > Here is the V7 patch set that addressed all the comments so far[1][2][3]. Thanks for the patch, few comments: 1) build_index_value_desc() /* Assume the index has been locked */ indexDesc = index_open(indexoid,

Re: Conflict detection and logging in logical replication

2024-07-30 Thread Dilip Kumar
On Tue, Jul 30, 2024 at 1:49 PM Zhijie Hou (Fujitsu) wrote: > > > On Monday, July 29, 2024 6:59 PM Dilip Kumar > > wrote: > > > > > > On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > > > I was going through v7-0001, and I have some initial comments. > > > >

RE: Conflict detection and logging in logical replication

2024-07-30 Thread Zhijie Hou (Fujitsu)
> On Monday, July 29, 2024 6:59 PM Dilip Kumar > wrote: > > > > On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > I was going through v7-0001, and I have some initial comments. > > Thanks for the comments ! > > > > > @@ -536,11 +542,9 @@

RE: Conflict detection and logging in logical replication

2024-07-29 Thread Zhijie Hou (Fujitsu)
On Monday, July 29, 2024 6:59 PM Dilip Kumar wrote: > > On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu) > wrote: > > > > I was going through v7-0001, and I have some initial comments. Thanks for the comments ! > > @@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo >

Re: Conflict detection and logging in logical replication

2024-07-29 Thread shveta malik
On Mon, Jul 29, 2024 at 9:31 AM Amit Kapila wrote: > > On Fri, Jul 26, 2024 at 4:28 PM shveta malik wrote: > > > > On Fri, Jul 26, 2024 at 3:56 PM Amit Kapila wrote: > > > > > > > > One more thing we need to consider is whether we should LOG or ERROR > > > for update/delete_differ conflicts. If

Re: Conflict detection and logging in logical replication

2024-07-29 Thread Dilip Kumar
On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu) wrote: > I was going through v7-0001, and I have some initial comments. @@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot, ExprContext *econtext; Datum values[INDEX_MAX_KEYS]; bool

Re: Conflict detection and logging in logical replication

2024-07-29 Thread Amit Kapila
On Mon, Jul 29, 2024 at 11:44 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, July 26, 2024 7:34 PM Amit Kapila wrote: > > > > On Thu, Jul 25, 2024 at 4:12 PM Amit Kapila > > wrote: > > > > > > A few more comments: > > Thanks for the comments. > > > 1. > > For duplicate key, the patch reports

Re: Conflict detection and logging in logical replication

2024-07-28 Thread Amit Kapila
On Fri, Jul 26, 2024 at 4:28 PM shveta malik wrote: > > On Fri, Jul 26, 2024 at 3:56 PM Amit Kapila wrote: > > > > > One more thing we need to consider is whether we should LOG or ERROR > > for update/delete_differ conflicts. If we LOG as the patch is doing > > then we are intentionally

Re: Conflict detection and logging in logical replication

2024-07-26 Thread Amit Kapila
On Thu, Jul 25, 2024 at 4:12 PM Amit Kapila wrote: > > On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu) > wrote: > > > > Here is the V6 patch set which addressed Shveta and Nisha's comments > > in [1][2][3][4]. > > > > Do we need an option detect_conflict for logging conflicts? The >

Re: Conflict detection and logging in logical replication

2024-07-26 Thread shveta malik
On Fri, Jul 26, 2024 at 3:56 PM Amit Kapila wrote: > > On Fri, Jul 26, 2024 at 3:37 PM shveta malik wrote: > > > > On Fri, Jul 26, 2024 at 3:03 PM Nisha Moond > > wrote: > > > > > > On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu) > > > wrote: > > > > Here is the V6 patch set which

Re: Conflict detection and logging in logical replication

2024-07-26 Thread Amit Kapila
On Fri, Jul 26, 2024 at 3:37 PM shveta malik wrote: > > On Fri, Jul 26, 2024 at 3:03 PM Nisha Moond wrote: > > > > On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu) > > wrote: > > > Here is the V6 patch set which addressed Shveta and Nisha's comments > > > in [1][2][3][4]. > > > > Thanks

Re: Conflict detection and logging in logical replication

2024-07-26 Thread shveta malik
On Fri, Jul 26, 2024 at 3:03 PM Nisha Moond wrote: > > On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu) > wrote: > > Here is the V6 patch set which addressed Shveta and Nisha's comments > > in [1][2][3][4]. > > Thanks for the patch. > I tested the v6-0001 patch with partition table

Re: Conflict detection and logging in logical replication

2024-07-26 Thread Nisha Moond
On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu) wrote: > Here is the V6 patch set which addressed Shveta and Nisha's comments > in [1][2][3][4]. Thanks for the patch. I tested the v6-0001 patch with partition table scenarios. Please review the following scenario where Pub updates a tuple,

Re: Conflict detection and logging in logical replication

2024-07-26 Thread Amit Kapila
On Fri, Jul 26, 2024 at 9:39 AM shveta malik wrote: > > On Thu, Jul 11, 2024 at 7:47 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wednesday, July 10, 2024 5:39 PM shveta malik > > wrote: > > > > > > > 2) > > > Another case which might confuse user: > > > > > > CREATE TABLE t1 (pk integer

Re: Conflict detection and logging in logical replication

2024-07-26 Thread shveta malik
On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, July 22, 2024 5:03 PM shveta malik wrote: > > > > On Fri, Jul 19, 2024 at 2:06 PM shveta malik wrote: > > > > > > On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > Attach the V5 patch

Re: Conflict detection and logging in logical replication

2024-07-25 Thread shveta malik
On Thu, Jul 11, 2024 at 7:47 AM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, July 10, 2024 5:39 PM shveta malik > wrote: > > > > 2) > > Another case which might confuse user: > > > > CREATE TABLE t1 (pk integer primary key, val1 integer, val2 integer); > > > > On PUB: insert into t1

Re: Conflict detection and logging in logical replication

2024-07-25 Thread Amit Kapila
On Thu, Jul 25, 2024 at 12:04 PM Zhijie Hou (Fujitsu) wrote: > > Here is the V6 patch set which addressed Shveta and Nisha's comments > in [1][2][3][4]. > Do we need an option detect_conflict for logging conflicts? The possible reason to include such an option is to avoid any overhead during

Re: Conflict detection and logging in logical replication

2024-07-23 Thread Nisha Moond
On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu) wrote: > > Attach the V5 patch set which changed the following: > Tested v5-0001 patch, and it fails to detect the update_exists conflict for a setup where Pub has a non-partitioned table and Sub has the same table partitioned. Below is a

Re: Conflict detection and logging in logical replication

2024-07-22 Thread shveta malik
On Fri, Jul 19, 2024 at 2:06 PM shveta malik wrote: > > On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu) > wrote: > > > > Attach the V5 patch set which changed the following. > Please find last batch of comments on v5: patch001: 1) create subscription sub1 ... (detect_conflict=true); I

Re: Conflict detection and logging in logical replication

2024-07-19 Thread shveta malik
On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu) wrote: > > Attach the V5 patch set which changed the following. Thanks for the patch. Tested that previous reported issues are fixed. Please have a look at below scenario, I was expecting it to raise 'update_differ' but it raised both

Re: Conflict detection and logging in logical replication

2024-07-17 Thread shveta malik
On Thu, Jul 18, 2024 at 7:52 AM Zhijie Hou (Fujitsu) wrote: > > On Thursday, July 11, 2024 1:03 PM shveta malik > wrote: > > Hi, > > Thanks for the comments! > > > > > I have one concern about how we deal with conflicts. As for insert_exists, > > we > > keep on erroring out while raising

Re: Conflict detection and logging in logical replication

2024-07-11 Thread shveta malik
On Wed, Jul 10, 2024 at 3:09 PM shveta malik wrote: > > On Wed, Jul 3, 2024 at 8:31 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wednesday, June 26, 2024 10:58 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > Hi, > > > > As suggested by Sawada-san in another thread[1]. > > > > I am attaching

Re: Conflict detection and logging in logical replication

2024-07-10 Thread shveta malik
On Thu, Jul 11, 2024 at 7:47 AM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, July 10, 2024 5:39 PM shveta malik > wrote: > > > > On Wed, Jul 3, 2024 at 8:31 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Wednesday, June 26, 2024 10:58 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > > > >

RE: Conflict detection and logging in logical replication

2024-07-10 Thread Zhijie Hou (Fujitsu)
On Wednesday, July 10, 2024 5:39 PM shveta malik wrote: > > On Wed, Jul 3, 2024 at 8:31 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wednesday, June 26, 2024 10:58 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > > Hi, > > > > As suggested by Sawada-san in another thread[1]. > > > > I am attaching

Re: Conflict detection and logging in logical replication

2024-07-10 Thread shveta malik
On Wed, Jul 3, 2024 at 8:31 AM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, June 26, 2024 10:58 AM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > As suggested by Sawada-san in another thread[1]. > > I am attaching the V4 patch set which tracks the delete_differ > conflict in logical

Re: Conflict detection and logging in logical replication

2024-06-24 Thread Nisha Moond
On Mon, Jun 24, 2024 at 7:39 AM Zhijie Hou (Fujitsu) wrote: > > When testing the patch, I noticed a bug that when reporting the conflict > after calling ExecInsertIndexTuples(), we might find the tuple that we > just inserted and report it.(we should only report conflict if there are > other

Re: Conflict detection and logging in logical replication

2024-06-24 Thread shveta malik
On Mon, Jun 24, 2024 at 7:39 AM Zhijie Hou (Fujitsu) wrote: > > When testing the patch, I noticed a bug that when reporting the conflict > after calling ExecInsertIndexTuples(), we might find the tuple that we > just inserted and report it.(we should only report conflict if there are > other