Re: Proposal: Conflict log history table for Logical Replication

2025-09-27 Thread Amit Kapila
On Sat, Sep 27, 2025 at 9:24 PM Dilip Kumar wrote: > > On Sat, Sep 27, 2025 at 8:53 PM Amit Kapila wrote: > > > > I am not able to understand what exact problem you are seeing here. I > > was thinking that during the CREATE SUBSCRIPTION command, a new table > > with user provided name will be cre

Re: Proposal: Conflict log history table for Logical Replication

2025-09-27 Thread Dilip Kumar
On Sat, Sep 27, 2025 at 8:53 PM Amit Kapila wrote: > > I am not able to understand what exact problem you are seeing here. I > was thinking that during the CREATE SUBSCRIPTION command, a new table > with user provided name will be created similar to how we create a > slot. The difference would be

Re: Proposal: Conflict log history table for Logical Replication

2025-09-27 Thread Amit Kapila
On Fri, Sep 26, 2025 at 4:42 PM Dilip Kumar wrote: > > On Thu, Sep 25, 2025 at 4:19 PM Dilip Kumar wrote: > > > > On Thu, Sep 25, 2025 at 11:53 AM Dilip Kumar wrote: > > > > > > > [1] > > > > /* > > > > * For logical decode we need combo CIDs to properly decode the > > > > * catalog > > > > */ >

Re: Proposal: Conflict log history table for Logical Replication

2025-09-26 Thread Dilip Kumar
On Thu, Sep 25, 2025 at 4:19 PM Dilip Kumar wrote: > > On Thu, Sep 25, 2025 at 11:53 AM Dilip Kumar wrote: > > > > > [1] > > > /* > > > * For logical decode we need combo CIDs to properly decode the > > > * catalog > > > */ > > > if (RelationIsAccessibleInLogicalDecoding(relation)) > > > log_heap

Re: Proposal: Conflict log history table for Logical Replication

2025-09-25 Thread Dilip Kumar
On Thu, Sep 25, 2025 at 11:53 AM Dilip Kumar wrote: > > > [1] > > /* > > * For logical decode we need combo CIDs to properly decode the > > * catalog > > */ > > if (RelationIsAccessibleInLogicalDecoding(relation)) > > log_heap_new_cid(relation, &tp); > > > > Meanwhile I am also exploring the optio

Re: Proposal: Conflict log history table for Logical Replication

2025-09-24 Thread Dilip Kumar
On Thu, Sep 25, 2025 at 11:09 AM Dilip Kumar wrote: > > On Sat, Sep 20, 2025 at 5:29 PM Amit Kapila wrote: > > > > On Thu, Sep 18, 2025 at 11:46 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Sep 18, 2025 at 1:33 AM Amit Kapila > > > wrote: > > > > > > > > If we compare conflict_history_ta

Re: Proposal: Conflict log history table for Logical Replication

2025-09-24 Thread Dilip Kumar
On Tue, Sep 23, 2025 at 11:29 PM Masahiko Sawada wrote: > > On Sat, Sep 20, 2025 at 4:59 AM Amit Kapila wrote: > > > > On Thu, Sep 18, 2025 at 11:46 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Sep 18, 2025 at 1:33 AM Amit Kapila > > > wrote: > > > > > > > > If we compare conflict_histor

Re: Proposal: Conflict log history table for Logical Replication

2025-09-20 Thread Dilip Kumar
On Thu, Sep 11, 2025 at 8:43 AM Amit Kapila wrote: > > On Thu, Sep 11, 2025 at 12:53 AM Bharath Rupireddy > wrote: > > > > On Tue, Aug 5, 2025 at 5:24 AM Dilip Kumar wrote: > > > > > > Currently we log conflicts to the server's log file and updates, this > > > approach has limitations, 1) Diffic

Re: Proposal: Conflict log history table for Logical Replication

2025-09-20 Thread Amit Kapila
On Thu, Sep 18, 2025 at 11:46 PM Masahiko Sawada wrote: > > On Thu, Sep 18, 2025 at 1:33 AM Amit Kapila wrote: > > > > If we compare conflict_history_table with the slot that gets created > > with subscription, one can say the same thing about slots. Users can > > drop the slots and whole replica

Re: Proposal: Conflict log history table for Logical Replication

2025-09-19 Thread Dilip Kumar
On Wed, Sep 10, 2025 at 4:32 PM Alastair Turner wrote: > >> Here we will have to create a built-in type of type table which is I >> think typcategory => 'C' and if we create this type it should be >> supplied with the "typrelid" that means there should be a backing >> catalog table. At least thats

Re: Proposal: Conflict log history table for Logical Replication

2025-09-18 Thread Amit Kapila
On Sun, Sep 14, 2025 at 12:23 PM Dilip Kumar wrote: > > On Sat, Sep 13, 2025 at 6:16 AM Bharath Rupireddy > wrote: > > Thanks for the feedback Bharath > > > On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > > > > > I was looking into another thread where we provide an error table for > > >

Re: Proposal: Conflict log history table for Logical Replication

2025-09-18 Thread Masahiko Sawada
On Thu, Sep 18, 2025 at 1:33 AM Amit Kapila wrote: > > On Sun, Sep 14, 2025 at 12:23 PM Dilip Kumar wrote: > > > > On Sat, Sep 13, 2025 at 6:16 AM Bharath Rupireddy > > wrote: > > > > Thanks for the feedback Bharath > > > > > On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > > > > > > > I

Re: Proposal: Conflict log history table for Logical Replication

2025-09-18 Thread Dilip Kumar
On Thu, Sep 18, 2025 at 2:03 PM Amit Kapila wrote: > > On Sun, Sep 14, 2025 at 12:23 PM Dilip Kumar wrote: > > > > On Sat, Sep 13, 2025 at 6:16 AM Bharath Rupireddy > > wrote: > > > > Thanks for the feedback Bharath > > > > > On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > > > > > > > I

Re: Proposal: Conflict log history table for Logical Replication

2025-09-17 Thread Dilip Kumar
On Sat, Sep 13, 2025 at 6:16 AM Bharath Rupireddy wrote: Thanks for the feedback Bharath > On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > > > I was looking into another thread where we provide an error table for > > COPY [1], it requires the user to pre-create the error table. And > > i

Re: Proposal: Conflict log history table for Logical Replication

2025-09-12 Thread Bharath Rupireddy
Hi, On Fri, Sep 12, 2025 at 3:13 AM Dilip Kumar wrote: > > I was looking into another thread where we provide an error table for > COPY [1], it requires the user to pre-create the error table. And > inside the COPY command we will validate the table, validation in that > context is a one-time pro

Re: Proposal: Conflict log history table for Logical Replication

2025-09-12 Thread Bharath Rupireddy
Hi, On Wed, Sep 10, 2025 at 8:13 PM Amit Kapila wrote: > > > How about streaming the conflicts in fixed format to a separate log > > file other than regular postgres server log file? > > I would prefer this info to be stored in tables as it would be easy to > query them. If we use separate LOGs t

Re: Proposal: Conflict log history table for Logical Replication

2025-09-11 Thread Amit Kapila
On Mon, Sep 8, 2025 at 12:01 PM Dilip Kumar wrote: > > On Sun, Sep 7, 2025 at 1:42 PM Alastair Turner wrote: > > > > Hi Dilip > > > > Thanks for working on this, I think it will make conflict detection a lot > > more useful. > > Thanks for the suggestions, please find my reply inline. > > > On S

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Amit Kapila
On Thu, Sep 11, 2025 at 12:53 AM Bharath Rupireddy wrote: > > On Tue, Aug 5, 2025 at 5:24 AM Dilip Kumar wrote: > > > > Currently we log conflicts to the server's log file and updates, this > > approach has limitations, 1) Difficult to query and analyze, parsing > > plain text log files for confl

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Bharath Rupireddy
Hi, On Tue, Aug 5, 2025 at 5:24 AM Dilip Kumar wrote: > > Currently we log conflicts to the server's log file and updates, this > approach has limitations, 1) Difficult to query and analyze, parsing > plain text log files for conflict details is inefficient. 2) Lack of > structured data, key conf

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Alastair Turner
On Wed, 10 Sept 2025 at 11:15, Dilip Kumar wrote: > On Wed, Sep 10, 2025 at 3:25 PM Amit Kapila > wrote: > > > ... > > > > How about having this as a built-in type? > > Here we will have to create a built-in type of type table which is I > think typcategory => 'C' and if we create this type it

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Dilip Kumar
On Wed, Sep 10, 2025 at 3:25 PM Amit Kapila wrote: > > On Mon, Sep 8, 2025 at 12:01 PM Dilip Kumar wrote: > > > > On Sun, Sep 7, 2025 at 1:42 PM Alastair Turner wrote: > > > > > > Hi Dilip > > > > > > Thanks for working on this, I think it will make conflict detection a lot > > > more useful. >

Re: Proposal: Conflict log history table for Logical Replication

2025-09-07 Thread Dilip Kumar
On Sun, Sep 7, 2025 at 1:42 PM Alastair Turner wrote: > > Hi Dilip > > Thanks for working on this, I think it will make conflict detection a lot > more useful. Thanks for the suggestions, please find my reply inline. > On Sat, 6 Sept 2025, 10:38 Dilip Kumar, wrote: >> >> While working on the p

Re: Proposal: Conflict log history table for Logical Replication

2025-09-07 Thread Alastair Turner
Hi Dilip Thanks for working on this, I think it will make conflict detection a lot more useful. On Sat, 6 Sept 2025, 10:38 Dilip Kumar, wrote: > While working on the patch, I see there are some open questions > > 1. We decided to pass the conflict history table name during > subscription creati

Re: Proposal: Conflict log history table for Logical Replication

2025-09-06 Thread Dilip Kumar
On Thu, Aug 21, 2025 at 9:17 AM Dilip Kumar wrote: > > On Wed, Aug 20, 2025 at 5:46 PM Amit Kapila wrote: > > > > On Wed, Aug 20, 2025 at 11:47 AM Dilip Kumar wrote: > > > > > > On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila > > > wrote: > > > > > > > > > > > One idea to keep things simple for t

Re: Proposal: Conflict log history table for Logical Replication

2025-08-20 Thread Dilip Kumar
On Wed, Aug 20, 2025 at 5:46 PM Amit Kapila wrote: > > On Wed, Aug 20, 2025 at 11:47 AM Dilip Kumar wrote: > > > > On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila > > wrote: > > > > > > > > One idea to keep things simple for the first version is that we allow > > > users to specify the table_name

Re: Proposal: Conflict log history table for Logical Replication

2025-08-20 Thread Amit Kapila
On Wed, Aug 20, 2025 at 11:47 AM Dilip Kumar wrote: > > On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila wrote: > > > > > One idea to keep things simple for the first version is that we allow > > users to specify the table_name for storing conflicts but the table > > should be created internally and

Re: Proposal: Conflict log history table for Logical Replication

2025-08-19 Thread Dilip Kumar
On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila wrote: > > On Fri, Aug 15, 2025 at 2:31 PM Dilip Kumar wrote: > > > > Yet another question is about table names, whether we keep some > > standard name like conflict_log_history_$subid or let users pass the > > name. > > > > It would be good if we can

Re: Proposal: Conflict log history table for Logical Replication

2025-08-17 Thread Amit Kapila
On Fri, Aug 15, 2025 at 2:31 PM Dilip Kumar wrote: > > Yet another question is about table names, whether we keep some > standard name like conflict_log_history_$subid or let users pass the > name. > It would be good if we can let the user specify the table_name and if she didn't specify then use

Re: Proposal: Conflict log history table for Logical Replication

2025-08-15 Thread Dilip Kumar
On Wed, Aug 13, 2025 at 3:39 PM Amit Kapila wrote: > > On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: > > > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > > > > > > > So logically for PostgreSQL its an > > > > user tabl

Re: Proposal: Conflict log history table for Logical Replication

2025-08-14 Thread Amit Kapila
On Thu, Aug 14, 2025 at 4:26 PM Alastair Turner wrote: > > On Wed, 13 Aug 2025 at 11:09, Amit Kapila wrote: >> >> On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: >> > >> > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: >> > > >> > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: >> >

Re: Proposal: Conflict log history table for Logical Replication

2025-08-14 Thread Alastair Turner
On Wed, 13 Aug 2025 at 11:09, Amit Kapila wrote: > On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik > wrote: > > > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar > wrote: > > > > > > > > So logically for PostgreSQL its an > > > > user tab

Re: Proposal: Conflict log history table for Logical Replication

2025-08-13 Thread Amit Kapila
On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > > > > > So logically for PostgreSQL its an > > > user table but yeah this is created and managed by the extension. > > > > > > >

Re: Proposal: Conflict log history table for Logical Replication

2025-08-08 Thread shveta malik
On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > > > > > So logically for PostgreSQL its an > > > user table but yeah this is created and managed by the extension. > > > > > > >

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread Dilip Kumar
On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > > > So logically for PostgreSQL its an > > user table but yeah this is created and managed by the extension. > > > > Any idea if the user can alter/drop or perform any DML on it? I could >

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > On Thu, Aug 7, 2025 at 1:43 PM shveta malik wrote: > > > > On Thu, Aug 7, 2025 at 12:25 PM shveta malik wrote: > > Thanks Shveta for your opinion on the design. > > > > On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > > > > > > > > This

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread Dilip Kumar
On Thu, Aug 7, 2025 at 1:43 PM shveta malik wrote: > > On Thu, Aug 7, 2025 at 12:25 PM shveta malik wrote: Thanks Shveta for your opinion on the design. > > On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > > > > > This proposal aims to address these limitations by introducing a > > > con

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 12:25 PM shveta malik wrote: > > On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > > > Currently we log conflicts to the server's log file and updates, this > > approach has limitations, 1) Difficult to query and analyze, parsing > > plain text log files for conflict de

Re: Proposal: Conflict log history table for Logical Replication

2025-08-06 Thread shveta malik
On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > Currently we log conflicts to the server's log file and updates, this > approach has limitations, 1) Difficult to query and analyze, parsing > plain text log files for conflict details is inefficient. 2) Lack of > structured data, key conflict