Re: Moving delta data faster

On Sun, Apr 7, 2024 at 2:25 AM Adrian Klaver wrote: > > I have no idea how this works in the code, but my suspicion is it is due > to the following: > > https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT > > "The optional ON CONFLICT clause specifies an alternative action to

Re: Moving delta data faster

On 4/6/24 13:04, yudhi s wrote: On Sat, Apr 6, 2024 at 10:25 PM Adrian Klaver > wrote: Your original problem description was: "Then subsequently these rows will be inserted/updated based on the delta number of rows that got inserted/updated in

Re: Moving delta data faster

On Sat, Apr 6, 2024 at 10:25 PM Adrian Klaver wrote: > > Your original problem description was: > > "Then subsequently these rows will be inserted/updated based on the > delta number of rows that got inserted/updated in the source database. > In some cases these changed data can flow multiple

Re: Moving delta data faster

On 4/6/24 08:47, yudhi s wrote: Thank you Adrian, Greg and Veem. I tried writing a small routine to see how the performance differs in these four approaches i.e. Upsert VS traditional update+insert VS Merge vs Truncate+load. Initially I was thinking Upsert will perform the same as Merge as

Re: Moving delta data faster

Thank you Adrian, Greg and Veem. I tried writing a small routine to see how the performance differs in these four approaches i.e. Upsert VS traditional update+insert VS Merge vs Truncate+load. Initially I was thinking Upsert will perform the same as Merge as the logic looks similar but it seems

Re: Moving delta data faster

On Fri, 5 Apr 2024 at 06:10, Adrian Klaver wrote: > > > S3 is not a database. You will need to be more specific about '... > then > > from the S3 it will be picked and gets merged to the target postgres > > database.' > > > > > > The data from S3 will be dumped into the stage table

Re: Moving delta data faster

On 4/4/24 13:42, yudhi s wrote: On Thu, Apr 4, 2024 at 9:04 PM Adrian Klaver > wrote: On 4/3/24 22:24, yudhi s wrote: > > On Thu, Apr 4, 2024 at 10:16 AM Adrian Klaver mailto:adrian.kla...@aklaver.com> >

Re: Moving delta data faster

So you want advice from this open source community about moving things from a proprietary database to another proprietary database, going through a proprietary storage system along the way? :) To answer the original question, use MERGE. That's it's job, and your table size is very tiny, so I

Re: Moving delta data faster

On Thu, Apr 4, 2024 at 9:04 PM Adrian Klaver wrote: > On 4/3/24 22:24, yudhi s wrote: > > > > On Thu, Apr 4, 2024 at 10:16 AM Adrian Klaver > > wrote: > > S3 is not a database. You will need to be more specific about '... then > from the S3 it will be picked

Re: Moving delta data faster

On 4/3/24 22:24, yudhi s wrote: On Thu, Apr 4, 2024 at 10:16 AM Adrian Klaver > wrote: On 4/3/24 20:54, yudhi s wrote: > On Thu, Apr 4, 2024 at 2:41 AM Adrian Klaver mailto:adrian.kla...@aklaver.com> >

Re: Moving delta data faster

On Thu, Apr 4, 2024 at 10:16 AM Adrian Klaver wrote: > On 4/3/24 20:54, yudhi s wrote: > > On Thu, Apr 4, 2024 at 2:41 AM Adrian Klaver > > wrote: > > > >> Thank you Adrian. > > > > And one thing i forgot to mention this target postgresql database would > >

Re: Moving delta data faster

On 4/3/24 20:54, yudhi s wrote: On Thu, Apr 4, 2024 at 2:41 AM Adrian Klaver > wrote: On 4/3/24 13:38, yudhi s wrote: >   Hi All, >   It's postgresql database version 15.4. We have a requirement in which > we will be initially moving full

Re: Moving delta data faster

On Thu, Apr 4, 2024 at 2:41 AM Adrian Klaver wrote: > On 4/3/24 13:38, yudhi s wrote: > > Hi All, > > It's postgresql database version 15.4. We have a requirement in which > > we will be initially moving full table data for 3-4 tables, from source > > database to target(i.e. postgres) .

Re: Moving delta data faster

On 4/3/24 13:38, yudhi s wrote:  Hi All,  It's postgresql database version 15.4. We have a requirement in which we will be initially moving full table data for 3-4 tables, from source database to target(i.e. postgres) . Maximum number of rows will be ~10million rows in those tables. Then

Moving delta data faster

Hi All, It's postgresql database version 15.4. We have a requirement in which we will be initially moving full table data for 3-4 tables, from source database to target(i.e. postgres) . Maximum number of rows will be ~10million rows in those tables. Then subsequently these rows will be