Re: Make default subscription streaming option as Parallel

2024-10-27 Thread Amit Kapila
On Tue, Oct 22, 2024 at 9:26 PM vignesh C wrote: > > The attached v5 version has the change to create subscriptions in > streaming off mode. I also did not find any other TAP test which > required further changes. > Pushed. -- With Regards, Amit Kapila.

Re: Make default subscription streaming option as Parallel

2024-10-23 Thread Amit Kapila
On Wed, Oct 23, 2024 at 1:21 AM Masahiko Sawada wrote: > > On Tue, Oct 22, 2024 at 2:17 AM Amit Kapila wrote: > > > > On Mon, Oct 21, 2024 at 8:40 PM vignesh C wrote: > > > > > > Attached v3 version patch has a caution added for the same. > > > > > > > Thanks, the patch looks good to me and I am

Re: Make default subscription streaming option as Parallel

2024-10-22 Thread Amit Kapila
On Wed, Oct 23, 2024 at 5:58 AM Peter Smith wrote: > > Hi Vignesh, here are some review comments for the patch v5-0001 (docs only). > > == > Commit message > > 1. > The commit message only refers to this as the "streaming option", but > an option of what? Somewhere we should mention this is an

Re: Make default subscription streaming option as Parallel

2024-10-22 Thread Peter Smith
Hi Vignesh, here are some review comments for the patch v5-0001 (docs only). == Commit message 1. The commit message only refers to this as the "streaming option", but an option of what? Somewhere we should mention this is an option of CREATE SUBSCRIPTION. ~ 2. Previously the default value

Re: Make default subscription streaming option as Parallel

2024-10-22 Thread Masahiko Sawada
On Tue, Oct 22, 2024 at 2:17 AM Amit Kapila wrote: > > On Mon, Oct 21, 2024 at 8:40 PM vignesh C wrote: > > > > Attached v3 version patch has a caution added for the same. > > > > Thanks, the patch looks good to me and I am planning to commit this > early next week unless there are objections or

Re: Make default subscription streaming option as Parallel

2024-10-22 Thread vignesh C
On Tue, 22 Oct 2024 at 16:24, Hayato Kuroda (Fujitsu) wrote: > > Dear Amit, Vignesh, > > > 1. Please ensure that none of the existing tests that use > > subscriptions with large changes will be impacted due to this change. > > I found at least 022_twophase_cascade.pl should be fixed. > The file ha

RE: Make default subscription streaming option as Parallel

2024-10-22 Thread Hayato Kuroda (Fujitsu)
Dear Amit, Vignesh, > 1. Please ensure that none of the existing tests that use > subscriptions with large changes will be impacted due to this change. I found at least 022_twophase_cascade.pl should be fixed. The file has a part which tests non-streaming case: ``` # --- # 2P

Re: Make default subscription streaming option as Parallel

2024-10-22 Thread Amit Kapila
On Mon, Oct 21, 2024 at 8:40 PM vignesh C wrote: > > Attached v3 version patch has a caution added for the same. > Thanks, the patch looks good to me and I am planning to commit this early next week unless there are objections or any major problems. I have slightly updated the docs and commit mes

Re: Make default subscription streaming option as Parallel

2024-10-21 Thread vignesh C
On Mon, 21 Oct 2024 at 14:36, Amit Kapila wrote: > > On Mon, Oct 7, 2024 at 11:05 AM vignesh C wrote: > > > > The tests demonstrate a significant performance improvement when using > > the parallel streaming option, insert shows 40-48 %improvement, delete > > shows 34-39 %improvement, update show

Re: Make default subscription streaming option as Parallel

2024-10-21 Thread Peter Smith
On Mon, Oct 21, 2024 at 5:09 PM Amit Kapila wrote: > > On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: > > > > On Mon, 7 Oct 2024 at 12:26, Hayato Kuroda (Fujitsu) > > wrote: > > > > > One comment for your patch; > > > Shouldn't we add a streaming=off case for pg_dump test? You added lines > >

Re: Make default subscription streaming option as Parallel

2024-10-21 Thread Amit Kapila
On Mon, Oct 7, 2024 at 11:05 AM vignesh C wrote: > > The tests demonstrate a significant performance improvement when using > the parallel streaming option, insert shows 40-48 %improvement, delete > shows 34-39 %improvement, update shows 26-30 %improvement. In the case > of rollback the improvemen

Re: Make default subscription streaming option as Parallel

2024-10-20 Thread Amit Kapila
On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: > > On Mon, 7 Oct 2024 at 12:26, Hayato Kuroda (Fujitsu) > wrote: > > > One comment for your patch; > > Shouldn't we add a streaming=off case for pg_dump test? You added lines but > > no one > > passes it. > > > > Update existing pg_dump tests to c

Re: Make default subscription streaming option as Parallel

2024-10-19 Thread Dilip Kumar
On Fri, 18 Oct 2024 at 5:24 PM, Amit Kapila wrote: > On Fri, Oct 18, 2024 at 9:48 AM Dilip Kumar wrote: > > > > On Tue, Oct 8, 2024 at 3:38 PM Amit Kapila > wrote: > >> > >> On Tue, Oct 8, 2024 at 2:25 PM shveta malik > wrote: > >> > > >> > On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: > >>

Re: Make default subscription streaming option as Parallel

2024-10-18 Thread Amit Kapila
On Fri, Oct 18, 2024 at 9:48 AM Dilip Kumar wrote: > > On Tue, Oct 8, 2024 at 3:38 PM Amit Kapila wrote: >> >> On Tue, Oct 8, 2024 at 2:25 PM shveta malik wrote: >> > >> > On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: >> > > >> > >> > With parallel streaming as default, do you think there is

Re: Make default subscription streaming option as Parallel

2024-10-17 Thread Dilip Kumar
On Tue, Oct 8, 2024 at 3:38 PM Amit Kapila wrote: > On Tue, Oct 8, 2024 at 2:25 PM shveta malik > wrote: > > > > On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: > > > > > > > With parallel streaming as default, do you think there is a need to > > increase the default for 'max_logical_replicatio

Re: Make default subscription streaming option as Parallel

2024-10-08 Thread shveta malik
On Tue, Oct 8, 2024 at 3:38 PM Amit Kapila wrote: > > On Tue, Oct 8, 2024 at 2:25 PM shveta malik wrote: > > > > On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: > > > > > > > With parallel streaming as default, do you think there is a need to > > increase the default for 'max_logical_replication

Re: Make default subscription streaming option as Parallel

2024-10-08 Thread vignesh C
On Mon, 7 Oct 2024 at 11:05, vignesh C wrote: > > Hi, > > By default, currently streaming of in-progress transactions for > subscriptions is disabled. All transactions are fully decoded on the > publisher before being sent to the subscriber. This approach can lead > to increased latency and reduce

Re: Make default subscription streaming option as Parallel

2024-10-08 Thread Amit Kapila
On Tue, Oct 8, 2024 at 2:25 PM shveta malik wrote: > > On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: > > > > With parallel streaming as default, do you think there is a need to > increase the default for 'max_logical_replication_workers' as IIUC > parallel workers are taken from the same pool.

Re: Make default subscription streaming option as Parallel

2024-10-08 Thread shveta malik
On Mon, Oct 7, 2024 at 4:03 PM vignesh C wrote: > With parallel streaming as default, do you think there is a need to increase the default for 'max_logical_replication_workers' as IIUC parallel workers are taken from the same pool. thanks Shveta

Re: Make default subscription streaming option as Parallel

2024-10-07 Thread vignesh C
On Mon, 7 Oct 2024 at 12:26, Hayato Kuroda (Fujitsu) wrote: > > Dear Vignesh, > > > One key point to consider is that the lock on transaction objects will > > be held for a longer duration when using streaming in parallel. This > > occurs because the parallel apply worker initiates the transaction

RE: Make default subscription streaming option as Parallel

2024-10-06 Thread Hayato Kuroda (Fujitsu)
Dear Vignesh, > One key point to consider is that the lock on transaction objects will > be held for a longer duration when using streaming in parallel. This > occurs because the parallel apply worker initiates the transaction as > soon as streaming begins, maintaining the lock until the transacti

Make default subscription streaming option as Parallel

2024-10-06 Thread vignesh C
Hi, By default, currently streaming of in-progress transactions for subscriptions is disabled. All transactions are fully decoded on the publisher before being sent to the subscriber. This approach can lead to increased latency and reduced performance, particularly under heavy load. By default, we