On Wed, Mar 4, 2020 at 4:04 PM Dilip Kumar <dilipbal...@gmail.com> wrote: > > On Wed, Mar 4, 2020 at 3:47 PM Amit Kapila <amit.kapil...@gmail.com> wrote: > > > > On Wed, Mar 4, 2020 at 11:16 AM Dilip Kumar <dilipbal...@gmail.com> wrote: > > > > > > On Wed, Mar 4, 2020 at 10:50 AM Amit Kapila <amit.kapil...@gmail.com> > > > wrote: > > > > > > > > On Wed, Mar 4, 2020 at 9:52 AM Dilip Kumar <dilipbal...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > IMHO, the threshold should be based on the commit LSN. Our main > > > > > reason we want to send empty transactions after a certain > > > > > transaction/duration is that we want the restart_lsn to be moving > > > > > forward so that if we need to restart the replication slot we don't > > > > > need to process a lot of extra WAL. So assume we set the threshold > > > > > based on transaction count then there is still a possibility that we > > > > > might process a few very big transactions then we will have to process > > > > > them again after the restart. > > > > > > > > > > > > > Won't the subscriber eventually send the flush location for the large > > > > transactions which will move the restart_lsn? > > > > > > I meant large empty transactions (basically we can not send anything > > > to the subscriber). So my point was if there are only large > > > transactions in the system which we can not stream because those > > > tables are not published. Then keeping threshold based on transaction > > > count will not help much because even if we don't reach the > > > transaction count threshold, we still might need to process a lot of > > > data if we don't stream the commit for the empty transactions. So > > > instead of tracking transaction count can we track LSN, and LSN > > > different since we last stream some change cross the threshold then we > > > will stream the next empty transaction. > > > > > > > You have a point and it may be better to keep threshold based on LSN > > if we want to keep any threshold, but keeping on transaction count > > seems to be a bit straightforward. Let us see if anyone else has any > > opinion on this matter? > > Ok, that make sense. >
Euler, can we try to update the patch based on the number of transactions threshold and see how it works? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com