RE: Failed transaction statistics to measure the logical replication progress

2022-03-28 Thread osumi.takami...@fujitsu.com
Hi On Friday, March 25, 2022 2:36 PM Masahiko Sawada wrote: > On Thu, Mar 24, 2022 at 12:30 PM Amit Kapila > wrote: > > > > On Thu, Mar 3, 2022 at 8:58 AM osumi.takami...@fujitsu.com > > wrote: > > > > > > > This patch introduces two new subscription statistics columns > > (apply_commit_count

Re: Failed transaction statistics to measure the logical replication progress

2022-03-24 Thread Masahiko Sawada
On Thu, Mar 24, 2022 at 12:30 PM Amit Kapila wrote: > > On Thu, Mar 3, 2022 at 8:58 AM osumi.takami...@fujitsu.com > wrote: > > > > This patch introduces two new subscription statistics columns > (apply_commit_count and apply_rollback_count) to the > pg_stat_subscription_stats view for counting

Re: Failed transaction statistics to measure the logical replication progress

2022-03-23 Thread Amit Kapila
On Thu, Mar 3, 2022 at 8:58 AM osumi.takami...@fujitsu.com wrote: > This patch introduces two new subscription statistics columns (apply_commit_count and apply_rollback_count) to the pg_stat_subscription_stats view for counting cumulative transactions commits/rollbacks for a particular

RE: Failed transaction statistics to measure the logical replication progress

2022-03-02 Thread osumi.takami...@fujitsu.com
On Wednesday, March 2, 2022 5:29 PM Shi, Yu/侍 雨 wrote: > A comments on the v26 patch. Thank you for checking the patch ! > > The following document about pg_stat_subscription_stats view only says that > "showing statistics about errors", should we add something about transactions > here? > >

RE: Failed transaction statistics to measure the logical replication progress

2022-03-02 Thread osumi.takami...@fujitsu.com
On Wednesday, March 2, 2022 2:18 PM Masahiko Sawada wrote: > On Wed, Mar 2, 2022 at 10:21 AM osumi.takami...@fujitsu.com > wrote: > > Also, I quickly checked other similar views(pg_stat_slru, > > pg_stat_wal_receiver) commit logs, especially when they introduce columns. > > But, I couldn't find

RE: Failed transaction statistics to measure the logical replication progress

2022-03-02 Thread shiy.f...@fujitsu.com
Hi, A comments on the v26 patch. The following document about pg_stat_subscription_stats view only says that "showing statistics about errors", should we add something about transactions here? pg_stat_subscription_statspg_stat_subscription_stats One row per subscription,

Re: Failed transaction statistics to measure the logical replication progress

2022-03-01 Thread Masahiko Sawada
On Wed, Mar 2, 2022 at 10:21 AM osumi.takami...@fujitsu.com wrote: > > > Also, I quickly checked other similar views(pg_stat_slru, > pg_stat_wal_receiver) > commit logs, especially when they introduce columns. > But, I couldn't find column name validations. > So, I feel this is aligned. > I've

RE: Failed transaction statistics to measure the logical replication progress

2022-03-01 Thread osumi.takami...@fujitsu.com
On Tuesday, March 1, 2022 4:12 PM Peter Smith wrote: > Please see below my review comments for v25. > > == > > 1. Commit message > > Introduce cumulative columns of transactions of logical replication subscriber > to the pg_stat_subscription_stats view. > > "cumulative columns of

Re: Failed transaction statistics to measure the logical replication progress

2022-02-28 Thread Peter Smith
Please see below my review comments for v25. == 1. Commit message Introduce cumulative columns of transactions of logical replication subscriber to the pg_stat_subscription_stats view. "cumulative columns of transactions" sounds a bit strange to me. SUGGESTED Introduce 2 new subscription

RE: Failed transaction statistics to measure the logical replication progress

2022-02-28 Thread osumi.takami...@fujitsu.com
On Friday, February 25, 2022 7:58 AM osumi.takami...@fujitsu.com wrote: > Kindly have a look at v24. Hi. The recent commit(7a85073) has redesigned the view pg_stat_subscription_workers and now we have pg_stat_subscription_stats. Therefore, I rebased my patch so that my statistics patch can be

RE: Failed transaction statistics to measure the logical replication progress

2022-02-24 Thread osumi.takami...@fujitsu.com
On Wednesday, February 23, 2022 3:30 PM Amit Kapila > On Tue, Feb 22, 2022 at 6:45 AM tanghy.f...@fujitsu.com > wrote: > > > > I found a problem when using it. When a replication workers exits, the > > transaction stats should be sent to stats collector if they were not > > sent before because

RE: Failed transaction statistics to measure the logical replication progress

2022-02-24 Thread osumi.takami...@fujitsu.com
On Thursday, February 24, 2022 11:07 AM Masahiko Sawada wrote: > I have some comments on v23 patch: > > @@ -66,6 +66,12 @@ typedef struct LogicalRepWorker > TimestampTz last_recv_time; > XLogRecPtr reply_lsn; > TimestampTz reply_time; > + > + /* > +* Transaction statistics of

Re: Failed transaction statistics to measure the logical replication progress

2022-02-23 Thread Masahiko Sawada
On Mon, Feb 21, 2022 at 12:45 PM osumi.takami...@fujitsu.com wrote: > > On Saturday, February 19, 2022 12:00 AM osumi.takami...@fujitsu.com > wrote: > > On Friday, February 18, 2022 3:34 PM Tang, Haiying/唐 海英 > > wrote: > > > On Wed, Jan 12, 2022 8:35 PM osumi.takami...@fujitsu.com > > >

Re: Failed transaction statistics to measure the logical replication progress

2022-02-22 Thread Amit Kapila
On Tue, Feb 22, 2022 at 6:45 AM tanghy.f...@fujitsu.com wrote: > > I found a problem when using it. When a replication workers exits, the > transaction stats should be sent to stats collector if they were not sent > before > because it didn't reach PGSTAT_STAT_INTERVAL. But I saw that the stats

RE: Failed transaction statistics to measure the logical replication progress

2022-02-21 Thread osumi.takami...@fujitsu.com
On Tuesday, February 22, 2022 10:15 AM Tang, Haiying/唐 海英 wrote: > On Mon, Feb 21, 2022 11:46 AM osumi.takami...@fujitsu.com > wrote: > > > > On Saturday, February 19, 2022 12:00 AM osumi.takami...@fujitsu.com > > wrote: > > > On Friday, February 18, 2022 3:34 PM Tang, Haiying/唐 海英 > > >

RE: Failed transaction statistics to measure the logical replication progress

2022-02-21 Thread tanghy.f...@fujitsu.com
On Mon, Feb 21, 2022 11:46 AM osumi.takami...@fujitsu.com wrote: > > On Saturday, February 19, 2022 12:00 AM osumi.takami...@fujitsu.com > wrote: > > On Friday, February 18, 2022 3:34 PM Tang, Haiying/唐 海英 > > wrote: > > > On Wed, Jan 12, 2022 8:35 PM osumi.takami...@fujitsu.com > > > wrote:

RE: Failed transaction statistics to measure the logical replication progress

2022-02-21 Thread osumi.takami...@fujitsu.com
On Monday, February 21, 2022 6:06 PM Monday, February 21, 2022 6:06 PM wrote: > On Mon, Feb 21, 2022 11:46 AM Osumi, Takamichi/大墨 昂道 > wrote: > >I've addressed this point in a new v23 patch, since there was no opinion on > this so far. > >Kindly have a look at the attached one. > Thanks for

RE: Failed transaction statistics to measure the logical replication progress

2022-02-21 Thread wangw.f...@fujitsu.com
On Mon, Feb 21, 2022 11:46 AM Osumi, Takamichi/大墨 昂道 wrote: >I've addressed this point in a new v23 patch, since there was no opinion on >this so far. >Kindly have a look at the attached one. Thanks for updating the patch. Here is a comment: In function apply_handle_stream_abort: @@ -1217,6

RE: Failed transaction statistics to measure the logical replication progress

2022-02-20 Thread osumi.takami...@fujitsu.com
On Saturday, February 19, 2022 12:00 AM osumi.takami...@fujitsu.com wrote: > On Friday, February 18, 2022 3:34 PM Tang, Haiying/唐 海英 > wrote: > > On Wed, Jan 12, 2022 8:35 PM osumi.takami...@fujitsu.com > > wrote: > > 4) I noticed that the abort_count doesn't include aborted streaming > >

RE: Failed transaction statistics to measure the logical replication progress

2022-02-18 Thread osumi.takami...@fujitsu.com
On Friday, February 18, 2022 3:34 PM Tang, Haiying/唐 海英 wrote: > On Wed, Jan 12, 2022 8:35 PM osumi.takami...@fujitsu.com > wrote: > > > > The attached v21 has a couple of other minor updates like a > > modification of error message text. > > > > > > Thanks for updating the patch. Here are

RE: Failed transaction statistics to measure the logical replication progress

2022-02-18 Thread osumi.takami...@fujitsu.com
On Friday, February 18, 2022 8:11 PM Amit Kapila wrote: > On Fri, Feb 18, 2022 at 2:04 PM osumi.takami...@fujitsu.com > wrote: > > > > On Thursday, February 17, 2022 6:45 PM Amit Kapila > wrote: > > > On Thu, Feb 17, 2022 at 3:13 PM Amit Kapila > > > > > > wrote: > > > > Can't we use

Re: Failed transaction statistics to measure the logical replication progress

2022-02-18 Thread Amit Kapila
On Fri, Feb 18, 2022 at 2:04 PM osumi.takami...@fujitsu.com wrote: > > On Thursday, February 17, 2022 6:45 PM Amit Kapila > wrote: > > On Thu, Feb 17, 2022 at 3:13 PM Amit Kapila > > wrote: > > > Can't we use pgstat_report_stat() here? Basically, you can update xact > > > completetion counters

RE: Failed transaction statistics to measure the logical replication progress

2022-02-18 Thread osumi.takami...@fujitsu.com
On Thursday, February 17, 2022 6:45 PM Amit Kapila wrote: > On Thu, Feb 17, 2022 at 3:13 PM Amit Kapila > wrote: > > > > On Tue, Jan 4, 2022 at 5:22 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Friday, December 31, 2021 10:12 AM Tang, Haiying/唐 海英 > wrote: > > > > 4) > > > > +void

RE: Failed transaction statistics to measure the logical replication progress

2022-02-17 Thread tanghy.f...@fujitsu.com
On Wed, Jan 12, 2022 8:35 PM osumi.takami...@fujitsu.com wrote: > > The attached v21 has a couple of other minor updates > like a modification of error message text. > > Thanks for updating the patch. Here are some comments. 1) I saw the following description about

Re: Failed transaction statistics to measure the logical replication progress

2022-02-17 Thread Amit Kapila
On Thu, Feb 17, 2022 at 3:13 PM Amit Kapila wrote: > > On Tue, Jan 4, 2022 at 5:22 PM osumi.takami...@fujitsu.com > wrote: > > > > On Friday, December 31, 2021 10:12 AM Tang, Haiying/唐 海英 > > wrote: > > > 4) > > > +void > > > +pgstat_send_subworker_xact_stats(LogicalRepWorker *repWorker, bool

Re: Failed transaction statistics to measure the logical replication progress

2022-02-17 Thread Amit Kapila
On Tue, Jan 4, 2022 at 5:22 PM osumi.takami...@fujitsu.com wrote: > > On Friday, December 31, 2021 10:12 AM Tang, Haiying/唐 海英 > wrote: > > 4) > > +void > > +pgstat_send_subworker_xact_stats(LogicalRepWorker *repWorker, bool > > +force) { > > + static TimestampTz last_report = 0; > > +

Re: Failed transaction statistics to measure the logical replication progress

2022-02-17 Thread Amit Kapila
On Wed, Jan 12, 2022 at 6:04 PM osumi.takami...@fujitsu.com wrote: > > On Thursday, December 23, 2021 6:37 PM Wang, Wei/王 威 > wrote: > > On Wednesday, December 22, 2021 10:30 PM osumi.takami...@fujitsu.com > > wrote: > > > On Wednesday, December 22, 2021 8:38 PM I wrote: > > > > Do we expect

RE: Failed transaction statistics to measure the logical replication progress

2022-01-12 Thread osumi.takami...@fujitsu.com
On Thursday, December 23, 2021 6:37 PM Wang, Wei/王 威 wrote: > On Wednesday, December 22, 2021 10:30 PM osumi.takami...@fujitsu.com > wrote: > > On Wednesday, December 22, 2021 8:38 PM I wrote: > > > Do we expect these commit counts which come from empty transactions ? > > This is another issue

RE: Failed transaction statistics to measure the logical replication progress

2022-01-04 Thread osumi.takami...@fujitsu.com
On Friday, December 31, 2021 10:12 AM Tang, Haiying/唐 海英 wrote: > On Wednesday, December 22, 2021 6:14 PM osumi.takami...@fujitsu.com > wrote: > > > > Attached the new patch v19. > > > > Thanks for your patch. I think it's better if you could add this patch to the > commitfest. > Here are some

RE: Failed transaction statistics to measure the logical replication progress

2022-01-04 Thread osumi.takami...@fujitsu.com
On Monday, January 3, 2022 2:46 PM Hou, Zhijie/侯 志杰 wrote: > On Wednesday, December 22, 2021 6:14 PM Osumi, Takamichi > wrote: > > Attached the new patch v19. > Hi, > > Thanks for updating the patch. > > --- a/src/include/pgstat.h > +++ b/src/include/pgstat.h > @@ -15,6 +15,7 @@ > #include

RE: Failed transaction statistics to measure the logical replication progress

2022-01-02 Thread houzj.f...@fujitsu.com
On Wednesday, December 22, 2021 6:14 PM Osumi, Takamichi wrote: > Attached the new patch v19. Hi, Thanks for updating the patch. --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -15,6 +15,7 @@ #include "portability/instr_time.h" #include "postmaster/pgarch.h" /* for MAX_XFN_CHARS */

RE: Failed transaction statistics to measure the logical replication progress

2021-12-30 Thread tanghy.f...@fujitsu.com
On Wednesday, December 22, 2021 6:14 PM osumi.takami...@fujitsu.com wrote: > > Attached the new patch v19. > Thanks for your patch. I think it's better if you could add this patch to the commitfest. Here are some comments: 1) + commit_count bigint + + + Number of

RE: Failed transaction statistics to measure the logical replication progress

2021-12-23 Thread wangw.f...@fujitsu.com
On Wednesday, December 22, 2021 10:30 PM osumi.takami...@fujitsu.com wrote: > On Wednesday, December 22, 2021 8:38 PM I wrote: > > Do we expect these commit counts which come from empty transactions ? > This is another issue discussed in [1] > where the patch in the thread is a work in progress,

RE: Failed transaction statistics to measure the logical replication progress

2021-12-22 Thread osumi.takami...@fujitsu.com
On Wednesday, December 22, 2021 9:38 PM Wang, Wei/王 威 wrote: > I have a question on the v19-0002 patch: > > When I tested for this patch, I found pg_stat_subscription_workers has some > unexpected data. > For example: > [Publisher] > create table replica_test1(a int, b text); create publication

RE: Failed transaction statistics to measure the logical replication progress

2021-12-22 Thread wangw.f...@fujitsu.com
On Mon, Dec 22, 2021 at 6:14 PM osumi.takami...@fujitsu.com wrote: > > Attached the new patch v19. > I have a question on the v19-0002 patch: When I tested for this patch, I found pg_stat_subscription_workers has some unexpected data. For example: [Publisher] create table replica_test1(a int,

RE: Failed transaction statistics to measure the logical replication progress

2021-12-22 Thread osumi.takami...@fujitsu.com
On Tuesday, December 21, 2021 6:00 PM Greg Nancarrow wrote: > Some review comments on the v18 patches: Thank you for your review ! > v18-0002 > > doc/src/sgml/monitoring.sgml > (1) tablesync worker stats? > > Shouldn't the comment below only mention the apply worker? (since we're no > longer

Re: Failed transaction statistics to measure the logical replication progress

2021-12-21 Thread Greg Nancarrow
On Mon, Dec 20, 2021 at 8:40 PM osumi.takami...@fujitsu.com wrote: > > Updated the patch to address your concern. > Some review comments on the v18 patches: v18-0002 doc/src/sgml/monitoring.sgml (1) tablesync worker stats? Shouldn't the comment below only mention the apply worker? (since

RE: Failed transaction statistics to measure the logical replication progress

2021-12-20 Thread osumi.takami...@fujitsu.com
Friday, December 17, 2021 2:03 PM Kyotaro Horiguchi wrote: > It sends stats packets at every commit-like operation on apply workers. The > current pgstat is so smart that it refrain from sending stats packets at too > high > frequency. We already suffer frequent stats packets so apply workers

Re: Failed transaction statistics to measure the logical replication progress

2021-12-16 Thread Kyotaro Horiguchi
At Thu, 16 Dec 2021 11:36:46 +, "osumi.takami...@fujitsu.com" wrote in > On Thursday, December 16, 2021 4:00 PM I wrote: > > Thank you, everyone for confirming the direction. > > I'll follow the consensus of the community and fix the patch, including > > other > > comments. > > I'll treat

RE: Failed transaction statistics to measure the logical replication progress

2021-12-16 Thread osumi.takami...@fujitsu.com
On Thursday, December 16, 2021 8:37 PM I wrote: > Hi, created a new version v17 according to the recent discussion with changes > to address other review comments. FYI, in v17 I've removed one part of commit message about spool file statistics on the subscriber. My intention is just to make the

RE: Failed transaction statistics to measure the logical replication progress

2021-12-16 Thread osumi.takami...@fujitsu.com
On Tuesday, December 14, 2021 12:45 AM vignesh C wrote: > Thanks for the updated patch, few comments: > 1) Can we change this: > /* > +* Report the success of table sync as one commit to consolidate all > +* transaction stats into one record. > +*/ > +

RE: Failed transaction statistics to measure the logical replication progress

2021-12-16 Thread osumi.takami...@fujitsu.com
On Monday, December 13, 2021 6:19 PM Amit Kapila wrote: > On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com > wrote: > > > > Few questions and comments: > > 1. > The pg_stat_subscription_workers view will > contain > one row per subscription worker on

RE: Failed transaction statistics to measure the logical replication progress

2021-12-16 Thread osumi.takami...@fujitsu.com
On Thursday, December 16, 2021 4:00 PM I wrote: > Thank you, everyone for confirming the direction. > I'll follow the consensus of the community and fix the patch, including other > comments. > I'll treat only the stats for apply workers. Hi, created a new version v17 according to the recent

Re: Failed transaction statistics to measure the logical replication progress

2021-12-15 Thread Greg Nancarrow
On Tue, Dec 14, 2021 at 1:28 PM Amit Kapila wrote: > > If we fear a large number of entries for such workers then won't it be > better to show the value of these stats only for apply workers. I > think normally the table sync workers perform only copy operation or > maybe a fixed number of xacts,

RE: Failed transaction statistics to measure the logical replication progress

2021-12-15 Thread osumi.takami...@fujitsu.com
On Wednesday, December 15, 2021 9:52 PM vignesh C wrote: > On Tue, Dec 14, 2021 at 7:58 AM Amit Kapila > wrote: > > > > On Mon, Dec 13, 2021 at 5:48 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Monday, December 13, 2021 6:19 PM Amit Kapila > wrote: > > > > On Tue, Dec 7, 2021 at

Re: Failed transaction statistics to measure the logical replication progress

2021-12-15 Thread vignesh C
On Tue, Dec 14, 2021 at 7:58 AM Amit Kapila wrote: > > On Mon, Dec 13, 2021 at 5:48 PM osumi.takami...@fujitsu.com > wrote: > > > > On Monday, December 13, 2021 6:19 PM Amit Kapila > > wrote: > > > On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com > > > wrote: > > > > > > Few

Re: Failed transaction statistics to measure the logical replication progress

2021-12-14 Thread Masahiko Sawada
On Tue, Dec 14, 2021 at 11:28 AM Amit Kapila wrote: > > On Mon, Dec 13, 2021 at 5:48 PM osumi.takami...@fujitsu.com > wrote: > > > > On Monday, December 13, 2021 6:19 PM Amit Kapila > > wrote: > > > On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com > > > wrote: > > > > > > Few

RE: Failed transaction statistics to measure the logical replication progress

2021-12-14 Thread houzj.f...@fujitsu.com
On Tues, Dec 14, 2021 10:28 AM Amit Kapila wrote: > On Mon, Dec 13, 2021 at 5:48 PM osumi.takami...@fujitsu.com > wrote: > > > > On Monday, December 13, 2021 6:19 PM Amit Kapila > wrote: > > > On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com > > > wrote: > > > > > > Few questions and

Re: Failed transaction statistics to measure the logical replication progress

2021-12-13 Thread Amit Kapila
On Mon, Dec 13, 2021 at 5:48 PM osumi.takami...@fujitsu.com wrote: > > On Monday, December 13, 2021 6:19 PM Amit Kapila > wrote: > > On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com > > wrote: > > > > Few questions and comments: > Thank you for your comments ! > > >

Re: Failed transaction statistics to measure the logical replication progress

2021-12-13 Thread vignesh C
On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com wrote: > > On Monday, December 6, 2021 11:27 PM vignesh C wrote: > > Thanks for the updated patch, few comments: > Thank you for your review ! > > > 1) We can keep the documentation similar to mention the count includes both > > table

RE: Failed transaction statistics to measure the logical replication progress

2021-12-13 Thread osumi.takami...@fujitsu.com
On Monday, December 13, 2021 6:19 PM Amit Kapila wrote: > On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com > wrote: > > Few questions and comments: Thank you for your comments ! > > 1. > The pg_stat_subscription_workers view will > contain > one row per

Re: Failed transaction statistics to measure the logical replication progress

2021-12-13 Thread Amit Kapila
On Tue, Dec 7, 2021 at 3:12 PM osumi.takami...@fujitsu.com wrote: > Few questions and comments: 1. The pg_stat_subscription_workers view will contain one row per subscription worker on which errors have occurred, for workers applying logical replication changes

RE: Failed transaction statistics to measure the logical replication progress

2021-12-07 Thread osumi.takami...@fujitsu.com
On Monday, December 6, 2021 11:27 PM vignesh C wrote: > Thanks for the updated patch, few comments: Thank you for your review ! > 1) We can keep the documentation similar to mention the count includes both > table sync worker / main apply worker in case of commit_count/error_count > and

Re: Failed transaction statistics to measure the logical replication progress

2021-12-06 Thread vignesh C
On Sat, Dec 4, 2021 at 6:32 PM osumi.takami...@fujitsu.com wrote: > > On Friday, December 3, 2021 3:12 PM vignesh C wrote: > > Thanks for the updated patch. > > Currently we are storing the commit count, error_count and abort_count for > > each table of the table sync operation. If we have

RE: Failed transaction statistics to measure the logical replication progress

2021-12-04 Thread osumi.takami...@fujitsu.com
On Friday, December 3, 2021 3:12 PM vignesh C wrote: > Thanks for the updated patch. > Currently we are storing the commit count, error_count and abort_count for > each table of the table sync operation. If we have thousands of tables, we > will > be storing the information for each of the

Re: Failed transaction statistics to measure the logical replication progress

2021-12-02 Thread vignesh C
On Wed, Dec 1, 2021 at 3:04 PM osumi.takami...@fujitsu.com wrote: > > On Friday, November 19, 2021 11:11 PM Masahiko Sawada > wrote: > > Besides that, I’m not sure how useful commit_bytes, abort_bytes, and > > error_bytes are. I originally thought these statistics track the size of > >

RE: Failed transaction statistics to measure the logical replication progress

2021-12-01 Thread osumi.takami...@fujitsu.com
On Friday, November 19, 2021 11:11 PM Masahiko Sawada wrote: > Besides that, I’m not sure how useful commit_bytes, abort_bytes, and > error_bytes are. I originally thought these statistics track the size of > received > data, i.g., how much data is transferred from the publisher and processed

Re: Failed transaction statistics to measure the logical replication progress

2021-11-23 Thread Masahiko Sawada
On Tue, Nov 23, 2021 at 3:21 PM Greg Nancarrow wrote: > > On Sat, Nov 20, 2021 at 1:11 AM Masahiko Sawada wrote: > > > > I'm concerned that these new names will introduce confusion; if we > > have last_error_relid, last_error_command, last_error_message, > > last_error_time, and last_error_xid,

Re: Failed transaction statistics to measure the logical replication progress

2021-11-23 Thread Masahiko Sawada
On Sat, Nov 20, 2021 at 3:25 PM Amit Kapila wrote: > > On Fri, Nov 19, 2021 at 7:41 PM Masahiko Sawada wrote: > > > > On Thu, Nov 18, 2021 at 12:26 PM Amit Kapila > > wrote: > > > > > > On Wed, Nov 17, 2021 at 7:12 PM osumi.takami...@fujitsu.com > > > wrote: > > > > > > > > On Wednesday,

Re: Failed transaction statistics to measure the logical replication progress

2021-11-22 Thread Greg Nancarrow
On Sat, Nov 20, 2021 at 1:11 AM Masahiko Sawada wrote: > > I'm concerned that these new names will introduce confusion; if we > have last_error_relid, last_error_command, last_error_message, > last_error_time, and last_error_xid, I think users might think that > first_error_time is the timestamp

Re: Failed transaction statistics to measure the logical replication progress

2021-11-21 Thread vignesh C
On Fri, Nov 19, 2021 at 7:41 PM Masahiko Sawada wrote: > > On Thu, Nov 18, 2021 at 12:26 PM Amit Kapila wrote: > > > > On Wed, Nov 17, 2021 at 7:12 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Wednesday, November 17, 2021 10:00 PM Amit Kapila > > > wrote: > > > > > > > > Can you

Re: Failed transaction statistics to measure the logical replication progress

2021-11-19 Thread Amit Kapila
On Fri, Nov 19, 2021 at 7:41 PM Masahiko Sawada wrote: > > On Thu, Nov 18, 2021 at 12:26 PM Amit Kapila wrote: > > > > On Wed, Nov 17, 2021 at 7:12 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Wednesday, November 17, 2021 10:00 PM Amit Kapila > > > wrote: > > > > > > > > Can you

Re: Failed transaction statistics to measure the logical replication progress

2021-11-19 Thread Masahiko Sawada
On Thu, Nov 18, 2021 at 12:26 PM Amit Kapila wrote: > > On Wed, Nov 17, 2021 at 7:12 PM osumi.takami...@fujitsu.com > wrote: > > > > On Wednesday, November 17, 2021 10:00 PM Amit Kapila > > wrote: > > > > > > Can you please tell us why you think the names in your proposed patch are > > >

RE: Failed transaction statistics to measure the logical replication progress

2021-11-18 Thread osumi.takami...@fujitsu.com
On Thursday, November 18, 2021 12:26 PM Amit Kapila wrote: > BTW, I think the way you are computing error_count in > pgstat_recv_subworker_error() doesn't seem correct to me because it will > accumulate the counter/bytes for the same error again and again. > You might want to update these

RE: Failed transaction statistics to measure the logical replication progress

2021-11-18 Thread osumi.takami...@fujitsu.com
On Thursday, November 18, 2021 8:35 PM vignesh C wrote: > On Tue, Nov 16, 2021 at 6:04 PM osumi.takami...@fujitsu.com > wrote: > > > > On Monday, November 15, 2021 9:14 PM I wrote: > > > I've conducted some update for this. > > > (The rebased part is only C code and checked by pgindent) > > I'll

Re: Failed transaction statistics to measure the logical replication progress

2021-11-18 Thread vignesh C
On Tue, Nov 16, 2021 at 6:04 PM osumi.takami...@fujitsu.com wrote: > > On Monday, November 15, 2021 9:14 PM I wrote: > > I've conducted some update for this. > > (The rebased part is only C code and checked by pgindent) > I'll update my patches since a new skip xid patch > has been shared in [1].

Re: Failed transaction statistics to measure the logical replication progress

2021-11-17 Thread Amit Kapila
On Wed, Nov 17, 2021 at 7:12 PM osumi.takami...@fujitsu.com wrote: > > On Wednesday, November 17, 2021 10:00 PM Amit Kapila > wrote: > > > > Can you please tell us why you think the names in your proposed patch are > > better than the existing names proposed in Sawada-San's patch? Is it because

RE: Failed transaction statistics to measure the logical replication progress

2021-11-17 Thread osumi.takami...@fujitsu.com
On Wednesday, November 17, 2021 10:00 PM Amit Kapila wrote: > On Wed, Nov 17, 2021 at 9:44 AM osumi.takami...@fujitsu.com > wrote: > > > > On Wednesday, November 17, 2021 12:19 PM Masahiko Sawada > wrote: > > > On Tue, Nov 16, 2021 at 9:34 PM osumi.takami...@fujitsu.com > > > wrote: > > > > >

Re: Failed transaction statistics to measure the logical replication progress

2021-11-17 Thread Amit Kapila
On Wed, Nov 17, 2021 at 9:44 AM osumi.takami...@fujitsu.com wrote: > > On Wednesday, November 17, 2021 12:19 PM Masahiko Sawada > wrote: > > On Tue, Nov 16, 2021 at 9:34 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Monday, November 15, 2021 9:14 PM I wrote: > > > > I've conducted

RE: Failed transaction statistics to measure the logical replication progress

2021-11-16 Thread osumi.takami...@fujitsu.com
On Wednesday, November 17, 2021 12:19 PM Masahiko Sawada wrote: > On Tue, Nov 16, 2021 at 9:34 PM osumi.takami...@fujitsu.com > wrote: > > > > On Monday, November 15, 2021 9:14 PM I wrote: > > > I've conducted some update for this. > > > (The rebased part is only C code and checked by pgindent)

Re: Failed transaction statistics to measure the logical replication progress

2021-11-16 Thread Masahiko Sawada
On Tue, Nov 16, 2021 at 9:34 PM osumi.takami...@fujitsu.com wrote: > > On Monday, November 15, 2021 9:14 PM I wrote: > > I've conducted some update for this. > > (The rebased part is only C code and checked by pgindent) > I'll update my patches since a new skip xid patch > has been shared in [1].

RE: Failed transaction statistics to measure the logical replication progress

2021-11-16 Thread osumi.takami...@fujitsu.com
On Monday, November 15, 2021 9:14 PM I wrote: > I've conducted some update for this. > (The rebased part is only C code and checked by pgindent) I'll update my patches since a new skip xid patch has been shared in [1]. This version includes some minor renames of functions that are related to

RE: Failed transaction statistics to measure the logical replication progress

2021-11-15 Thread osumi.takami...@fujitsu.com
On Monday, November 15, 2021 6:28 PM I wrote: > On Thursday, November 11, 2021 9:47 PM vignesh C > wrote: > > Few more comments: ... > Fixed. Also, its name was too long > when aligned with other PgStat_StatDBEntry memebers. > Thus I renamed it as subworkers_preparedsizes. > > This depends on

RE: Failed transaction statistics to measure the logical replication progress

2021-11-15 Thread osumi.takami...@fujitsu.com
On Wednesday, November 10, 2021 6:13 PM I wrote: > On Wednesday, November 10, 2021 3:43 PM Dilip Kumar > wrote: > > On Tue, Nov 9, 2021 at 5:05 PM osumi.takami...@fujitsu.com > > wrote: > > > On Tuesday, November 9, 2021 12:08 PM Greg Nancarrow > > wrote: > > > > On Fri, Nov 5, 2021 at 7:11 PM

RE: Failed transaction statistics to measure the logical replication progress

2021-11-15 Thread osumi.takami...@fujitsu.com
On Wednesday, November 10, 2021 7:13 PM vignesh C wrote: > On Tue, Nov 9, 2021 at 5:05 PM osumi.takami...@fujitsu.com > wrote: > > Yes. I've rebased and updated the patch, paying attention to this point. > > Attached the updated version. > > Thanks for the updated patch, few comments: > 1) you

RE: Failed transaction statistics to measure the logical replication progress

2021-11-15 Thread osumi.takami...@fujitsu.com
On Thursday, November 11, 2021 9:47 PM vignesh C wrote: > Few more comments: > 1) Here the tuple length is not considered in the calculation, else it will > always > show the fixed size for any size tuple. Ex varchar insert with 1 byte or > varchar > insert with 100's of bytes. So I feel we

Re: Failed transaction statistics to measure the logical replication progress

2021-11-11 Thread vignesh C
On Wed, Nov 10, 2021 at 3:43 PM vignesh C wrote: > > On Tue, Nov 9, 2021 at 5:05 PM osumi.takami...@fujitsu.com > wrote: > > Yes. I've rebased and updated the patch, paying attention to this point. > > Attached the updated version. > > Thanks for the updated patch, few comments: > 6) Few places

Re: Failed transaction statistics to measure the logical replication progress

2021-11-10 Thread vignesh C
On Tue, Nov 9, 2021 at 5:05 PM osumi.takami...@fujitsu.com wrote: > Yes. I've rebased and updated the patch, paying attention to this point. > Attached the updated version. Thanks for the updated patch, few comments: 1) you could rename PgStat_StatSubWorkerPreparedXact to

RE: Failed transaction statistics to measure the logical replication progress

2021-11-10 Thread osumi.takami...@fujitsu.com
On Wednesday, November 10, 2021 3:43 PM Dilip Kumar wrote: > On Tue, Nov 9, 2021 at 5:05 PM osumi.takami...@fujitsu.com > wrote: > > On Tuesday, November 9, 2021 12:08 PM Greg Nancarrow > wrote: > > > On Fri, Nov 5, 2021 at 7:11 PM osumi.takami...@fujitsu.com > > > wrote: > > > > > > > > > >

Re: Failed transaction statistics to measure the logical replication progress

2021-11-09 Thread Dilip Kumar
On Tue, Nov 9, 2021 at 5:05 PM osumi.takami...@fujitsu.com wrote: > > On Tuesday, November 9, 2021 12:08 PM Greg Nancarrow > wrote: > > On Fri, Nov 5, 2021 at 7:11 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > > I did a quick scan through the latest v8 patch and noticed the following

RE: Failed transaction statistics to measure the logical replication progress

2021-11-09 Thread osumi.takami...@fujitsu.com
On Tuesday, November 9, 2021 8:35 PM I wrote: > Yes. I've rebased and updated the patch, paying attention to this point. > Attached the updated version. Forgot to note one thing. This is based on the skip xid v20 shared in [1] [1] -

RE: Failed transaction statistics to measure the logical replication progress

2021-11-09 Thread osumi.takami...@fujitsu.com
On Monday, November 8, 2021 3:12 PM vignesh C wrote: > On Fri, Nov 5, 2021 at 1:42 PM osumi.takami...@fujitsu.com > wrote: > > Lastly, I removed one unnecessary test that checked publisher's stats > > in the TAP tests. > > Also I introduced ApplyTxnExtraData structure to remove void* argument >

RE: Failed transaction statistics to measure the logical replication progress

2021-11-09 Thread osumi.takami...@fujitsu.com
On Tuesday, November 9, 2021 12:08 PM Greg Nancarrow wrote: > On Fri, Nov 5, 2021 at 7:11 PM osumi.takami...@fujitsu.com > wrote: > > > > I did a quick scan through the latest v8 patch and noticed the following > things: I appreciate your review ! > src/backend/postmaster/pgstat.c > > (1)

Re: Failed transaction statistics to measure the logical replication progress

2021-11-08 Thread Greg Nancarrow
On Fri, Nov 5, 2021 at 7:11 PM osumi.takami...@fujitsu.com wrote: > I did a quick scan through the latest v8 patch and noticed the following things: src/backend/postmaster/pgstat.c (1) pgstat_recv_subworker_twophase_xact() The copying from msg->m_gid to key.gid does not seem to be correct.

Re: Failed transaction statistics to measure the logical replication progress

2021-11-08 Thread Greg Nancarrow
On Fri, Nov 5, 2021 at 7:11 PM osumi.takami...@fujitsu.com wrote: > > I'm not sure about the last part. > > additionally increase the available subscriber memory, > Which GUC parameter did you mean by this ? > Could we point out and enalrge the memory size only for > subscriber's apply processing

Re: Failed transaction statistics to measure the logical replication progress

2021-11-07 Thread vignesh C
On Fri, Nov 5, 2021 at 1:42 PM osumi.takami...@fujitsu.com wrote: > > On Thursday, November 4, 2021 9:54 AM Greg Nancarrow > wrote: > > On Tue, Nov 2, 2021 at 12:18 AM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Thursday, October 28, 2021 11:19 PM I wrote: > > > > I've created a new

RE: Failed transaction statistics to measure the logical replication progress

2021-11-05 Thread osumi.takami...@fujitsu.com
On Thursday, November 4, 2021 9:54 AM Greg Nancarrow wrote: > On Tue, Nov 2, 2021 at 12:18 AM osumi.takami...@fujitsu.com > wrote: > > > > On Thursday, October 28, 2021 11:19 PM I wrote: > > > I've created a new patch that extends pg_stat_subscription_workers > > > to include other transaction

Re: Failed transaction statistics to measure the logical replication progress

2021-11-03 Thread Greg Nancarrow
On Tue, Nov 2, 2021 at 12:18 AM osumi.takami...@fujitsu.com wrote: > > On Thursday, October 28, 2021 11:19 PM I wrote: > > I've created a new patch that extends pg_stat_subscription_workers to > > include > > other transaction statistics. > > > > Note that this patch depends on v18 patch-set in

RE: Failed transaction statistics to measure the logical replication progress

2021-11-02 Thread osumi.takami...@fujitsu.com
On Monday, November 1, 2021 10:18 PM I wrote: > On Thursday, October 28, 2021 11:19 PM I wrote: > > I've created a new patch that extends pg_stat_subscription_workers to > > include other transaction statistics. > > > > Note that this patch depends on v18 patch-set in [1]... > Rebased based on the

RE: Failed transaction statistics to measure the logical replication progress

2021-11-01 Thread osumi.takami...@fujitsu.com
On Thursday, October 28, 2021 11:19 PM I wrote: > I've created a new patch that extends pg_stat_subscription_workers to include > other transaction statistics. > > Note that this patch depends on v18 patch-set in [1]... Rebased based on the v19 in [1]. Also, fixed documentation a little and made

RE: Failed transaction statistics to measure the logical replication progress

2021-10-28 Thread osumi.takami...@fujitsu.com
On Thursday, October 21, 2021 10:19 AM Masahiko Sawada wrote: > On Mon, Oct 18, 2021 at 7:03 PM Amit Kapila > wrote: > > > > On Thu, Oct 14, 2021 at 9:23 AM houzj.f...@fujitsu.com > > wrote: > > > > > > On Thursday, September 30, 2021 12:15 PM Amit Kapila > > > > > > > > > > > These all views

Re: Failed transaction statistics to measure the logical replication progress

2021-10-21 Thread Amit Kapila
On Thu, Oct 21, 2021 at 6:49 AM Masahiko Sawada wrote: > > On Mon, Oct 18, 2021 at 7:03 PM Amit Kapila wrote: > > > > On Thu, Oct 14, 2021 at 9:23 AM houzj.f...@fujitsu.com > > wrote: > > > > > > On Thursday, September 30, 2021 12:15 PM Amit Kapila > > > > > > > > > > > These all views are

Re: Failed transaction statistics to measure the logical replication progress

2021-10-20 Thread Masahiko Sawada
On Mon, Oct 18, 2021 at 7:03 PM Amit Kapila wrote: > > On Thu, Oct 14, 2021 at 9:23 AM houzj.f...@fujitsu.com > wrote: > > > > On Thursday, September 30, 2021 12:15 PM Amit Kapila > > > > > > > > These all views are related to untransmitted to the collector but what > > > we really need is a

RE: Failed transaction statistics to measure the logical replication progress

2021-10-18 Thread osumi.takami...@fujitsu.com
On Monday, October 18, 2021 11:52 AM Hou, Zhijie/侯 志杰 wrote: > On Thursday, October 14, 2021 2:13 PM Osumi, Takamichi wrote: > > On Thursday, October 14, 2021 12:54 PM Hou, > Zhijie wrote: > > > On Thursday, September 30, 2021 12:15 PM Amit Kapila > > > > On Thu, Sep 30, 2021 at 8:22 AM Hou,

RE: Failed transaction statistics to measure the logical replication progress

2021-10-18 Thread houzj.f...@fujitsu.com
On Monday, October 18, 2021 6:04 PM Amit Kapila wrote: > On Thu, Oct 14, 2021 at 9:23 AM houzj.f...@fujitsu.com > wrote: > > > > On Thursday, September 30, 2021 12:15 PM Amit Kapila > > > > > > > > These all views are related to untransmitted to the collector but > > > what we really need is a

Re: Failed transaction statistics to measure the logical replication progress

2021-10-18 Thread Amit Kapila
On Thu, Oct 14, 2021 at 9:23 AM houzj.f...@fujitsu.com wrote: > > On Thursday, September 30, 2021 12:15 PM Amit Kapila > > > > These all views are related to untransmitted to the collector but what > > we really need is a view similar to pg_stat_archiver or > > pg_stat_bgwriter which gives

RE: Failed transaction statistics to measure the logical replication progress

2021-10-17 Thread houzj.f...@fujitsu.com
On Thursday, October 14, 2021 2:13 PM Osumi, Takamichi wrote: > On Thursday, October 14, 2021 12:54 PM Hou, Zhijie > wrote: > > On Thursday, September 30, 2021 12:15 PM Amit Kapila > > > On Thu, Sep 30, 2021 at 8:22 AM Hou, Zhijie/侯 志杰wrote: > > > > > > > > On Tues, Sep 28, 2021 6:05 PM Amit

RE: Failed transaction statistics to measure the logical replication progress

2021-10-14 Thread osumi.takami...@fujitsu.com
On Thursday, October 14, 2021 12:54 PM Hou, Zhijie/侯 志杰 wrote: > On Thursday, September 30, 2021 12:15 PM Amit Kapila > > > On Thu, Sep 30, 2021 at 8:22 AM Hou, Zhijie/侯 志杰 > wrote: > > > > > > On Tues, Sep 28, 2021 6:05 PM Amit Kapila > wrote: > > > > > > > > Can't we keep the current and

RE: Failed transaction statistics to measure the logical replication progress

2021-10-13 Thread houzj.f...@fujitsu.com
On Thursday, September 30, 2021 12:15 PM Amit Kapila > On Thu, Sep 30, 2021 at 8:22 AM Hou, Zhijie/侯 志杰 > wrote: > > > > On Tues, Sep 28, 2021 6:05 PM Amit Kapila wrote: > > > > > > Can't we keep the current and new stats both in-memory and persist on > > > disk? So, the persistent stats data

RE: Failed transaction statistics to measure the logical replication progress

2021-10-01 Thread osumi.takami...@fujitsu.com
On Thursday, September 30, 2021 8:13 PM Amit Kapila wrote: > On Thu, Sep 30, 2021 at 1:02 PM Osumi, Takamichi/大墨 昂道 > wrote: > > > > On Thursday, September 30, 2021 1:15 PM Amit Kapila > wrote: > > > On Thu, Sep 30, 2021 at 8:22 AM Hou, Zhijie/侯 志杰 > > > wrote: > > > > > > > > On Tues, Sep

  1   2   >