Re: [COMMITTERS] pgsql: Logical replication support for initial data copy
On Fri, Mar 24, 2017 at 12:59 AM, Peter Eisentraut wrote: > On 3/23/17 10:25, Petr Jelinek wrote: >> On 23/03/17 15:17, Stephen Frost wrote: >>> Petr, >>> >>> * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: On 23/03/17 14:47, Petr Jelinek wrote: > I am looking into buildfarm failures. > > Looks like the clang issue (mylodon, longfin) is because we are missing > extern in include/replication/worker_internal.h for ApplyCacheContext. > > Still analyzing the rest. Ah missing fields in copy/equal function for AlterSubscriptionStmt. Attached patch adds the above mentioned extern and fixes the copy and equal functions. >>> >>> Pretty sure I saw a complaint about a trailing comma in a struct or enum >>> too.. >>> >> >> Ah, missed that in the wall of text, thanks, fixed that too. > > Also fixed. > The copyright in some files(tablesync.c and pg_subscriptiion_rel.h) introduced by this commit are old. Is that intentional? Attached patch. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center update_copyright.patch Description: Binary data -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Logical replication support for initial data copy
On 3/23/17 10:25, Petr Jelinek wrote: > On 23/03/17 15:17, Stephen Frost wrote: >> Petr, >> >> * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >>> On 23/03/17 14:47, Petr Jelinek wrote: I am looking into buildfarm failures. Looks like the clang issue (mylodon, longfin) is because we are missing extern in include/replication/worker_internal.h for ApplyCacheContext. Still analyzing the rest. >>> >>> Ah missing fields in copy/equal function for AlterSubscriptionStmt. >>> >>> Attached patch adds the above mentioned extern and fixes the copy and >>> equal functions. >> >> Pretty sure I saw a complaint about a trailing comma in a struct or enum >> too.. >> > > Ah, missed that in the wall of text, thanks, fixed that too. Also fixed. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Logical replication support for initial data copy
On 3/23/17 10:12, Petr Jelinek wrote: > On 23/03/17 14:47, Petr Jelinek wrote: >> I am looking into buildfarm failures. >> >> Looks like the clang issue (mylodon, longfin) is because we are missing >> extern in include/replication/worker_internal.h for ApplyCacheContext. >> >> Still analyzing the rest. >> > Ah missing fields in copy/equal function for AlterSubscriptionStmt. > > Attached patch adds the above mentioned extern and fixes the copy and > equal functions. Fixed. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Logical replication support for initial data copy
On 23/03/17 15:17, Stephen Frost wrote: > Petr, > > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> On 23/03/17 14:47, Petr Jelinek wrote: >>> I am looking into buildfarm failures. >>> >>> Looks like the clang issue (mylodon, longfin) is because we are missing >>> extern in include/replication/worker_internal.h for ApplyCacheContext. >>> >>> Still analyzing the rest. >> >> Ah missing fields in copy/equal function for AlterSubscriptionStmt. >> >> Attached patch adds the above mentioned extern and fixes the copy and >> equal functions. > > Pretty sure I saw a complaint about a trailing comma in a struct or enum > too.. > Ah, missed that in the wall of text, thanks, fixed that too. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 67c7de6..93d4eb2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -4450,7 +4450,10 @@ _copyAlterSubscriptionStmt(const AlterSubscriptionStmt *from) { AlterSubscriptionStmt *newnode = makeNode(AlterSubscriptionStmt); + COPY_SCALAR_FIELD(kind); COPY_STRING_FIELD(subname); + COPY_STRING_FIELD(conninfo); + COPY_NODE_FIELD(publication); COPY_NODE_FIELD(options); return newnode; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 4d8e7fe..6b40b56 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2199,7 +2199,10 @@ static bool _equalAlterSubscriptionStmt(const AlterSubscriptionStmt *a, const AlterSubscriptionStmt *b) { + COMPARE_SCALAR_FIELD(kind); COMPARE_STRING_FIELD(subname); + COMPARE_STRING_FIELD(conninfo); + COMPARE_NODE_FIELD(publication); COMPARE_NODE_FIELD(options); return true; diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index fb55c30..cc31bc6 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -175,7 +175,7 @@ typedef enum WALRCV_OK_TUPLES, /* Query returned tuples. */ WALRCV_OK_COPY_IN, /* Query started COPY FROM. */ WALRCV_OK_COPY_OUT, /* Query started COPY TO. */ - WALRCV_OK_COPY_BOTH, /* Query started COPY BOTH replication protocol. */ + WALRCV_OK_COPY_BOTH /* Query started COPY BOTH replication protocol. */ } WalRcvExecStatus; /* diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index bf96d34..5bebca3 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -46,7 +46,7 @@ typedef struct LogicalRepWorker } LogicalRepWorker; /* Memory context for cached variables in apply worker. */ -MemoryContext ApplyCacheContext; +extern MemoryContextApplyCacheContext; /* libpqreceiver connection */ extern struct WalReceiverConn *wrconn; -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Logical replication support for initial data copy
Petr, * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: > On 23/03/17 14:47, Petr Jelinek wrote: > > I am looking into buildfarm failures. > > > > Looks like the clang issue (mylodon, longfin) is because we are missing > > extern in include/replication/worker_internal.h for ApplyCacheContext. > > > > Still analyzing the rest. > > Ah missing fields in copy/equal function for AlterSubscriptionStmt. > > Attached patch adds the above mentioned extern and fixes the copy and > equal functions. Pretty sure I saw a complaint about a trailing comma in a struct or enum too.. Thanks! Stephen signature.asc Description: Digital signature
Re: [COMMITTERS] pgsql: Logical replication support for initial data copy
On 23/03/17 14:47, Petr Jelinek wrote: > I am looking into buildfarm failures. > > Looks like the clang issue (mylodon, longfin) is because we are missing > extern in include/replication/worker_internal.h for ApplyCacheContext. > > Still analyzing the rest. > Ah missing fields in copy/equal function for AlterSubscriptionStmt. Attached patch adds the above mentioned extern and fixes the copy and equal functions. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 67c7de6..93d4eb2 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -4450,7 +4450,10 @@ _copyAlterSubscriptionStmt(const AlterSubscriptionStmt *from) { AlterSubscriptionStmt *newnode = makeNode(AlterSubscriptionStmt); + COPY_SCALAR_FIELD(kind); COPY_STRING_FIELD(subname); + COPY_STRING_FIELD(conninfo); + COPY_NODE_FIELD(publication); COPY_NODE_FIELD(options); return newnode; diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index 4d8e7fe..6b40b56 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2199,7 +2199,10 @@ static bool _equalAlterSubscriptionStmt(const AlterSubscriptionStmt *a, const AlterSubscriptionStmt *b) { + COMPARE_SCALAR_FIELD(kind); COMPARE_STRING_FIELD(subname); + COMPARE_STRING_FIELD(conninfo); + COMPARE_NODE_FIELD(publication); COMPARE_NODE_FIELD(options); return true; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index bf96d34..5bebca3 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -46,7 +46,7 @@ typedef struct LogicalRepWorker } LogicalRepWorker; /* Memory context for cached variables in apply worker. */ -MemoryContext ApplyCacheContext; +extern MemoryContextApplyCacheContext; /* libpqreceiver connection */ extern struct WalReceiverConn *wrconn; -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Logical replication support for initial data copy
I am looking into buildfarm failures. Looks like the clang issue (mylodon, longfin) is because we are missing extern in include/replication/worker_internal.h for ApplyCacheContext. Still analyzing the rest. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Logical replication support for initial data copy
Logical replication support for initial data copy Add functionality for a new subscription to copy the initial data in the tables and then sync with the ongoing apply process. For the copying, add a new internal COPY option to have the COPY source data provided by a callback function. The initial data copy works on the subscriber by receiving COPY data from the publisher and then providing it locally into a COPY that writes to the destination table. A WAL receiver can now execute full SQL commands. This is used here to obtain information about tables and publications. Several new options were added to CREATE and ALTER SUBSCRIPTION to control whether and when initial table syncing happens. Change pg_dump option --no-create-subscription-slots to --no-subscription-connect and use the new CREATE SUBSCRIPTION ... NOCONNECT option for that. Author: Petr Jelinek Tested-by: Erik Rijkers Branch -- master Details --- http://git.postgresql.org/pg/commitdiff/7c4f52409a8c7d85ed169bbbc1f6092274d03920 Modified Files -- contrib/file_fdw/file_fdw.c| 5 +- doc/src/sgml/catalogs.sgml | 78 ++ doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 55 +- doc/src/sgml/monitoring.sgml | 9 +- doc/src/sgml/protocol.sgml | 9 +- doc/src/sgml/ref/alter_subscription.sgml | 50 +- doc/src/sgml/ref/create_subscription.sgml | 38 + doc/src/sgml/ref/pg_dump.sgml | 15 +- src/backend/catalog/Makefile | 1 + src/backend/catalog/heap.c | 6 + src/backend/catalog/pg_publication.c | 4 +- src/backend/catalog/pg_subscription.c | 282 +++ src/backend/catalog/system_views.sql | 1 + src/backend/commands/copy.c| 23 +- src/backend/commands/subscriptioncmds.c| 468 ++-- src/backend/parser/gram.y | 39 +- src/backend/postmaster/pgstat.c| 6 + .../libpqwalreceiver/libpqwalreceiver.c| 203 - src/backend/replication/logical/Makefile | 2 +- src/backend/replication/logical/launcher.c | 130 +++- src/backend/replication/logical/relation.c | 7 + src/backend/replication/logical/snapbuild.c| 85 ++- src/backend/replication/logical/tablesync.c| 840 + src/backend/replication/logical/worker.c | 203 +++-- src/backend/replication/repl_gram.y| 32 +- src/backend/replication/repl_scanner.l | 5 +- src/backend/replication/walsender.c| 104 ++- src/backend/tcop/postgres.c| 5 +- src/backend/utils/adt/misc.c | 20 + src/backend/utils/cache/syscache.c | 14 +- src/backend/utils/misc/guc.c | 12 + src/bin/pg_dump/pg_backup.h| 2 +- src/bin/pg_dump/pg_dump.c | 9 +- src/bin/pg_dump/t/002_pg_dump.pl | 2 +- src/include/catalog/catversion.h | 2 +- src/include/catalog/indexing.h | 7 +- src/include/catalog/pg_proc.h | 5 +- src/include/catalog/pg_subscription_rel.h | 78 ++ src/include/commands/copy.h| 5 +- src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 13 + src/include/nodes/replnodes.h | 9 + src/include/parser/kwlist.h| 1 + src/include/pgstat.h | 4 +- src/include/replication/logical.h | 13 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/snapbuild.h| 1 + src/include/replication/walreceiver.h | 67 +- src/include/replication/walsender.h| 12 +- src/include/replication/worker_internal.h | 30 +- src/include/utils/syscache.h | 1 + src/test/regress/expected/object_address.out | 3 +- src/test/regress/expected/rules.out| 3 +- src/test/regress/expected/sanity_check.out | 1 + src/test/regress/expected/subscription.out | 45 +- src/test/regress/sql/object_address.sql| 2 +- src/test/regress/sql/subscription.sql | 11 +- src/test/subscription/t/001_rep_changes.pl | 36 +- src/test/subscription/t/002_types.pl | 6 + src/test/subscription/t/003_constraints.pl | 2 +- src/test/subscription/t/004_sync.pl| 159 62 files changed, 2966 insertions(+), 341 deletions(-) -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/
Re: [COMMITTERS] pgsql: Logical replication
On 2017-01-21 09:49:28 +0530, Amit Kapila wrote: > On Fri, Jan 20, 2017 at 7:36 PM, Peter Eisentraut wrote: > > Logical replication > > > > - Add PUBLICATION catalogs and DDL > > - Add SUBSCRIPTION catalog and DDL > > - Define logical replication protocol and output plugin > > - Add logical replication workers > > > > From: Petr Jelinek > > Reviewed-by: Steve Singer > > Reviewed-by: Andres Freund > > Reviewed-by: Erik Rijkers > > Reviewed-by: Peter Eisentraut > Great work, Congrats Peter Jelinek and Thanks to all involved. > Getting a feature of this magnitude deserves a big round of applause. Seconded! Or fourthed ;) -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Logical replication
2017-01-21 6:56 GMT+01:00 Amit Langote : > On Sat, Jan 21, 2017 at 1:19 PM, Amit Kapila > wrote: > > On Fri, Jan 20, 2017 at 7:36 PM, Peter Eisentraut > wrote: > >> Logical replication > >> > >> - Add PUBLICATION catalogs and DDL > >> - Add SUBSCRIPTION catalog and DDL > >> - Define logical replication protocol and output plugin > >> - Add logical replication workers > >> > >> From: Petr Jelinek > >> Reviewed-by: Steve Singer > >> Reviewed-by: Andres Freund > >> Reviewed-by: Erik Rijkers > >> Reviewed-by: Peter Eisentraut > >> > >> > > .. > >> 119 files changed, 13354 insertions(+), 95 deletions(-) > >> > > > > > > Great work, Congrats Peter Jelinek and Thanks to all involved. > > Getting a feature of this magnitude deserves a big round of applause. > > +1, congrats! > +1 pretty big work is done! Pavel > > Thanks, > Amit > > > -- > Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-committers >
Re: [COMMITTERS] pgsql: Logical replication
On Sat, Jan 21, 2017 at 1:19 PM, Amit Kapila wrote: > On Fri, Jan 20, 2017 at 7:36 PM, Peter Eisentraut wrote: >> Logical replication >> >> - Add PUBLICATION catalogs and DDL >> - Add SUBSCRIPTION catalog and DDL >> - Define logical replication protocol and output plugin >> - Add logical replication workers >> >> From: Petr Jelinek >> Reviewed-by: Steve Singer >> Reviewed-by: Andres Freund >> Reviewed-by: Erik Rijkers >> Reviewed-by: Peter Eisentraut >> >> > .. >> 119 files changed, 13354 insertions(+), 95 deletions(-) >> > > > Great work, Congrats Peter Jelinek and Thanks to all involved. > Getting a feature of this magnitude deserves a big round of applause. +1, congrats! Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Logical replication
On Fri, Jan 20, 2017 at 7:36 PM, Peter Eisentraut wrote: > Logical replication > > - Add PUBLICATION catalogs and DDL > - Add SUBSCRIPTION catalog and DDL > - Define logical replication protocol and output plugin > - Add logical replication workers > > From: Petr Jelinek > Reviewed-by: Steve Singer > Reviewed-by: Andres Freund > Reviewed-by: Erik Rijkers > Reviewed-by: Peter Eisentraut > > .. > 119 files changed, 13354 insertions(+), 95 deletions(-) > Great work, Congrats Peter Jelinek and Thanks to all involved. Getting a feature of this magnitude deserves a big round of applause. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Logical replication
* Peter Eisentraut (pete...@gmx.net) wrote: > Logical replication > > - Add PUBLICATION catalogs and DDL > - Add SUBSCRIPTION catalog and DDL > - Define logical replication protocol and output plugin > - Add logical replication workers The buildfarm is rather upset with this. Looks like an issue in pg_upgrade / pg_dump. Thanks! Stephen signature.asc Description: Digital signature
[COMMITTERS] pgsql: Logical replication
Logical replication - Add PUBLICATION catalogs and DDL - Add SUBSCRIPTION catalog and DDL - Define logical replication protocol and output plugin - Add logical replication workers From: Petr Jelinek Reviewed-by: Steve Singer Reviewed-by: Andres Freund Reviewed-by: Erik Rijkers Reviewed-by: Peter Eisentraut Branch -- master Details --- http://git.postgresql.org/pg/commitdiff/665d1fad99e7b11678b0d5fa24d2898424243cd6 Modified Files -- doc/src/sgml/catalogs.sgml | 309 + doc/src/sgml/config.sgml | 41 + doc/src/sgml/filelist.sgml |1 + doc/src/sgml/func.sgml |2 +- doc/src/sgml/logical-replication.sgml | 396 ++ doc/src/sgml/monitoring.sgml | 74 + doc/src/sgml/postgres.sgml |1 + doc/src/sgml/protocol.sgml | 721 ++ doc/src/sgml/ref/allfiles.sgml |6 + doc/src/sgml/ref/alter_publication.sgml| 139 ++ doc/src/sgml/ref/alter_subscription.sgml | 139 ++ doc/src/sgml/ref/create_publication.sgml | 206 +++ doc/src/sgml/ref/create_subscription.sgml | 176 +++ doc/src/sgml/ref/drop_publication.sgml | 107 ++ doc/src/sgml/ref/drop_subscription.sgml| 110 ++ doc/src/sgml/ref/pg_dump.sgml | 21 + doc/src/sgml/ref/psql-ref.sgml | 28 + doc/src/sgml/reference.sgml|6 + src/Makefile |1 + src/backend/access/transam/xact.c |2 + src/backend/catalog/Makefile |7 +- src/backend/catalog/aclchk.c | 57 + src/backend/catalog/catalog.c |8 +- src/backend/catalog/dependency.c | 24 + src/backend/catalog/objectaddress.c| 219 +++ src/backend/catalog/pg_publication.c | 457 +++ src/backend/catalog/pg_shdepend.c | 11 + src/backend/catalog/pg_subscription.c | 207 +++ src/backend/catalog/system_views.sql | 25 + src/backend/commands/Makefile |8 +- src/backend/commands/alter.c | 10 + src/backend/commands/dbcommands.c | 17 + src/backend/commands/define.c | 28 + src/backend/commands/dropcmds.c|4 + src/backend/commands/event_trigger.c |8 + src/backend/commands/publicationcmds.c | 754 +++ src/backend/commands/subscriptioncmds.c| 643 + src/backend/commands/tablecmds.c | 12 + src/backend/executor/Makefile |2 +- src/backend/executor/execMain.c|3 +- src/backend/executor/execReplication.c | 553 src/backend/nodes/copyfuncs.c | 78 ++ src/backend/nodes/equalfuncs.c | 73 + src/backend/parser/gram.y | 262 +++- src/backend/postmaster/bgworker.c | 23 +- src/backend/postmaster/pgstat.c|6 + src/backend/postmaster/postmaster.c|9 + .../libpqwalreceiver/libpqwalreceiver.c| 217 ++- src/backend/replication/logical/Makefile |4 +- src/backend/replication/logical/launcher.c | 759 +++ src/backend/replication/logical/proto.c| 637 + src/backend/replication/logical/relation.c | 489 +++ src/backend/replication/logical/worker.c | 1429 src/backend/replication/pgoutput/Makefile | 32 + src/backend/replication/pgoutput/pgoutput.c| 596 src/backend/replication/walreceiver.c | 20 +- src/backend/storage/ipc/ipci.c |3 + src/backend/storage/lmgr/lwlocknames.txt |2 + src/backend/tcop/utility.c | 83 ++ src/backend/utils/cache/inval.c| 35 +- src/backend/utils/cache/relcache.c | 136 +- src/backend/utils/cache/syscache.c | 69 + src/backend/utils/misc/guc.c | 13 + src/backend/utils/misc/postgresql.conf.sample |1 + src/bin/pg_dump/common.c | 12 + src/bin/pg_dump/pg_backup.h|3 + src/bin/pg_dump/pg_backup_archiver.c |7 +- src/bin/pg_dump/pg_dump.c | 464 +++ src/bin/pg_dump/pg_dump.h | 46 +- src/bin/pg_dump/pg_dump_sort.c | 20 +- src/bin/pg_dump/pg_restore.c |3 + src/bin/pg_dump/t/002_pg_dump.pl | 76 +- src/bin/psql/command.c | 16 + src/bin/psql/d