On 21.02.22 13:09, Euler Taveira wrote:
A new tool called pg_subscriber does this conversion and is tightly
integrated
with Postgres.
Are we comfortable with the name pg_subscriber? It seems too general.
Are we planning other subscriber-related operations in the future? If
so, we should at least make this one use a --create option or
something like that.
doc/src/sgml/ref/pg_subscriber.sgml
Attached is a patch that reorganizes the man page a bit. I moved the
description of the steps to the Notes section and formatted it
differently. I think the steps are interesting but not essential for
the using of the program, so I wanted to get them out of the main
description.
src/bin/pg_subscriber/pg_subscriber.c
+ if (made_temp_replslot)
+ {
+ conn = connect_database(dbinfo[0].pubconninfo, true);
+ drop_replication_slot(conn, &dbinfo[0], temp_replslot);
+ disconnect_database(conn);
+ }
Temp slots don't need to be cleaned up.
+/*
+ * Obtain the system identifier from control file. It will be used to
compare
+ * if a data directory is a clone of another one. This routine is used
locally
+ * and avoids a replication connection.
+ */
+static char *
+get_control_from_datadir(const char *datadir)
This could return uint64 directly, without string conversion.
get_sysid_from_conn() could then convert to uint64 internally.
+ {"verbose", no_argument, NULL, 'v'},
I'm not sure if the --verbose option is all that useful.
+ {"stop-subscriber", no_argument, NULL, 1},
This option doesn't seem to be otherwise supported or documented.
+ pub_sysid = pg_malloc(32);
+ pub_sysid = get_sysid_from_conn(dbinfo[0].pubconninfo);
+ sub_sysid = pg_malloc(32);
+ sub_sysid = get_control_from_datadir(subscriber_dir);
These mallocs don't appears to be of any use.
+ dbname_conninfo = pg_malloc(NAMEDATALEN);
This seems wrong.
Overall, this code could use a little bit more structuring. There are
a lot of helper functions that don't seem to do a lot and are mostly
duplicate runs-this-SQL-command calls. But the main() function is
still huge. There is room for refinement.
src/bin/pg_subscriber/t/001_basic.pl
Good start, but obviously, we'll need some real test cases here also.
src/bin/initdb/initdb.c
src/bin/pg_ctl/pg_ctl.c
src/common/file_utils.c
src/include/common/file_utils.h
I recommend skipping this refactoring. The readfile() function from
pg_ctl is not general enough to warrant the pride of place of a
globally available function. Note that it is specifically geared
toward some of pg_ctl's requirements, for example that the underlying
file can change while it is being read.
The requirements of pg_subscriber can be satisfied more easily: Just
call pg_ctl to start the server. You are already using that in
pg_subscriber. Is there a reason it can't be used here as well?From a0ad5fdaddc17ef74594bfd3c65c777649d1544b Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pe...@eisentraut.org>
Date: Tue, 15 Mar 2022 14:39:19 +0100
Subject: [PATCH] fixup! Create a new logical replica from a base backup or
standby server.
---
doc/src/sgml/ref/pg_subscriber.sgml | 172 ++++++++++++++++------------
1 file changed, 99 insertions(+), 73 deletions(-)
diff --git a/doc/src/sgml/ref/pg_subscriber.sgml
b/doc/src/sgml/ref/pg_subscriber.sgml
index e68a19092e..df63c6a993 100644
--- a/doc/src/sgml/ref/pg_subscriber.sgml
+++ b/doc/src/sgml/ref/pg_subscriber.sgml
@@ -43,79 +43,6 @@ <title>Description</title>
replication connections from the target server (known as subscriber server).
The target server should accept local logical replication connection.
</para>
-
- <para>
- The transformation proceeds in eight steps. First,
- <application>pg_subscriber</application> checks if the given target data
- directory has the same system identifier than the source data directory.
- Since it uses the recovery process as one of the steps, it starts the target
- server as a replica from the source server. If the system identifier is not
- the same, <application>pg_subscriber</application> will terminate with an
- error.
- </para>
-
- <para>
- Second, <application>pg_subscriber</application> checks if the target data
- directory is used by a standby server. Stop the standby server if it is
- running. One of the next steps is to add some recovery parameters that
- requires a server start. This step avoids an error.
- </para>
-
- <para>
- Next, <application>pg_subscriber</application> creates one replication slot
- for each specified database on the source server. The replication slot name
- contains a <literal>pg_subscriber</literal> prefix. These replication slots
- will be used by the subscriptions in a future step. Another replication
- slot is used to get a consistent start location. This consistent LSN will be
- used (a) as a stopping point in the <xref
- linkend="guc-recovery-target-lsn"/> parameter and (b) by the subscriptions
- as a replication starting point. It guarantees that no transaction will be
- lost.
- </para>
-
- <para>
- Next, write recovery parameters into the target data directory and start the
- target server. It specifies a LSN (consistent LSN that was obtained in the
- previous step) of write-ahead log location up to which recovery will
- proceed. It also specifies <literal>promote</literal> as the action that the
- server should take once the recovery target is reached. This step finishes
- once the server ends standby mode and is accepting read-write operations.
- </para>
-
- <para>
- Next, <application>pg_subscriber</application> creates one publication for
- each specified database on the source server. Each publication replicates
- changes for all tables in the database. The publication name contains a
- <literal>pg_subscriber</literal> prefix. These publication will be used by a
- corresponding subscription in a next step.
- </para>
-
- <para>
- Next, <application>pg_subscriber</application> creates one subscription for
- each specified database on the target server. Each subscription name
- contains a <literal>pg_subscriber</literal> prefix. The replication slot
- name is identical to the subscription name. It also does not copy existing
- data from the source server. It does not create a replication slot. Instead,
- it uses the replication slot that was created in a previous step. The
- subscription is created but it is not enabled yet. The reason is the
- replication progress must be set to the consistent LSN but replication
- origin name contains the subscription oid in its name. Hence, the
- subscription will be enabled in a separate step.
- </para>
-
- <para>
- Next, <application>pg_subscriber</application> sets the replication progress
- to the consistent LSN that was obtained in a previous step. When the target
- server started the recovery process, it caught up to the consistent LSN.
- This is the exact LSN to be used as a initial location for the logical
- replication.
- </para>
-
- <para>
- Finally, <application>pg_subscriber</application> enables the subscription
- for each specified database on the target server. The subscription starts
- streaming from the consistent LSN.
- </para>
</refsect1>
<refsect1>
@@ -209,6 +136,105 @@ <title>Options</title>
</refsect1>
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ The transformation proceeds in the following steps:
+ </para>
+
+ <procedure>
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the given target data
+ directory has the same system identifier than the source data directory.
+ Since it uses the recovery process as one of the steps, it starts the
+ target server as a replica from the source server. If the system
+ identifier is not the same, <application>pg_subscriber</application> will
+ terminate with an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> checks if the target data
+ directory is used by a standby server. Stop the standby server if it is
+ running. One of the next steps is to add some recovery parameters that
+ requires a server start. This step avoids an error.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one replication slot for
+ each specified database on the source server. The replication slot name
+ contains a <literal>pg_subscriber</literal> prefix. These replication
+ slots will be used by the subscriptions in a future step. Another
+ replication slot is used to get a consistent start location. This
+ consistent LSN will be used (a) as a stopping point in the <xref
+ linkend="guc-recovery-target-lsn"/> parameter and (b) by the
+ subscriptions as a replication starting point. It guarantees that no
+ transaction will be lost.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> writes recovery parameters into
+ the target data directory and start the target server. It specifies a LSN
+ (consistent LSN that was obtained in the previous step) of write-ahead
+ log location up to which recovery will proceed. It also specifies
+ <literal>promote</literal> as the action that the server should take once
+ the recovery target is reached. This step finishes once the server ends
+ standby mode and is accepting read-write operations.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Next, <application>pg_subscriber</application> creates one publication
+ for each specified database on the source server. Each publication
+ replicates changes for all tables in the database. The publication name
+ contains a <literal>pg_subscriber</literal> prefix. These publication
+ will be used by a corresponding subscription in a next step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> creates one subscription for
+ each specified database on the target server. Each subscription name
+ contains a <literal>pg_subscriber</literal> prefix. The replication slot
+ name is identical to the subscription name. It also does not copy
+ existing data from the source server. It does not create a replication
+ slot. Instead, it uses the replication slot that was created in a
+ previous step. The subscription is created but it is not enabled yet. The
+ reason is the replication progress must be set to the consistent LSN but
+ replication origin name contains the subscription oid in its name. Hence,
+ the subscription will be enabled in a separate step.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ <application>pg_subscriber</application> sets the replication progress to
+ the consistent LSN that was obtained in a previous step. When the target
+ server started the recovery process, it caught up to the consistent LSN.
+ This is the exact LSN to be used as a initial location for the logical
+ replication.
+ </para>
+ </step>
+
+ <step>
+ <para>
+ Finally, <application>pg_subscriber</application> enables the subscription
+ for each specified database on the target server. The subscription starts
+ streaming from the consistent LSN.
+ </para>
+ </step>
+ </procedure>
+ </refsect1>
+
<refsect1>
<title>Examples</title>
--
2.35.1
