On Wed, Nov 18, 2026 at 4:54 AM Zsolt Parragi <[email protected]>
wrote:

> Hello
>
> 1.
>
> > Subject: [PATCH v2 2/2] Add TAP test for primary shutdown during
> streaming
>
> This should also have a meson.build edit so the meson builds also run
> the new test
>
> 2. Stopping the primary and waiting a bit gives this on the standby:
>
> LOG:  replication terminated by primary server
> DETAIL:  End of WAL reached on timeline 1 at 0/030000B0.
> FATAL:  could not receive database system identifier and timeline ID
> from the primary server: no connection to the server
>
> After the early return walreceiver waits for new orders and then runs
> IDENTIFY_SYSTEM on the closed connection. The test passes because it
> only checks that the old message is gone.
>
> 3. On master, a clean publisher shutdown hits the same error in the
> logical apply worker, and that disables a disable_on_error
> subscription:
>
> LOG: data stream from publisher has ended
> ERROR: could not send end-of-streaming message to primary: server
> closed the connection unexpectedly
> LOG: subscription "sub" has been disabled because of an error
>
> With the patch only the first line remains and the subscription stays
> enabled. This doesn't seem like a bad thing, but probably should be
> mentioned at least in the commit message, and maybe tested too?
>
> 4. Partial WAL segment archived on archive_mode=always standbys
>
> Since endstreaming no longer fails, walreceiver continues into the
> end-of-timeline block in WalReceiverMain(), which closes the current
> segment and calls XLogArchiveNotify() on it, even though the segment
> is incomplete.
>
> Setup:
>
> * standby with archive_mode = always and archive_command = 'test ! -f
> arch/%f && cp %p arch/%f'
> * stop the primary
> * wait a few seconds
> * start it again
> * run pg_switch_wal() a few times:
>
> master:  archived while primary down: (none)
>          pg_stat_archiver: 5|000000010000000000000007|0|
> patched: archived while primary down: 000000010000000000000003
>          pg_stat_archiver:
> 1|000000010000000000000003|6|000000010000000000000003
>
> 5. Would something like this work instead?
>
>  libpqrcv_endstreaming():
> +       *next_tli = 0;
> +       if (PQstatus(conn->streamConn) == CONNECTION_BAD)
> +               return;
>  walreceiver.c, after walrcv_endstreaming(wrconn, &primaryTLI):
> +       if (primaryTLI == 0)
> +               proc_exit(0);
>


Hi Zsolt,

Thanks for the careful review. Agreed on points 1 to 4. I missed those
follow ons in v2. On point 5 I like the direction, but CONNECTION_BAD
alone does not cover the original report.

After COPY 0, libpq has left COPY while PQstatus() can still be
CONNECTION_OK, so PQputCopyEnd() still fails with no COPY in progress.
That is why copyEnded stays. CONNECTION_BAD is a useful complement for
an abrupt close, so I kept that too.

v3 also does proc_exit(0) in physical walreceiver when endstreaming
reports next_tli == 0. Timeline switch still sends a non zero next TLI
and keeps waiting mode. Clean shutdown does not, and falling through
was what caused the IDENTIFY_SYSTEM FATAL and the incomplete segment
archive under archive_mode=always.

Both TAP tests are in meson.build now, and they cover the endstreaming
FATAL, IDENTIFY_SYSTEM, disable_on_error, incomplete segment archive,
and reconnect after restart.

Attached.

Regards,
Stepan Neretin
From 3ef58af9762947825974b560fae537bb46b04489 Mon Sep 17 00:00:00 2001
From: Stepan Neretin <[email protected]>
Date: Wed, 18 Nov 2026 11:17:03 +0700
Subject: [PATCH v3 2/2] Add TAP tests for primary/publisher shutdown during
 streaming

Verify that a clean remote shutdown does not FATAL on endstreaming,
reuse the dead connection, archive an incomplete segment, or disable a
disable_on_error subscription, and that streaming resumes after
restart.

Discussion: https://postgr.es/m/CA+Yyo5S_n3ZcZh-3LXNj2-rxqXnbGd8901Yres0LPUS=ttl...@mail.gmail.com
---
 src/test/recovery/meson.build                 |  1 +
 .../t/061_primary_shutdown_streaming.pl       | 96 +++++++++++++++++++
 src/test/subscription/meson.build             |  1 +
 .../subscription/t/039_publisher_shutdown.pl  | 71 ++++++++++++++
 4 files changed, 169 insertions(+)
 create mode 100644 src/test/recovery/t/061_primary_shutdown_streaming.pl
 create mode 100644 src/test/subscription/t/039_publisher_shutdown.pl

diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b6ac9dd07a7..489413275d8 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -69,6 +69,7 @@ tests += {
       't/058_shutdown_crash_restart.pl',
       't/059_remote_apply_status_interval.pl',
       't/060_multixact_crashed_updater.pl',
+      't/061_primary_shutdown_streaming.pl',
     ],
   },
 }
diff --git a/src/test/recovery/t/061_primary_shutdown_streaming.pl b/src/test/recovery/t/061_primary_shutdown_streaming.pl
new file mode 100644
index 00000000000..faf8de8f4b3
--- /dev/null
+++ b/src/test/recovery/t/061_primary_shutdown_streaming.pl
@@ -0,0 +1,96 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Clean primary shutdown must not FATAL in endstreaming, reuse the dead
+# connection for IDENTIFY_SYSTEM, or archive an incomplete WAL segment
+# under archive_mode=always. Streaming must resume after restart.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+# pg_walfile_name() is unavailable during recovery. Assumes the default
+# 16MB wal_segment_size used by the TAP suite.
+sub walfile_from_lsn
+{
+	my ($lsn, $tli) = @_;
+	$tli = 1 unless defined $tli;
+	my ($hi_hex, $lo_hex) = split(m{/}, $lsn);
+	my $hi = hex($hi_hex);
+	my $lo = hex($lo_hex);
+	my $segsize = 16 * 1024 * 1024;
+	my $segs_per_id = 256;    # 2^32 / 16MB
+	my $logSegNo = $hi * $segs_per_id + int($lo / $segsize);
+	return sprintf("%08X%08X%08X", $tli, int($logSegNo / $segs_per_id),
+		$logSegNo % $segs_per_id);
+}
+
+my $node_primary = PostgreSQL::Test::Cluster->new('primary');
+$node_primary->init(allows_streaming => 1);
+$node_primary->start;
+$node_primary->backup('backup');
+
+my $node_standby = PostgreSQL::Test::Cluster->new('standby');
+$node_standby->init_from_backup(
+	$node_primary, 'backup',
+	has_streaming => 1);
+# init_from_backup ignores has_archiving. Set up standby archiving by hand.
+$node_standby->enable_archiving;
+$node_standby->append_conf('postgresql.conf', 'archive_mode = always');
+$node_standby->start;
+
+# Finish one segment, then leave the next one incomplete.
+$node_primary->safe_psql('postgres',
+	"CREATE TABLE shutdown_stream_tab AS SELECT generate_series(1, 100) AS id");
+$node_primary->safe_psql('postgres', 'SELECT pg_switch_wal()');
+$node_primary->safe_psql('postgres',
+	"INSERT INTO shutdown_stream_tab VALUES (101)");
+$node_primary->wait_for_catchup($node_standby);
+
+my $log_offset = -s $node_standby->logfile;
+$node_primary->stop;
+
+# Wait until a successor walreceiver fails to connect. That means the old
+# process finished endstreaming and exited.
+$node_standby->wait_for_log(qr/could not connect to the primary server/,
+	$log_offset);
+
+ok( !$node_standby->log_contains(
+		'could not send end-of-streaming message', $log_offset),
+	'walreceiver does not fail ending the stream after primary shutdown');
+ok( !$node_standby->log_contains(
+		'could not receive database system identifier and timeline ID',
+		$log_offset),
+	'walreceiver does not reuse the closed connection for IDENTIFY_SYSTEM');
+
+# Shutdown checkpoint may complete the pre stop segment. Check the one that
+# remains incomplete afterwards.
+my $receive_lsn = $node_standby->safe_psql('postgres',
+	"SELECT pg_last_wal_receive_lsn()");
+my $incomplete_seg = walfile_from_lsn($receive_lsn);
+my $archive_status =
+  $node_standby->data_dir . "/pg_wal/archive_status/$incomplete_seg.ready";
+ok( !-f $archive_status,
+	"incomplete segment $incomplete_seg does not get a .ready file");
+ok( !-f $node_standby->archive_dir . "/$incomplete_seg",
+	"incomplete segment $incomplete_seg is not archived while primary is down");
+
+my $reconnect_offset = -s $node_standby->logfile;
+$node_primary->start;
+$node_standby->wait_for_log(
+	qr/(?:started streaming WAL|restarted WAL streaming)/,
+	$reconnect_offset);
+$node_primary->safe_psql('postgres',
+	"INSERT INTO shutdown_stream_tab VALUES (102)");
+$node_primary->wait_for_catchup($node_standby);
+
+my $result =
+  $node_standby->safe_psql('postgres',
+	"SELECT count(*) FROM shutdown_stream_tab");
+is($result, '102', 'standby reconnects and catches up after primary restart');
+
+$node_standby->stop;
+$node_primary->stop;
+
+done_testing();
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index e71e95c6297..c4d790e40f0 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -48,6 +48,7 @@ tests += {
       't/036_sequences.pl',
       't/037_except.pl',
       't/038_walsnd_shutdown_timeout.pl',
+      't/039_publisher_shutdown.pl',
       't/100_bugs.pl',
     ],
   },
diff --git a/src/test/subscription/t/039_publisher_shutdown.pl b/src/test/subscription/t/039_publisher_shutdown.pl
new file mode 100644
index 00000000000..5daf49856f8
--- /dev/null
+++ b/src/test/subscription/t/039_publisher_shutdown.pl
@@ -0,0 +1,71 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Clean publisher shutdown must not ERROR in endstreaming or disable a
+# disable_on_error subscription. Replication continues after restart.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->start;
+
+$node_publisher->safe_psql(
+	'postgres', qq(
+	CREATE TABLE test_tab (id int PRIMARY KEY);
+	INSERT INTO test_tab VALUES (1);
+	CREATE PUBLICATION test_pub FOR TABLE test_tab;
+));
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_subscriber->safe_psql(
+	'postgres', qq(
+	CREATE TABLE test_tab (id int PRIMARY KEY);
+	CREATE SUBSCRIPTION test_sub
+		CONNECTION '$publisher_connstr'
+		PUBLICATION test_pub
+		WITH (disable_on_error = true);
+));
+
+$node_subscriber->wait_for_subscription_sync($node_publisher, 'test_sub');
+
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->stop;
+
+# Wait until a successor apply worker fails to connect. That means the
+# previous process finished endstreaming with the subscription still enabled.
+$node_subscriber->wait_for_log(
+	qr/apply worker for subscription "test_sub" could not connect to the publisher/,
+	$log_offset);
+
+ok( !$node_subscriber->log_contains(
+		'could not send end-of-streaming message', $log_offset),
+	'apply worker does not fail ending the stream after publisher shutdown');
+ok( !$node_subscriber->log_contains(
+		'subscription "test_sub" has been disabled because of an error',
+		$log_offset),
+	'disable_on_error subscription is not disabled on clean publisher shutdown');
+
+my $enabled = $node_subscriber->safe_psql('postgres',
+	"SELECT subenabled FROM pg_subscription WHERE subname = 'test_sub'");
+is($enabled, 't', 'subscription remains enabled');
+
+$node_publisher->start;
+$node_publisher->safe_psql('postgres', "INSERT INTO test_tab VALUES (2)");
+$node_publisher->wait_for_catchup('test_sub');
+
+my $result = $node_subscriber->safe_psql('postgres',
+	"SELECT count(*) FROM test_tab");
+is($result, '2', 'subscription resumes after publisher restart');
+
+$node_subscriber->stop;
+$node_publisher->stop;
+
+done_testing();
-- 
2.56.0

From 0a20f28fb019020a9ed2e3186fb2373f85d8e0ef Mon Sep 17 00:00:00 2001
From: Stepan Neretin <[email protected]>
Date: Wed, 18 Nov 2026 11:17:03 +0700
Subject: [PATCH v3 1/2] Fix FATAL when primary ends streaming with COPY 0

If the primary shuts down cleanly, walsender ends the COPY with
CommandComplete ("COPY 0") and exits without waiting for CopyDone.
libpqrcv_receive() already consumes that result, then
libpqrcv_endstreaming() still calls PQputCopyEnd() and fails with "no
COPY in progress", which becomes a FATAL in the walreceiver.

Remember that CommandComplete already ended the COPY and skip
PQputCopyEnd() in that case.  Also skip it when the connection is
already CONNECTION_BAD.  Clear the flag in libpqrcv_startstreaming(),
because receive() is also used for tablesync COPY TO STDOUT and that
must not cause a later replication stream on the same connection to
skip CopyDone.

If endstreaming reports no next timeline, exit with proc_exit(0)
instead of falling into waiting mode, so the dead connection is not
reused and an incomplete segment is not archived.

Discussion: https://postgr.es/m/CA+Yyo5S_n3ZcZh-3LXNj2-rxqXnbGd8901Yres0LPUS=ttl...@mail.gmail.com
---
 .../libpqwalreceiver/libpqwalreceiver.c       | 19 +++++++++++++++++++
 src/backend/replication/walreceiver.c         | 11 +++++++++++
 2 files changed, 30 insertions(+)

diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index dcc738f4e38..27eb2fd5a40 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -52,6 +52,8 @@ struct WalReceiverConn
 	bool		logical;
 	/* Buffer for currently read records */
 	char	   *recvBuf;
+	/* set when CommandComplete already ended the COPY */
+	bool		copyEnded;
 };
 
 /* Prototypes for interface functions */
@@ -618,6 +620,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
 	Assert(options->logical == conn->logical);
 	Assert(options->slotname || !options->logical);
 
+	/* may be left set by a prior non streaming COPY */
+	conn->copyEnded = false;
+
 	initStringInfo(&cmd);
 
 	/* Build the command. */
@@ -709,6 +714,19 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli)
 {
 	PGresult   *res;
 
+	/*
+	 * Skip PQputCopyEnd() if CommandComplete already ended the COPY, or if
+	 * the connection is already gone. copyEnded is still required after
+	 * COPY 0 because PQstatus() may remain CONNECTION_OK.
+	 */
+	if (conn->copyEnded ||
+		PQstatus(conn->streamConn) == CONNECTION_BAD)
+	{
+		conn->copyEnded = false;
+		*next_tli = 0;
+		return;
+	}
+
 	/*
 	 * Send copy-end message.  As in libpqsrv_exec, this could theoretically
 	 * block, but the risk seems small.
@@ -887,6 +905,7 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer,
 		if (PQresultStatus(res) == PGRES_COMMAND_OK)
 		{
 			PQclear(res);
+			conn->copyEnded = true;
 
 			/* Verify that there are no more results. */
 			res = libpqsrv_get_result(conn->streamConn,
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 11b9a12c52c..28b96d9829f 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -29,6 +29,10 @@
  * nudge the existing walreceiver process that's waiting, instead of launching
  * a new one.
  *
+ * A clean primary shutdown is different. There is no next timeline to follow
+ * on this connection, so walreceiver exits and a fresh process is started
+ * when streaming is requested again.
+ *
  * Normal termination is by SIGTERM, which instructs the walreceiver to
  * ereport(FATAL). Emergency termination is by SIGQUIT; like any postmaster
  * child process, the walreceiver will simply abort and exit on SIGQUIT. A
@@ -691,6 +695,13 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len)
 			 */
 			walrcv_endstreaming(wrconn, &primaryTLI);
 
+			/*
+			 * Clean remote shutdown. Exit instead of waiting mode, which would
+			 * reuse a dead connection and archive an incomplete segment.
+			 */
+			if (primaryTLI == 0)
+				proc_exit(0);
+
 			/*
 			 * If the server had switched to a new timeline that we didn't
 			 * know about when we began streaming, fetch its timeline history
-- 
2.56.0

Reply via email to