Hi Shihao, Thanks for testing it and for the review. v2 is attached, on today's master. Point by point:
> 1. I think it kills more than client writes. With a 2s timeout it also > killed > a session waiting for sync rep, and a COPY FROM STDIN client that > paused(in primary), Reproduced: v1 terminated both after 2.02 s, and after 30.02 s with the default; the sync rep one with "might not have been replicated to the standby". Neither can absorb the barrier while it waits: SyncRepWaitForLSN() doesn't run CHECK_FOR_INTERRUPTS(), and a COPY FROM STDIN read only services ProcDiePending. On master, DROP DATABASE was still waiting for them after 45 s and finished 15 ms after each one was released. v2 terminates only a client backend or WAL sender whose wait event is ClientWrite. Any other straggler is left alone and checked again at each wakeup. With v2, at 2 s and at 30 s, both sessions survive, DROP waits as on master and finishes 4-15 ms after they move on, and the five stuck cases from v1 still clear once the timeout expires. Both are now negative cases in 061. One gray zone is left: a session still sending a single large value to a slow client also sits in ClientWrite, with no CHECK_FOR_INTERRUPTS() until the value is out. Through a proxy limited to 2 MB/s, a 100 MB value made DROP wait 50.5 s on master; v2 with the default 30 s terminated that session at 30.05 s. I documented it in config.sgml, but I don't see a way to tell it apart from a client that stopped reading, with what another process can see. Ideas welcome. > 2. 061 failed 4 of 11 runs here. slow_reader stops reading before the > checksums launcher waits, so it gets killed. Right: pg_disable_data_checksums() returns before the launcher emits its barriers. 061 passed 11/11 here, but holding the launcher at datachecksumsworker-launcher-delay until the reader had blocked, v1's loop got the reader terminated 2/2. v2 keeps reading until data_checksums is off: 0/2. Pinned to one CPU, a second race showed up: woken by the barrier's signal, a backend blocked in ClientWrite on a Unix socket can still fit one more send (16 kB more queued while psql wasn't reading), reach a CHECK_FOR_INTERRUPTS() and absorb, so the first DROP did not wait. v2's 061 wakes each stuck session with pg_log_backend_memory_contexts() until it no longer logs. 061 then passed 11/11 normally and 11/11 pinned to one CPU. The mutants (no grace period, SIGINT, no kill, no ClientWrite check) all fail it, without hanging now. > 3. shouldTerminateBarrierStraggler() recheck pss_barrierGeneration > under pss_mutex? because it can kill a new backend in the same slot. Yes. Reproduced with gdb: with the waiter stopped on entry to TerminateBarrierStraggler(), the stuck session exits and a new session, also in ClientWrite, takes the same slot; v1 then terminated the new one, 5/5. v2 reads pss_pid and pss_barrierGeneration together under pss_mutex and skips the slot once the generation is reached, since a new process starts at the current one: 0/5 (3/3 again with the recheck removed). 061 can't reach that window; I can turn it into an injection point test if you think it's worth it. make check, the recovery suite (857 tests, including 061) and test_checksums with checksum_extended pass on master f2edf5dd00d. Regards, Manu
From dc0eb541574dbc92ca7367839668a0ea75669f77 Mon Sep 17 00:00:00 2001 From: Manu <[email protected]> Date: Thu, 22 Oct 2026 01:49:11 -0300 Subject: [PATCH v2] Terminate sessions that do not absorb a ProcSignalBarrier in time WaitForProcSignalBarrier() waits until every process has absorbed the barrier. A client backend or WAL sender that is blocked sending data to a client that has stopped reading never does: while blocked in a client write it services only ProcDiePending (see ProcessClientWriteInterrupt()), not the barrier and not a recovery conflict signal. Whatever waits on the barrier then waits for as long as that client stays stuck. That includes the end of recovery and the replay of the record that enables logical decoding when wal_level=replica, so a standby can neither promote nor keep replaying, as well as changing data checksums online, and DROP DATABASE and the other users of PROCSIGNAL_BARRIER_SMGRRELEASE. Add procsignal_barrier_timeout (default 30s, 0 disables). Once a barrier wait has lasted that long, a client backend or WAL sender that still has not absorbed the barrier and is blocked writing to its client is sent SIGTERM, as pg_terminate_backend() would do, and the wait goes on until it exits. A session waiting for anything else is left alone, even if it cannot absorb the barrier while it waits either: a backend waiting for synchronous replication has already committed locally, and one in COPY FROM STDIN is waiting for its client to send more data. Other kinds of process are never signaled. A LOG message names each process terminated this way. The slot's PID and barrier generation are read together under its spinlock before signaling, so that a process that took over the slot after the one that did not absorb the barrier exited is not signaled. This applies to every waiter, not only to the startup process, since the hang is the same on a primary. Reported-by: shihao zhong <[email protected]> Suggested-by: Masahiko Sawada <[email protected]> Reviewed-by: shihao zhong <[email protected]> Discussion: https://postgr.es/m/caa4ek1lc09tq3vtr04ppxrbyqnmb66_pgyues2c_5zowfyr...@mail.gmail.com --- doc/src/sgml/config.sgml | 38 ++ src/backend/storage/ipc/procsignal.c | 118 ++++- src/backend/utils/misc/guc_parameters.dat | 10 + src/backend/utils/misc/guc_tables.c | 1 + src/backend/utils/misc/postgresql.conf.sample | 3 + src/include/storage/procsignal.h | 3 + src/test/recovery/meson.build | 1 + .../t/061_procsignal_barrier_timeout.pl | 433 ++++++++++++++++++ 8 files changed, 605 insertions(+), 2 deletions(-) create mode 100644 src/test/recovery/t/061_procsignal_barrier_timeout.pl diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index f7db299..df7df90 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -11836,6 +11836,44 @@ dynamic_library_path = '/usr/local/lib/postgresql:$libdir' </listitem> </varlistentry> + <varlistentry id="guc-procsignal-barrier-timeout" xreflabel="procsignal_barrier_timeout"> + <term><varname>procsignal_barrier_timeout</varname> (<type>integer</type>) + <indexterm> + <primary><varname>procsignal_barrier_timeout</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Some operations change state that every running process must take + into account before the operation can continue, and wait until all + processes have done so. Among them are the end of recovery and the + replay of WAL on a standby when logical decoding is being enabled, + enabling or disabling data checksums online (see + <xref linkend="functions-admin-checksum"/>) and + <command>DROP DATABASE</command>. A session can only do this + between other work; one that is blocked sending data to a client that + has stopped reading cannot, and the operation would wait for it + indefinitely. This parameter sets how long the operation waits + before it terminates a client backend or WAL sender process that is + blocked that way, as <function>pg_terminate_backend</function> would. + This includes a session that is still sending a single large value + to a slow client. Sessions waiting for anything else, for example + for synchronous replication or for more data from the client during + <command>COPY FROM STDIN</command>, are not terminated, nor are other + processes; the operation keeps waiting for them. A message is + written to the server log for each process terminated this way. + </para> + <para> + If this value is specified without units, it is taken as + milliseconds. A value of zero disables the timeout, so the + operation waits as long as it takes. The default is 30 seconds + (<literal>30s</literal>). This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command + line. + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-max-locks-per-transaction" xreflabel="max_locks_per_transaction"> <term><varname>max_locks_per_transaction</varname> (<type>integer</type>) <indexterm> diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c index eb0f4d6..5d93ce7 100644 --- a/src/backend/storage/ipc/procsignal.c +++ b/src/backend/storage/ipc/procsignal.c @@ -38,6 +38,7 @@ #include "storage/subsystems.h" #include "tcop/tcopprot.h" #include "utils/memutils.h" +#include "utils/timestamp.h" #include "utils/wait_event.h" /* @@ -119,9 +120,13 @@ const ShmemCallbacks ProcSignalShmemCallbacks = { NON_EXEC_STATIC ProcSignalHeader *ProcSignal = NULL; +/* GUC variable */ +int ProcSignalBarrierTimeout = 30000; + static ProcSignalSlot *MyProcSignalSlot = NULL; static bool CheckProcSignal(ProcSignalReason reason); +static bool TerminateBarrierStraggler(int procno, uint64 generation); static void CleanupProcSignalState(int status, Datum arg); static void ResetProcSignalBarrierBits(uint32 flags); @@ -437,6 +442,8 @@ EmitProcSignalBarrier(ProcSignalBarrierType type) void WaitForProcSignalBarrier(uint64 generation) { + TimestampTz wait_start = 0; + Assert(generation <= pg_atomic_read_u64(&ProcSignal->psh_barrierGeneration)); elog(DEBUG1, @@ -448,6 +455,7 @@ WaitForProcSignalBarrier(uint64 generation) { ProcSignalSlot *slot = &ProcSignal->psh_slot[i]; uint64 oldval; + bool terminated = false; /* * It's important that we check only pss_barrierGeneration here and @@ -458,9 +466,46 @@ WaitForProcSignalBarrier(uint64 generation) oldval = pg_atomic_read_u64(&slot->pss_barrierGeneration); while (oldval < generation) { + long sleep_ms = 5000; + bool report_on_timeout = true; + + /* + * Once we have waited procsignal_barrier_timeout in total, + * terminate the process that has not absorbed the barrier if it + * is blocked writing to its client, then keep waiting for it to + * exit. A process that exits advertises PG_UINT64_MAX and wakes + * us up (see CleanupProcSignalState()). A process that is + * waiting for anything else is left alone, but checked again each + * time we wake up. + * + * XXX: This applies to every caller, not only to the startup + * process. A process blocked on its client (see + * TerminateBarrierStraggler()) can hold up the end of recovery + * and the replay of WAL just as well as DROP DATABASE or + * pg_enable_data_checksums() on a primary. + */ + if (ProcSignalBarrierTimeout > 0 && !terminated) + { + TimestampTz now = GetCurrentTimestamp(); + long elapsed_ms; + + if (wait_start == 0) + wait_start = now; + elapsed_ms = TimestampDifferenceMilliseconds(wait_start, now); + + if (elapsed_ms >= ProcSignalBarrierTimeout) + terminated = TerminateBarrierStraggler(i, generation); + else if (ProcSignalBarrierTimeout - elapsed_ms < sleep_ms) + { + sleep_ms = ProcSignalBarrierTimeout - elapsed_ms; + report_on_timeout = false; + } + } + if (ConditionVariableTimedSleep(&slot->pss_barrierCV, - 5000, - WAIT_EVENT_PROC_SIGNAL_BARRIER)) + sleep_ms, + WAIT_EVENT_PROC_SIGNAL_BARRIER) && + report_on_timeout) ereport(LOG, (errmsg("still waiting for backend with PID %d to accept ProcSignalBarrier", (int) pg_atomic_read_u32(&slot->pss_pid)))); @@ -484,6 +529,75 @@ WaitForProcSignalBarrier(uint64 generation) pg_memory_barrier(); } +/* + * TerminateBarrierStraggler - terminate the process using ProcSignal slot + * 'procno' if it has still not absorbed barrier 'generation' and is blocked + * writing to its client. Returns true if it was signaled. + * + * Only a client backend or walsender waiting in ClientWrite is terminated, + * with SIGTERM, as pg_terminate_backend() would do. While blocked in a client + * write, a backend services only ProcDiePending (see + * ProcessClientWriteInterrupt()), so it cannot absorb the barrier until its + * client reads again, and SIGTERM is what releases it; a recovery conflict + * signal would not. + * + * Processes waiting for anything else are not signaled, even when they cannot + * absorb the barrier either while they wait. A backend waiting for + * synchronous replication has already committed its transaction locally (see + * SyncRepWaitForLSN()), and one in COPY FROM STDIN is waiting for its client + * to send more data, which the client may be about to do. + */ +static bool +TerminateBarrierStraggler(int procno, uint64 generation) +{ + ProcSignalSlot *slot = &ProcSignal->psh_slot[procno]; + PGPROC *proc = GetPGProcByNumber(procno); + pid_t pid; + bool absorbed; + + /* + * The process that our caller saw has not absorbed the barrier may have + * exited since, and another may have taken over the slot. The PID and + * the barrier generation change together under pss_mutex, and a new + * process starts out with the current barrier generation (see + * ProcSignalInit()), so reading both under the lock tells whether the PID + * is still that of a process that has not absorbed the barrier. + */ + SpinLockAcquire(&slot->pss_mutex); + pid = (pid_t) pg_atomic_read_u32(&slot->pss_pid); + absorbed = pg_atomic_read_u64(&slot->pss_barrierGeneration) >= generation; + SpinLockRelease(&slot->pss_mutex); + + if (absorbed || pid == 0 || pid == MyProcPid) + return false; + + /* The slot is indexed by ProcNumber, so 'proc' is its owner */ + if (proc->pid != pid) + return false; + if (proc->backendType != B_BACKEND && proc->backendType != B_WAL_SENDER) + return false; + if (*((volatile uint32 *) &proc->wait_event_info) != WAIT_EVENT_CLIENT_WRITE) + return false; + + ereport(LOG, + (errmsg("terminating process %d because it has not accepted ProcSignalBarrier within %d ms", + (int) pid, ProcSignalBarrierTimeout), + errdetail("The process is blocked sending data to its client."))); + + /* + * As in pg_signal_backend(), the process could exit and its PID be reused + * before we send the signal; that seems too unlikely to worry about. + * Signal the whole process group if we have setsid(). + */ +#ifdef HAVE_SETSID + (void) kill(-pid, SIGTERM); +#else + (void) kill(pid, SIGTERM); +#endif + + return true; +} + /* * Handle receipt of an interrupt indicating a global barrier event. * diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat index 380679a..c579b07 100644 --- a/src/backend/utils/misc/guc_parameters.dat +++ b/src/backend/utils/misc/guc_parameters.dat @@ -2423,6 +2423,16 @@ check_hook => 'check_primary_slot_name', }, +{ name => 'procsignal_barrier_timeout', type => 'int', context => 'PGC_SIGHUP', group => 'LOCK_MANAGEMENT', + short_desc => 'Sets the time to wait for a session to accept a global state change before terminating it.', + long_desc => '0 disables the timeout.', + flags => 'GUC_UNIT_MS', + variable => 'ProcSignalBarrierTimeout', + boot_val => '30000', + min => '0', + max => 'INT_MAX', +}, + { name => 'quote_all_identifiers', type => 'bool', context => 'PGC_USERSET', group => 'COMPAT_OPTIONS_PREVIOUS', short_desc => 'When generating SQL fragments, quote all identifiers.', variable => 'quote_all_identifiers', diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 342aaee..8c03a4f 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -88,6 +88,7 @@ #include "storage/predicate.h" #include "storage/proc.h" #include "storage/procnumber.h" +#include "storage/procsignal.h" #include "storage/standby.h" #include "tcop/backend_startup.h" #include "tcop/tcopprot.h" diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 01f98de..9786f25 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -876,6 +876,9 @@ #------------------------------------------------------------------------------ #deadlock_timeout = 1s +#procsignal_barrier_timeout = 30s # terminate a session that has not + # accepted a global state change + # after this long; 0 disables #max_locks_per_transaction = 128 # min 10 # (change requires restart) #max_pred_locks_per_transaction = 64 # min 10 diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h index aaa158b..9f048c2 100644 --- a/src/include/storage/procsignal.h +++ b/src/include/storage/procsignal.h @@ -66,6 +66,9 @@ typedef enum */ #define MAX_CANCEL_KEY_LENGTH 32 +/* GUC variable */ +extern PGDLLIMPORT int ProcSignalBarrierTimeout; + /* * prototypes for functions in procsignal.c */ diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index b6ac9dd..4942a0c 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -69,6 +69,7 @@ tests += { 't/058_shutdown_crash_restart.pl', 't/059_remote_apply_status_interval.pl', 't/060_multixact_crashed_updater.pl', + 't/061_procsignal_barrier_timeout.pl', ], }, } diff --git a/src/test/recovery/t/061_procsignal_barrier_timeout.pl b/src/test/recovery/t/061_procsignal_barrier_timeout.pl new file mode 100644 index 0000000..85cc255 --- /dev/null +++ b/src/test/recovery/t/061_procsignal_barrier_timeout.pl @@ -0,0 +1,433 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test procsignal_barrier_timeout. Operations that wait for every process to +# absorb a ProcSignalBarrier must not wait forever behind a session that is +# blocked writing to a client that has stopped reading: after the timeout such +# a session is terminated. Sessions that are merely idle, sleeping or reading +# slowly must not be, nor sessions that wait for something other than their +# client reading, such as synchronous replication or more COPY FROM STDIN data. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use Time::HiRes qw(time usleep); + +my $copy_query = + "COPY (SELECT generate_series(1, 1000000000), repeat('x', 200)) TO STDOUT"; +my $terminated_re = + 'because it has not accepted ProcSignalBarrier within 2000 ms'; +my $timeout = $PostgreSQL::Test::Utils::timeout_default; + +# Start a psql running $sql, with its output captured but not read. +sub start_psql +{ + my ($node, $dbname, $appname, $sql, $out) = @_; + my $err = ''; + + return IPC::Run::start( + [ + 'psql', '--no-psqlrc', '--quiet', + '--dbname' => $node->connstr($dbname) + . " application_name=$appname", + '--command' => $sql + ], + '>' => $out, + '2>' => \$err); +} + +# Return the PID of the session with the given application_name, once its +# wait_event is $wait_event (or once it exists, if $wait_event is undef). +sub session_pid +{ + my ($node, $appname, $wait_event) = @_; + my $cond = defined $wait_event ? "wait_event = '$wait_event'" : 'true'; + + $node->poll_query_until('postgres', + "SELECT $cond FROM pg_stat_activity WHERE application_name = '$appname'" + ) or die "timed out waiting for session \"$appname\""; + return $node->safe_psql('postgres', + "SELECT pid FROM pg_stat_activity WHERE application_name = '$appname'"); +} + +# Start a COPY TO STDOUT whose output is never read. psql blocks writing to +# its stdout and stops reading from the server, so the backend ends up blocked +# in a client write, where it cannot absorb a ProcSignalBarrier. +sub start_stuck_session +{ + my ($node, $dbname, $appname) = @_; + my $out = ''; + my $h = start_psql($node, $dbname, $appname, $copy_query, \$out); + my $pid = session_pid($node, $appname, 'ClientWrite'); + + wait_until_stuck($node, $pid); + return ($h, $pid); +} + +# Wait until the session $pid, which shows ClientWrite, is blocked for good. It +# can show ClientWrite while psql is still reading, and even once psql has +# stopped, a backend that is woken up can find room for one more send on a +# Unix-domain socket and run on to its next write, absorbing any pending +# barrier on the way. So wake it up until it stays put: +# pg_log_backend_memory_contexts() makes it log once it reaches its next +# CHECK_FOR_INTERRUPTS(), which it only does if it was not blocked for good. +sub wait_until_stuck +{ + my ($node, $pid) = @_; + + for (my $i = 0;; $i++) + { + my $offset = -s $node->logfile; + + die "session $pid does not stay blocked" if $i >= 20; + $node->safe_psql('postgres', + "SELECT pg_log_backend_memory_contexts($pid)"); + usleep(500_000); + last + unless $node->log_contains("logging memory contexts of PID $pid", + $offset); + $node->poll_query_until('postgres', + "SELECT wait_event = 'ClientWrite' FROM pg_stat_activity WHERE pid = $pid" + ) or die "timed out waiting for session $pid to block again"; + } +} + +sub session_alive +{ + my ($node, $pid) = @_; + + return $node->safe_psql('postgres', + "SELECT count(*) FROM pg_stat_activity WHERE pid = $pid") eq '1'; +} + +# Check that the session $pid was terminated by the barrier timeout. +sub check_terminated +{ + my ($node, $pid, $log_offset, $what) = @_; + + ok( $node->log_contains( + "terminating process $pid $terminated_re", $log_offset), + "$what: blocked session was terminated after the timeout"); + ok( $node->log_contains( + qr/\[$pid\].*FATAL: terminating connection due to administrator command/, + $log_offset), + "$what: blocked session exited"); + ok($node->poll_query_until('postgres', + "SELECT count(*) = 0 FROM pg_stat_activity WHERE pid = $pid"), + "$what: blocked session is gone"); +} + +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 1, no_data_checksums => 1); +$primary->append_conf('postgresql.conf', 'procsignal_barrier_timeout = 2s'); +$primary->start; +$primary->safe_psql('postgres', 'CREATE DATABASE other'); +$primary->safe_psql('postgres', 'CREATE DATABASE target1'); +$primary->safe_psql('postgres', 'CREATE DATABASE target2'); +$primary->safe_psql('postgres', 'CREATE DATABASE target3'); +$primary->safe_psql('postgres', 'CREATE DATABASE target4'); +$primary->safe_psql('postgres', 'CREATE DATABASE target5'); +$primary->safe_psql('postgres', 'CREATE DATABASE target6'); +$primary->safe_psql('postgres', 'CREATE TABLE copy_target (a int)'); + +my ($h, $pid, $log_offset, $start, $elapsed); + +# +# DROP DATABASE waits for every process to close its files of the dropped +# database, including a session connected to another database. +# +($h, $pid) = start_stuck_session($primary, 'other', 'stuck_dropdb'); +$log_offset = -s $primary->logfile; +$start = time; +$primary->safe_psql('postgres', 'DROP DATABASE target1', + timeout => $timeout); +$elapsed = time - $start; +is( $primary->safe_psql( + 'postgres', + "SELECT count(*) FROM pg_database WHERE datname = 'target1'"), + '0', + 'DROP DATABASE: completed with a blocked session on another database'); +cmp_ok($elapsed, '>=', 2, 'DROP DATABASE: waited for the timeout'); +check_terminated($primary, $pid, $log_offset, 'DROP DATABASE'); +$h->kill_kill; + +# +# Enabling data checksums online. +# +($h, $pid) = start_stuck_session($primary, 'postgres', 'stuck_checksums'); +$log_offset = -s $primary->logfile; +$primary->safe_psql('postgres', 'SELECT pg_enable_data_checksums()'); +ok( $primary->poll_query_until( + 'postgres', + "SELECT setting FROM pg_settings WHERE name = 'data_checksums'", 'on'), + 'pg_enable_data_checksums(): reached "on" with a blocked session'); +check_terminated($primary, $pid, $log_offset, 'pg_enable_data_checksums()'); +$h->kill_kill; + +# +# A WAL sender blocked sending a base backup to a client that has stopped +# reading. +# +my ($bb_out, $bb_err) = ('', ''); +$h = IPC::Run::start( + [ + 'pg_basebackup', + '--pgdata' => '-', + '--format' => 'tar', + '--wal-method' => 'none', + '--checkpoint' => 'fast', + '--dbname' => $primary->connstr('postgres') + . ' application_name=stuck_walsender' + ], + '>' => \$bb_out, + '2>' => \$bb_err); +$pid = session_pid($primary, 'stuck_walsender', 'ClientWrite'); +wait_until_stuck($primary, $pid); +is( $primary->safe_psql( + 'postgres', "SELECT backend_type FROM pg_stat_activity WHERE pid = $pid"), + 'walsender', + 'base backup is served by a WAL sender'); +$log_offset = -s $primary->logfile; +$primary->safe_psql('postgres', 'DROP DATABASE target4', + timeout => $timeout); +check_terminated($primary, $pid, $log_offset, 'WAL sender'); +$h->kill_kill; + +# +# Negative control: idle sessions, a session in pg_sleep() and a client that +# reads slowly but keeps reading are not terminated, and do not delay the +# operations. +# +my $idle = $primary->background_psql('postgres'); +my $idle_pid = $idle->query_safe('SELECT pg_backend_pid()'); +my $sleep_out = ''; +my $sleeper = start_psql($primary, 'postgres', 'sleeper', + 'SELECT pg_sleep(600)', \$sleep_out); +my $sleeper_pid = session_pid($primary, 'sleeper', 'PgSleep'); +my $slow_out = ''; +my $slow = start_psql($primary, 'other', 'slow_reader', $copy_query, + \$slow_out); +my $slow_pid = session_pid($primary, 'slow_reader', 'ClientWrite'); + +# Run $sql, reading a little from the slow client now and then until $sql has +# completed and the SQL condition $done holds. The slow client must keep +# reading for as long as a barrier may be waiting for it: +# pg_disable_data_checksums() returns before the data checksums launcher +# emits and waits for its barriers. Returns the time $sql took. +sub run_while_reading_slowly +{ + my ($sql, $done) = @_; + my $out = ''; + my $t0 = time; + my $op = start_psql($primary, 'postgres', 'op', $sql, \$out); + my $op_elapsed; + + while (1) + { + $slow->pump_nb; + $slow_out = ''; + if (!defined $op_elapsed) + { + if ($op->pumpable) + { + $op->pump_nb; + } + else + { + $op->finish; + $op_elapsed = time - $t0; + } + } + last + if defined $op_elapsed + && $primary->safe_psql('postgres', "SELECT $done") eq 't'; + die "timed out running \"$sql\"" if time - $t0 > $timeout; + usleep(50_000); + } + return $op_elapsed; +} + +$log_offset = -s $primary->logfile; +$elapsed = run_while_reading_slowly('DROP DATABASE target2', + "NOT EXISTS (SELECT FROM pg_database WHERE datname = 'target2')"); +is( $primary->safe_psql( + 'postgres', + "SELECT count(*) FROM pg_database WHERE datname = 'target2'"), + '0', + 'negative control: DROP DATABASE completed'); +cmp_ok($elapsed, '<', 2, + 'negative control: DROP DATABASE did not wait for the timeout'); +run_while_reading_slowly('SELECT pg_disable_data_checksums()', + "(SELECT setting FROM pg_settings WHERE name = 'data_checksums') = 'off'"); +pass('negative control: data checksums disabled'); +ok(!$primary->log_contains('terminating process', $log_offset), + 'negative control: no session was terminated'); +ok( session_alive($primary, $idle_pid) + && session_alive($primary, $sleeper_pid) + && session_alive($primary, $slow_pid), + 'negative control: idle, sleeping and slowly reading sessions survived'); +$idle->quit; +$sleeper->kill_kill; +$slow->kill_kill; + +# +# Sessions that cannot absorb the barrier while they wait, but are not blocked +# writing to their client, are not terminated either. A session waiting for +# synchronous replication has already committed locally, and one in COPY FROM +# STDIN waits for its client to send more data. The operation keeps waiting +# for them and completes once they move on. +# +sub check_not_terminated +{ + my ($pid, $dbname, $what, $release) = @_; + my $offset = -s $primary->logfile; + my $dropper = $primary->background_psql('postgres'); + + $dropper->query_safe('SET synchronous_commit = local'); + $dropper->query_until(qr/dropping/, + "\\echo dropping\nDROP DATABASE $dbname;\n"); + + # Reported only once the timeout has expired and the session was left + # alone. + $primary->wait_for_log( + qr/still waiting for backend with PID $pid to accept ProcSignalBarrier/, + $offset); + ok(session_alive($primary, $pid), "$what: session is not terminated"); + ok(!$primary->log_contains('terminating process', $offset), + "$what: no session was terminated"); + is( $primary->safe_psql( + 'postgres', + "SELECT count(*) FROM pg_database WHERE datname = '$dbname'"), + '1', + "$what: DROP DATABASE is still waiting"); + + $release->(); + $dropper->quit; + is( $primary->safe_psql( + 'postgres', + "SELECT count(*) FROM pg_database WHERE datname = '$dbname'"), + '0', + "$what: DROP DATABASE completed once the session moved on"); +} + +# A session waiting for a synchronous standby that is not connected. +$primary->safe_psql('postgres', + "ALTER SYSTEM SET synchronous_standby_names = 'nosuch'"); +$primary->reload; +my $syncrep_out = ''; +my $syncrep = start_psql($primary, 'postgres', 'syncrep_waiter', + 'CREATE TABLE syncrep_target ()', \$syncrep_out); +my $syncrep_pid = session_pid($primary, 'syncrep_waiter', 'SyncRep'); +check_not_terminated( + $syncrep_pid, + 'target5', + 'synchronous replication', + sub { + $primary->safe_psql('postgres', + 'ALTER SYSTEM RESET synchronous_standby_names'); + $primary->reload; + }); +$syncrep->finish; + +# A COPY FROM STDIN whose client sent one row and then paused. +my ($copy_in, $copy_out, $copy_err) = ("COPY copy_target FROM STDIN;\n1\n", '', ''); +my $copy = IPC::Run::start( + [ + 'psql', '--no-psqlrc', '--quiet', + '--dbname' => $primary->connstr('postgres') + . ' application_name=copy_paused' + ], + '<' => \$copy_in, + '>' => \$copy_out, + '2>' => \$copy_err); +$copy->pump_nb; +$primary->poll_query_until('postgres', + "SELECT wait_event = 'ClientRead' AND state = 'active' FROM pg_stat_activity WHERE application_name = 'copy_paused'" +) or die 'timed out waiting for COPY FROM STDIN to start'; +my $copy_pid = session_pid($primary, 'copy_paused'); +check_not_terminated( + $copy_pid, + 'target6', + 'COPY FROM STDIN', + sub { + $copy_in .= "2\n\\.\n"; + $copy->finish; + }); +is($primary->safe_psql('postgres', 'SELECT count(*) FROM copy_target'), + '2', 'COPY FROM STDIN: completed'); + +# +# With procsignal_barrier_timeout = 0, the operation keeps waiting for the +# blocked session. +# +$primary->append_conf('postgresql.conf', 'procsignal_barrier_timeout = 0'); +$primary->reload; +$primary->poll_query_until('postgres', 'SHOW procsignal_barrier_timeout', + '0') + or die 'timed out waiting for the reload'; +($h, $pid) = start_stuck_session($primary, 'other', 'stuck_disabled'); +$log_offset = -s $primary->logfile; +my $dropper = $primary->background_psql('postgres'); +$dropper->query_until(qr/dropping/, "\\echo dropping\nDROP DATABASE target3;\n"); +$primary->wait_for_log( + qr/still waiting for backend with PID $pid to accept ProcSignalBarrier/, + $log_offset); +ok(session_alive($primary, $pid), + 'timeout disabled: blocked session is not terminated'); +is( $primary->safe_psql( + 'postgres', + "SELECT count(*) FROM pg_database WHERE datname = 'target3'"), + '1', + 'timeout disabled: DROP DATABASE is still waiting'); +$primary->safe_psql('postgres', "SELECT pg_terminate_backend($pid)"); +$dropper->quit; +is( $primary->safe_psql( + 'postgres', + "SELECT count(*) FROM pg_database WHERE datname = 'target3'"), + '0', + 'timeout disabled: DROP DATABASE completed once the session was terminated' +); +$h->kill_kill; +$primary->append_conf('postgresql.conf', 'procsignal_barrier_timeout = 2s'); +$primary->reload; + +# +# Standby: replay of the record that enables logical decoding with +# wal_level = replica, and the end of recovery when logical decoding is +# enabled, wait for every process to update its cached state. +# +$primary->backup('backup'); +my $standby = PostgreSQL::Test::Cluster->new('standby'); +$standby->init_from_backup($primary, 'backup', has_streaming => 1); +$standby->append_conf('postgresql.conf', 'procsignal_barrier_timeout = 2s'); +$standby->start; +$primary->wait_for_replay_catchup($standby); + +($h, $pid) = start_stuck_session($standby, 'postgres', 'stuck_replay'); +$log_offset = -s $standby->logfile; +$primary->safe_psql('postgres', + "SELECT pg_create_logical_replication_slot('primary_slot', 'pgoutput')"); +$primary->wait_for_replay_catchup($standby); +is($standby->safe_psql('postgres', 'SHOW effective_wal_level'), + 'logical', 'replay: logical decoding enabled with a blocked session'); +check_terminated($standby, $pid, $log_offset, 'replay'); +$h->kill_kill; + +$standby->create_logical_slot_on_standby($primary, 'standby_slot', + 'postgres'); +$idle = $standby->background_psql('postgres'); +$idle_pid = $idle->query_safe('SELECT pg_backend_pid()'); +($h, $pid) = start_stuck_session($standby, 'postgres', 'stuck_promote'); +$log_offset = -s $standby->logfile; +$standby->promote; +is($standby->safe_psql('postgres', 'SELECT pg_is_in_recovery()'), + 'f', 'promotion: completed with a blocked session'); +check_terminated($standby, $pid, $log_offset, 'promotion'); +ok(session_alive($standby, $idle_pid), + 'promotion: idle session survived'); +$idle->quit; +$h->kill_kill; + +done_testing(); -- 2.56.0
