Hi Amit,
> If we anyway would like to implement killing of connections that
> didn't absorb barriers then do we even need v6. I am slightly hesitant
> about v6 because it adds more assumptions in the code like which
> processes can have stale cache.
With the docs note for 19 settled, I'd like to bring this back to
HEAD. v2 [1] implements option (2): once a barrier wait exceeds
procsignal_barrier_timeout (default 30s), a client backend or WAL
sender that hasn't absorbed it and is blocked in ClientWrite gets
SIGTERM. Nothing else is signaled.
To answer your question with numbers, I ran the same scenarios on
master (f2edf5dd00d) with three builds: unpatched, v6, and v2 (timeout
2 s for these runs), 3 runs per cell. The stuck backend is a session
whose client is alive but has stopped reading. The full matrix, with
how each row was measured, and the scripts to reproduce every row are
attached.
Promotion (failover slot synced from the start, local slot, slot
created or toggled during recovery, crash restart before promotion):
master hung every time, v6 promoted in 9-26 ms, v2 after its timeout.
Replay stuck behind the backend, then the primary creates its first
logical slot: master and v6 stalled replay for the whole 15 s, v2
caught up after its timeout.
On a primary (enabling checksums, DROP DATABASE of another database,
DROP DATABASE with a walsender in a base backup): master and v6 hung,
v2 finished after its timeout.
Controls (idle, pg_sleep, a slow reader, a sync rep waiter, a paused
COPY FROM STDIN) behaved the same in all three builds, and none was
terminated in 45 runs.
So v2 alone clears every case here, including the ones v6 doesn't
reach, and doesn't depend on v6. What v6 adds is speed: without it, a
promotion behind such a backend waits for the whole timeout. On the
stale caches, I read XLogLogicalInfo of every standby process with gdb
before and after pg_promote(). With the status on since startup,
master had the same five processes stale before promotion
(checkpointer, background writer, both io workers, startup) in every
run, and v6 had none, also after a crash restart. After promotion
nothing was stale in any build. That is evidence for these setups, not
a proof.
Two limits of v2. It doesn't change recovery conflicts: as Sawada-san
noted, a backend in ClientWrite doesn't service them, and with v1
replay still stopped behind one, 3/3; v2 only narrows whom it
terminates. And a session sending one large value to a slow client
for longer than the timeout is terminated too (documented in
config.sgml).
Reviews of v2 would be very welcome.
Regards,
Manu
[1]
https://www.postgresql.org/message-id/179264658926.1656483.9003619221600313130%40gmail.com
ProcSignalBarrier waits behind a session stuck writing to its client:
which change clears which case
=====================================================================
Which change clears which case, measured. Same base for every column:
master master at f2edf5dd00d, unpatched
note the docs note for 19 (no code change); its two remedies, measured on
master: (a) pg_terminate_backend() on the logged PID, (b)
tcp_user_timeout set ahead of time (10s here; TCP clients only)
v6 master + v6_fix_masahiko.patch (applies cleanly)
v2 master + v2-0001 (option 2), procsignal_barrier_timeout = 2s
All builds: --enable-cassert --enable-debug --enable-tap-tests
--enable-injection-points. The stuck session is a client that is alive but has
stopped reading, so its backend sits in ClientWrite. 3 runs per cell unless
stated. Times are in ms from the start of the operation unless stated; the
"hung" cells are bounded by the observation window (12 s for promotion, 15 s
otherwise), not by the server.
row master note (a) / (b) v6 v2
--------------------------------------------------------------------------------
Standby, wal_level = replica on both nodes
1 failover slot at start hung 4-6 / 10.5 s 20-24 2024-2031
2 local slot at start hung 6-7 9-26 2011-2019
3 slot made in recovery(6) hung 4-6 12-26 2010-2038
4 toggled in recovery hung 6 9-25 2013-2021
5 stuck, then first slot replay 15s+ 4-6 / 10.5 s replay 15s+ 2040-2063
9 crash, then promotion hung not measured 23-24 2021-2031
Primary
6 enable data checksums hung 70-121 (*) hung 2066-2074
7 DROP DATABASE, other db hung 5-6 (*) hung 2008-2012
8 DROP DATABASE, walsender hung 4-6 (*) hung 2009-2013
Negative controls (DROP DATABASE as the barrier; must survive)
idle session 8-9 - 7-9 7-10
pg_sleep(600) 9-10 - 9 8-10
slow reader (keeps reading) 8-9 - 8-10 8-10
sync rep waiter, no standby waits 15s+ - waits 15s+ waits 15s+
COPY FROM STDIN, paused waits 15s+ - waits 15s+ waits 15s+
--------------------------------------------------------------------------------
(a) ms from pg_terminate_backend() of the stuck backend to completion.
(b) total time from the moment the backend blocked (see "tcp_user_timeout").
(*) the note is about promotion and does not cover rows 6-8; (a) still works.
"replay 15s+": replay of the record that enables logical decoding did not
reach it within 15 s. Row 3 is 3 runs with a synced failover slot plus 3 with
a slot made on the standby.
Rows
----
1 Failover slot created on the primary before the base backup, synced to the
standby by the slot sync worker. Logical decoding is on from standby start.
2 Same status, but the logical slot is created on the standby itself.
3 The primary creates its first logical slot after the standby started (a
replayed enable record); synced failover slot, or a slot on the standby.
4 On at start, off (last primary slot dropped, checkpoint), on again, all
while the standby is in recovery.
5 The backend gets stuck first; then the primary creates its first logical
slot. Measured: does replay of the enable record pass the stuck backend?
9 Row 1 setup plus a slot on the standby; kill -9 of a standby backend
(crash restart), then the stuck session and promotion.
6-8 On a primary: pg_enable_data_checksums(); DROP DATABASE of another
database while the session is connected elsewhere; DROP DATABASE while a
walsender is stuck sending a base backup.
In every promotion row, after promotion 5 rows inserted on the new primary
were decoded through the logical slot, in every run of every column.
How it was measured
-------------------
Rows 1-5, 9: repro-standby.sh (with stall.py). Primary and standby on
127.0.0.1, both wal_level = replica. The stuck session runs COPY ... TO
STDOUT from stall.py, which never reads. Before acting, the script waits
until the backend stays blocked: its socket's bytes_acked stops moving, and a
wakeup with pg_log_backend_memory_contexts() no longer makes it log (a woken
backend can fit one more send and absorb a pending barrier otherwise). Then
pg_promote(false); "hung" = not promoted in 12 s, after which the script
terminates the stuck backend and records the release time.
Rows 6-8: t/090_barrier_primary_rows.pl; controls: t/091_barrier_controls.pl.
Both are TAP tests run inside the source tree of each build. The controls
set synchronous_commit = local and only the sync rep control session turns it
on, so CREATE/DROP DATABASE never wait for the missing standby. For the two
"waits" controls, the barrier wait was logged ("still waiting for backend with
PID ...") in every run, and DROP DATABASE finished 9-20 ms after the control
was released (standby names reset / end of COPY data sent). 45 control runs
(5 controls x 3 runs x 3 builds): every session survived, none was terminated.
In every v2 cell that cleared, the stuck backend was terminated by the
timeout ("terminating process ... within 2000 ms"; checked in the log for
rows 6-8, 9/9). Every "hung" cell was released by pg_terminate_backend() of
the stuck backend in 4-175 ms.
tcp_user_timeout (column note, (b))
-----------------------------------
tcp_user_timeout = 10s in postgresql.conf, measured with the attached script
(repro-standby.sh, which waits until the backend is blocked for good before
acting; WINDOW=40 RWAIT=40). Promotion (row 1): promoted 10.49 s after the
backend blocked (10489-10493 ms), 3/3. Replay (row 5): caught up 10.52-10.56 s
after the backend blocked, 3/3. Every run logged "could not send data to
client: Connection timed out". Not measured: clients on a Unix-domain socket
(tcp_user_timeout does not apply to them), rows 6-8 and row 9 with (b).
Stale caches (XLogLogicalInfo)
------------------------------
With CENSUS=1, repro-standby.sh reads the process-local XLogLogicalInfo of
every standby child with gdb (needs gdb, debug symbols and permission to
attach), just before pg_promote() and again after promotion.
Rows 1-2 (status on since standby start): master and v2 had 5 stale
processes (checkpointer, background writer, both io workers, startup) out
of 7-8 in 12/12 runs; v6 had none, 6/6. Row 9, after the crash restart:
master and v2 5/7, v6 0/7. Rows 3-5: no stale process in any column.
After promotion: no stale process in any run of any column.
Discarded runs
--------------
tcp_user_timeout = 3s: the backend had already timed out before the operation
started, so those runs (all "promoted in 8-25 ms") do not measure the bound.
One replay run that acted as soon as the backend first showed ClientWrite:
the backend was not blocked yet and absorbed the barrier (caught up in 60 ms).
Three earlier promotion runs without the settle step (10.50-10.52 s after the
backend blocked) agree with the numbers above but used a different method;
they are superseded by the runs made with the attached script. One row 1 run on
master: setup timed out (slot never synced within 170 s); replaced. A first
controls run hung in the harness (CREATE DATABASE waited for the missing sync
standby); fixed as described above.
Related, measured on other setups
---------------------------------
- REL_19_STABLE at 2dc899f780f + v6: 25/25 promotions cleared over the same
setups (unpatched 22/22 hung); row 5 stalled 15 s with and without v6.
- v2's own test (recovery/061): 11/11 runs, 11/11 pinned to one CPU.
- v2 limit: a session sending one 100 MB value to a client reading 2 MB/s is
terminated at the timeout (30.05 s with the default 30s); master waits until
the value is out (50.5 s).
- Recovery conflicts against such a backend are not cleared by any column.
#!/usr/bin/env bash
# repro-standby.sh: rows 1-5 and 9 of coverage-matrix.txt (standby promotion /
replay).
# Primary + standby (wal_level=replica on both), one backend on the standby
stuck in ClientWrite
# (stall.py), then pg_promote(false). Prints one line of key=value results.
# usage: PGBIN=/path/to/install/bin ./repro-standby.sh CASE [PORT]
# CASE: failover-at-start | local-at-start | failover-in-recovery |
local-in-recovery |
# toggled | replay-stuck | crash-then-promote (needs
contrib/test_decoding)
# Optional env: EXTRA_CONF (extra postgresql.conf lines, e.g.
"procsignal_barrier_timeout = 2s"),
# WINDOW (s to wait for promotion, default 12), RWAIT (s to wait for replay
in replay-stuck,
# default 15), CENSUS=1 (gdb census of stale caches, see coverage-matrix.txt).
# For tcp_user_timeout use EXTRA_CONF="tcp_user_timeout = 10s" WINDOW=40
RWAIT=40 and read
# since_block_ms (time from the backend blocking to promotion / replay
passing it).
set -u
CASE=$1 PP=${2:-61500}; SP=$((PP + 1))
B=${PGBIN:?set PGBIN to the bin directory of the build under test}
HERE=$(cd "$(dirname "$0")" && pwd); STALL=${STALL:-$HERE/stall.py}
RUN=$(mktemp -d); WINDOW=${WINDOW:-12}
trap 'down; rm -rf "$RUN"; exit 143' TERM INT
now_ms() { date +%s%3N; }
q() { "$B/psql" -X -h 127.0.0.1 -p "$1" -d postgres -Atq -c "$2"; }
wait_until() { local dl=$(( $(now_ms) + $1 * 1000 ))
while [ "$(now_ms)" -lt "$dl" ]; do
[ "$(q "$2" "$3" 2>/dev/null)" = "$4" ] && return 0; sleep 0.05; done
return 1; }
down() { for p in ${KIDS:-}; do kill -9 "$p" 2>/dev/null; wait "$p"
2>/dev/null; done
for d in "$RUN/standby" "$RUN/primary"; do
[ -f "$d/postmaster.pid" ] && "$B/pg_ctl" -D "$d" -m immediate -w stop
>/dev/null 2>&1; done; }
fail() { echo "case=$CASE result=SETUP_FAIL($1)"; down; rm -rf "$RUN"; exit 1; }
conf() { printf "port = %s\nlisten_addresses = '127.0.0.1'\n" "$2"
printf "unix_socket_directories = '%s'\n" "$1"
printf "wal_level = replica\nmax_wal_senders = 10\nhot_standby = on\nfsync =
off\n"
[ -n "${EXTRA_CONF:-}" ] && printf "%b\n" "$EXTRA_CONF"; } >>
"$1/postgresql.conf"
caught_up() { local l; l=$(q "$PP" "select pg_current_wal_insert_lsn()")
wait_until "$1" "$SP" "select pg_last_wal_replay_lsn() >= '$l'" t; }
primary_slot() { # $1 = failover true/false
if [ "$1" = true ]; then
q "$PP" "select pg_create_logical_replication_slot('fo', 'test_decoding',
false, false, true)"
else q "$PP" "select pg_create_logical_replication_slot('p1',
'test_decoding')"; fi >/dev/null; }
standby_slot() { # creating a slot on a standby waits for a running-xacts
record from the primary
rm -f "$RUN/mk"; { q "$SP" "select pg_create_logical_replication_slot('sb',
'test_decoding')" \
> "$RUN/mk.out" 2>&1; touch "$RUN/mk"; } &
for _ in $(seq 100); do q "$PP" "select pg_log_standby_snapshot()" >/dev/null
[ -f "$RUN/mk" ] && break; sleep 0.1; done; }
wait_synced() { for _ in $(seq 120); do
[ "$(q "$SP" "select count(*) from pg_replication_slots where slot_name =
'fo' and synced
and not temporary")" = 1 ] && return 0
q "$PP" "select pg_log_standby_snapshot()" >/dev/null
q "$PP" "select pg_replication_slot_advance('fo', pg_current_wal_lsn())"
>/dev/null 2>&1
sleep 0.5; done; return 1; }
# Wait until the stuck backend stays blocked: its socket stops moving, and a
wakeup no longer
# lets it reach CHECK_FOR_INTERRUPTS() (it would log its memory contexts if it
did).
settle() { local last="" cur same=0 off
for _ in $(seq 60); do
cur=$(ss -tinH "( sport = :$SP and dport = :$CLPORT )" | grep -o
'bytes_acked:[0-9]*')
if [ -n "$cur" ] && [ "$cur" = "$last" ]; then same=$((same + 1)); else
same=0; fi
[ $same -ge 4 ] && break; last=$cur; sleep 0.5; done
for _ in $(seq 20); do off=$(stat -c %s "$RUN/standby.log")
q "$SP" "select pg_log_backend_memory_contexts($HPID)" >/dev/null; sleep 0.5
tail -c +$((off + 1)) "$RUN/standby.log" |
grep -q "logging memory contexts of PID $HPID" || return 0
wait_until 10 "$SP" "select wait_event from pg_stat_activity where pid =
$HPID" ClientWrite
done; }
holder() {
python3 -I "$STALL" "$SP" postgres \
"COPY (select generate_series(1, 200000000), repeat('x', 200)) TO STDOUT" >
"$RUN/cl.out" 2>&1 &
CPID=$!; KIDS="${KIDS:-} $CPID"
for _ in $(seq 1000); do grep -q '^SENT' "$RUN/cl.out" && break; sleep 0.01;
done
HPID=$(awk '/^SENT/{print $2}' "$RUN/cl.out"); CLPORT=$(awk '/^SENT/{print
$3}' "$RUN/cl.out")
wait_until 20 "$SP" "select wait_event from pg_stat_activity where pid =
$HPID" ClientWrite ||
return 1
TBLK=$(now_ms); settle; }
census() { # stale = processes whose XLogLogicalInfo differs from 1 (needs gdb
and debug symbols)
[ "${CENSUS:-0}" = 1 ] || { echo "-"; return; }
local pm st=0 tot=0 v; pm=$(head -1 "$RUN/standby/postmaster.pid")
for p in $(pgrep -P "$pm"); do [ "$p" = "${HPID:-x}" ] && [ "$1" = after ] &&
continue
v=$(gdb -p "$p" -batch -ex 'printf "XLI=%d\n", XLogLogicalInfo' 2>/dev/null
|
grep -o 'XLI=[0-9]')
[ -z "$v" ] && continue; tot=$((tot + 1)); [ "$v" != XLI=1 ] && st=$((st +
1)); done
echo "$st/$tot"; }
case $CASE in failover-*|crash-then-promote) SYN=true;; *) SYN=false;; esac
"$B/initdb" -D "$RUN/primary" -A trust --no-sync >/dev/null 2>&1 || fail initdb
conf "$RUN/primary" "$PP"
{ echo "host replication all 127.0.0.1/32 trust"; cat
"$RUN/primary/pg_hba.conf"; } > "$RUN/h"
mv "$RUN/h" "$RUN/primary/pg_hba.conf"
"$B/pg_ctl" -D "$RUN/primary" -l "$RUN/primary.log" -w start >/dev/null || fail
primary_start
q "$PP" "create table t(i int)" >/dev/null
[ "$SYN" = true ] && q "$PP" "select
pg_create_physical_replication_slot('phys')" >/dev/null
case $CASE in *-at-start|toggled|crash-then-promote) primary_slot "$SYN";; esac
"$B/pg_basebackup" -d "host=127.0.0.1 port=$PP dbname=postgres" -D
"$RUN/standby" -R -X stream \
--no-sync >/dev/null 2>&1 || fail basebackup
sed -i -e "s/^port = .*/port = $SP/" \
-e "s#^unix_socket_directories = .*#unix_socket_directories =
'$RUN/standby'#" \
"$RUN/standby/postgresql.conf"
[ "$SYN" = true ] && printf "%s\n" "primary_slot_name = 'phys'"
"hot_standby_feedback = on" \
"sync_replication_slots = on" >> "$RUN/standby/postgresql.conf"
"$B/pg_ctl" -D "$RUN/standby" -l "$RUN/standby.log" -w start >/dev/null || fail
standby_start
wait_until 30 "$PP" "select count(*) from pg_stat_replication where state =
'streaming'" 1 ||
fail streaming
REPLAY=-
case $CASE in
*-in-recovery) primary_slot "$SYN"; caught_up 20 || fail catchup;;
toggled) # status on at start, off (last slot dropped), then on again, all
during recovery
q "$PP" "select pg_drop_replication_slot('p1')" >/dev/null; q "$PP"
"checkpoint" >/dev/null
wait_until 20 "$PP" "show effective_wal_level" replica || fail no_disable
caught_up 20 || fail catchup; primary_slot false; caught_up 20 || fail
catchup2;;
replay-stuck) # backend stuck first, then the primary enables logical
decoding
holder || fail holder
primary_slot false; L=$(q "$PP" "select pg_current_wal_insert_lsn()");
T=$(now_ms)
if wait_until "${RWAIT:-15}" "$SP" "select pg_last_wal_replay_lsn() >=
'$L'" t; then
REPLAY="caught_up_$(( $(now_ms) - T ))ms(since_block_$(( $(now_ms) - TBLK
))ms)"
else REPLAY="stalled_${RWAIT:-15}s"
q "$SP" "select pg_terminate_backend($HPID)" >/dev/null; T=$(now_ms)
wait_until 15 "$SP" "select pg_last_wal_replay_lsn() >= '$L'" t \
&& REPLAY="$REPLAY;after_terminate_$(( $(now_ms) - T ))ms"; fi
kill -9 "$CPID" 2>/dev/null; wait "$CPID" 2>/dev/null; HPID="";;
esac
case $CASE in
failover-*) wait_synced || fail not_synced;;
local-*|toggled|replay-stuck) standby_slot;;
crash-then-promote) # synced slot plus a local one, then kill -9 a backend:
crash restart
wait_synced || fail not_synced; standby_slot
"$B/psql" -X -h 127.0.0.1 -p "$SP" -d postgres -Atqc "select pg_sleep(600)"
>/dev/null 2>&1 &
KIDS="${KIDS:-} $!"; sleep 0.5
kill -9 "$(q "$SP" "select pid from pg_stat_activity where query like
'select pg_sleep(600)%'")"
sleep 1; wait_until 30 "$SP" "select 1" 1 || fail restart
grep -q 'all server processes terminated; reinitializing'
"$RUN/standby.log" || fail no_crash;;
esac
SLOT=$(q "$SP" "select slot_name from pg_replication_slots where slot_type =
'logical'
and invalidation_reason is null order by slot_name limit 1")
[ -n "$SLOT" ] || fail no_standby_slot
holder || fail holder2
BEFORE=$(census before)
T=$(now_ms); q "$SP" "select pg_promote(false)" >/dev/null; RES=hung; PMS=-;
SINCE=-; REL=-
while [ "$(now_ms)" -lt $((T + WINDOW * 1000)) ]; do
[ "$(q "$SP" "select pg_is_in_recovery()" 2>/dev/null)" = f ] && {
RES=promoted; PMS=$(( $(now_ms) - T )); SINCE=$(( $(now_ms) - TBLK ));
break; }
sleep 0.01; done
if [ $RES = hung ]; then q "$SP" "select pg_terminate_backend($HPID)"
>/dev/null; T=$(now_ms)
wait_until 15 "$SP" "select pg_is_in_recovery()" f && REL=$(( $(now_ms) - T
)); fi
sleep 1; AFTER=$(census after)
q "$SP" "select pg_replication_slot_advance('$SLOT', pg_current_wal_lsn())"
>/dev/null 2>&1
q "$SP" "insert into t select generate_series(1, 5)" >/dev/null
DEC=$(q "$SP" "select count(*) from pg_logical_slot_get_changes('$SLOT', null,
null)
where data like 'table public.t: INSERT%'" 2>&1 | head -1)
echo "case=$CASE result=$RES promote_ms=$PMS since_block_ms=$SINCE" \
"release_after_terminate_ms=$REL" \
"replay=$REPLAY stale_before=$BEFORE stale_after=$AFTER decoded_rows=$DEC" \
"still_waiting_lines=$(grep -c 'still waiting for backend with PID'
"$RUN/standby.log")" \
"tcp_timed_out=$(grep -c 'could not send data to client: Connection timed
out' \
"$RUN/standby.log")"
down; rm -rf "$RUN"
#!/usr/bin/env python3
# stall.py: a client that gets its backend stuck in ClientWrite and stays alive.
# Used by repro-standby.sh (rows 1-5 and 9 of coverage-matrix.txt).
# usage: python3 stall.py PORT DBNAME "SQL" (TCP to 127.0.0.1, trust auth)
# It sends one simple Query, never reads the result, and prints "SENT <backend
pid> <local port>".
import os, socket, struct, sys, time
port, dbname, sql = int(sys.argv[1]), sys.argv[2], sys.argv[3]
user = os.environ.get("PGUSER") or os.environ.get("USER", "postgres")
s = socket.create_connection(("127.0.0.1", port))
def recv_exact(n):
buf = b""
while len(buf) < n:
c = s.recv(n - len(buf))
if not c:
sys.exit("server closed the connection")
buf += c
return buf
params = b"user\0" + user.encode() + b"\0database\0" + dbname.encode() + b"\0\0"
s.sendall(struct.pack("!II", 8 + len(params), 196608) + params)
pid = 0
while True:
t = recv_exact(1)
(ln,) = struct.unpack("!I", recv_exact(4))
body = recv_exact(ln - 4)
if t == b"R" and struct.unpack("!I", body[:4])[0] != 0:
sys.exit("only trust authentication is supported")
if t == b"K":
pid = struct.unpack("!I", body[:4])[0]
if t == b"E":
sys.exit("error during startup: %r" % body)
if t == b"Z":
break
q = sql.encode() + b"\0"
s.sendall(b"Q" + struct.pack("!I", 4 + len(q)) + q)
print("SENT", pid, s.getsockname()[1], flush=True)
while True:
time.sleep(3600)
# Copyright (c) 2026, PostgreSQL Global Development Group
#
# Rows 6-8 of coverage-matrix.txt: barrier waits on a primary behind a session
blocked in
# ClientWrite (pg_enable_data_checksums(), DROP DATABASE with the session on
another database,
# DROP DATABASE with a walsender stuck in a base backup). Prints one "ROW ..."
line per case.
# Run inside a PostgreSQL source tree (unpatched, + v6 or + v2): copy to
src/test/recovery/t/
# without the .txt suffix, then:
# make -C src/test/recovery check PROVE_TESTS=t/090_barrier_primary_rows.pl
PROVE_FLAGS=-v
# Env: GUC (one extra postgresql.conf line, e.g.
GUC='procsignal_barrier_timeout = 2s' for v2),
# WINDOW (seconds, default 15), REPS (default 3).
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use Time::HiRes qw(time usleep);
my $WINDOW = $ENV{WINDOW} // 15;
my $REPS = $ENV{REPS} // 3;
my $copy = "COPY (SELECT generate_series(1, 1000000000), repeat('x', 200)) TO
STDOUT";
my $node = PostgreSQL::Test::Cluster->new('p');
$node->init(allows_streaming => 1, no_data_checksums => 1);
$node->append_conf('postgresql.conf', $ENV{GUC}) if $ENV{GUC};
$node->start;
$node->safe_psql('postgres', 'CREATE DATABASE other');
sub psql_bg {
my ($db, $app, $sql, $out) = @_;
IPC::Run::start(
[ 'psql', '--no-psqlrc', '--quiet',
'--dbname' => $node->connstr($db) . "
application_name=$app",
'--command' => $sql
],
'>' => $out,
'2>' => \my $e);
}
sub pid_of {
my ($app) = @_;
$node->poll_query_until('postgres',
"SELECT wait_event = 'ClientWrite' FROM pg_stat_activity WHERE
application_name = '$app'")
or die "no $app";
$node->safe_psql('postgres',
"SELECT pid FROM pg_stat_activity WHERE application_name =
'$app'");
}
# Wake the session until it stays blocked: a woken backend can still fit one
more send on a
# Unix-domain socket, reach CHECK_FOR_INTERRUPTS() and absorb a pending barrier.
sub settle {
my ($pid) = @_;
for my $i (0 .. 20) {
my $o = -s $node->logfile;
$node->safe_psql('postgres', "SELECT
pg_log_backend_memory_contexts($pid)");
usleep(500_000);
return unless $node->log_contains("logging memory contexts of
PID $pid", $o);
$node->poll_query_until('postgres',
"SELECT wait_event = 'ClientWrite' FROM
pg_stat_activity WHERE pid = $pid");
}
die "never settled";
}
sub judge {
my ($what, $pid, $done, $off) = @_;
my $t0 = time;
my $t;
while (time - $t0 < $WINDOW) {
if ($done->()) { $t = time - $t0; last; }
usleep(20_000);
}
my $term = $node->log_contains("terminating process $pid ", $off) ? 1 :
0;
my $rel = '-';
if (!defined $t) {
$node->safe_psql('postgres', "SELECT
pg_terminate_backend($pid)");
my $t1 = time;
usleep(5_000) until $done->() || time - $t1 > 60;
$rel = sprintf('%.0fms', 1000 * (time - $t1));
}
diag(
sprintf(
"ROW %s: %s terminated_by_timeout=%d manual_release=%s",
$what, defined $t ? sprintf('CLEARED %.0fms', 1000 *
$t) : "HUNG >${WINDOW}s",
$term, $rel));
pass($what);
}
for my $r (1 .. $REPS) {
$node->safe_psql('postgres', "CREATE DATABASE d$r; CREATE DATABASE
w$r");
my $o = '';
my $h = psql_bg('other', "s_drop$r", $copy, \$o);
my $pid = pid_of("s_drop$r");
settle($pid);
my $off = -s $node->logfile;
my $oo = '';
my $op = psql_bg('postgres', 'op', "DROP DATABASE d$r", \$oo);
judge(
'DROP DATABASE (session on another db)', $pid,
sub { $op->pump_nb if $op->pumpable; !$op->pumpable }, $off);
$h->kill_kill;
$o = '';
$h = psql_bg('postgres', "s_cs$r", $copy, \$o);
$pid = pid_of("s_cs$r");
settle($pid);
$off = -s $node->logfile;
$node->safe_psql('postgres', 'SELECT pg_enable_data_checksums()');
judge(
'pg_enable_data_checksums()',
$pid,
sub {
$node->safe_psql('postgres',
"SELECT setting FROM pg_settings WHERE name =
'data_checksums'") eq 'on';
},
$off);
$h->kill_kill;
$node->safe_psql('postgres', 'SELECT pg_disable_data_checksums()');
$node->poll_query_until('postgres',
"SELECT setting = 'off' FROM pg_settings WHERE name =
'data_checksums'")
or die;
my ($bo, $be) = ('', '');
$h = IPC::Run::start(
[ 'pg_basebackup',
'--pgdata' => '-',
'--format' => 'tar',
'--wal-method' => 'none',
'--checkpoint' => 'fast',
'--dbname' => $node->connstr('postgres') . "
application_name=s_ws$r"
],
'>' => \$bo,
'2>' => \$be);
$pid = pid_of("s_ws$r");
settle($pid);
$off = -s $node->logfile;
$oo = '';
$op = psql_bg('postgres', 'op', "DROP DATABASE w$r", \$oo);
judge(
'DROP DATABASE (walsender in a base backup)', $pid,
sub { $op->pump_nb if $op->pumpable; !$op->pumpable }, $off);
$h->kill_kill;
$node->poll_query_until('postgres',
"SELECT count(*) = 0 FROM pg_stat_activity WHERE
application_name LIKE 's\\_%'");
}
done_testing();
# Copyright (c) 2026, PostgreSQL Global Development Group
#
# Negative controls of coverage-matrix.txt: DROP DATABASE (a barrier) with one
control session
# present: idle, pg_sleep(), a slow reader that keeps reading, a synchronous
replication waiter
# (no standby), and a paused COPY FROM STDIN. The session must survive; prints
"CTRL ..." lines.
# Run inside a PostgreSQL source tree (unpatched, + v6 or + v2): copy to
src/test/recovery/t/
# without the .txt suffix, then:
# make -C src/test/recovery check PROVE_TESTS=t/091_barrier_controls.pl
PROVE_FLAGS=-v
# Env: GUC (one extra postgresql.conf line, e.g.
GUC='procsignal_barrier_timeout = 2s' for v2),
# WINDOW (seconds, default 15), REPS (default 3).
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use Time::HiRes qw(time usleep);
my $WINDOW = $ENV{WINDOW} // 15;
my $REPS = $ENV{REPS} // 3;
my $node = PostgreSQL::Test::Cluster->new('c');
$node->init(allows_streaming => 1, no_data_checksums => 1);
$node->append_conf('postgresql.conf', $ENV{GUC}) if $ENV{GUC};
# only the sync rep control session waits for a synchronous standby, not
CREATE/DROP DATABASE
$node->append_conf('postgresql.conf', 'synchronous_commit = local');
$node->start;
$node->safe_psql('postgres', 'CREATE TABLE t (a int)');
my $n = 0;
sub alive {
$node->safe_psql('postgres', "SELECT count(*) FROM pg_stat_activity
WHERE pid = $_[0]") eq '1';
}
sub wpid {
my ($app, $cond) = @_;
$node->poll_query_until('postgres',
"SELECT $cond FROM pg_stat_activity WHERE application_name =
'$app'")
or die "no $app";
$node->safe_psql('postgres',
"SELECT pid FROM pg_stat_activity WHERE application_name =
'$app'");
}
# $tick->() runs while waiting (slow reader pump); $release->() after the
window if still waiting
sub run_case {
my ($what, $pid, $tick, $release) = @_;
my $db = 'x' . $n++;
$node->safe_psql('postgres', "CREATE DATABASE $db");
my $off = -s $node->logfile;
my ($o, $e) = ('', '');
my $op = IPC::Run::start(
[ 'psql', '--no-psqlrc', '--quiet',
'--dbname' => $node->connstr('postgres'),
'--command' => "DROP DATABASE $db"
],
'>' => \$o,
'2>' => \$e);
my $t0 = time;
my $t;
while (time - $t0 < $WINDOW) {
$tick->();
$op->pump_nb if $op->pumpable;
if (!$op->pumpable) { $t = time - $t0; last; }
usleep(50_000);
}
my $al = alive($pid) ? 1 :
0;
my $term = $node->log_contains("terminating process $pid ", $off) ? 1 :
0;
my $bw =
$node->log_contains("still waiting for backend with PID $pid to
accept ProcSignalBarrier",
$off) ? 1 : 0;
my $after = '-';
if (!defined $t) {
my $t1 = time;
$release->();
while ($op->pumpable && time - $t1 < 60) { $op->pump_nb;
usleep(5_000); }
$after = sprintf('%.0fms', 1000 * (time - $t1));
$op->finish;
}
diag(
sprintf(
"CTRL %s: drop=%s survived=%d terminated=%d
barrier_wait_logged=%d drop_after_release=%s",
$what, defined $t ? sprintf('%.0fms', 1000 * $t) :
"WAITING >${WINDOW}s",
$al, $term, $bw, $after));
pass($what);
}
for my $r (1 .. $REPS) {
my $idle = $node->background_psql('postgres');
my $ip = $idle->query_safe('SELECT pg_backend_pid()');
run_case('idle', $ip, sub { }, sub { });
$idle->quit;
my $so = '';
my $sl = IPC::Run::start(
[ 'psql', '--no-psqlrc', '--quiet',
'--dbname' => $node->connstr('postgres') . "
application_name=sl$r",
'--command' => 'SELECT pg_sleep(600)'
],
'>' => \$so,
'2>' => \my $se);
run_case('pg_sleep', wpid("sl$r", "wait_event = 'PgSleep'"), sub { },
sub { });
$sl->kill_kill;
my $ro = '';
my $rd = IPC::Run::start(
[ 'psql', '--no-psqlrc', '--quiet',
'--dbname' => $node->connstr('postgres') . "
application_name=rd$r",
'--command' =>
"COPY (SELECT generate_series(1, 1000000000),
repeat('x', 200)) TO STDOUT"
],
'>' => \$ro,
'2>' => \my $re);
my $rp = wpid("rd$r", "wait_event = 'ClientWrite'");
run_case(
'slow reader', $rp,
sub { $rd->pump_nb; $ro = ''; },
sub { $node->safe_psql('postgres', "SELECT
pg_terminate_backend($rp)") });
$rd->kill_kill;
$node->safe_psql('postgres', "ALTER SYSTEM SET
synchronous_standby_names = 'nosuch'");
$node->reload;
my $yo = '';
my $sy = IPC::Run::start(
[ 'psql', '--no-psqlrc', '--quiet',
'--dbname' => $node->connstr('postgres') . "
application_name=sy$r",
'--command' => 'SET synchronous_commit = on; INSERT
INTO t VALUES (1)'
],
'>' => \$yo,
'2>' => \my $ye);
run_case(
'sync-rep waiter',
wpid("sy$r", "wait_event = 'SyncRep'"),
sub { },
sub {
$node->safe_psql('postgres', 'ALTER SYSTEM RESET
synchronous_standby_names');
$node->reload;
});
$sy->finish;
my ($ci, $co) = ("COPY t FROM STDIN;\n1\n", '');
my $cp = IPC::Run::start(
[ 'psql', '--no-psqlrc',
'--quiet', '--dbname' => $node->connstr('postgres') . "
application_name=cp$r"
],
'<' => \$ci,
'>' => \$co,
'2>' => \my $ce);
$cp->pump_nb;
run_case(
'COPY FROM STDIN paused',
wpid("cp$r", "wait_event = 'ClientRead' AND state = 'active'"),
sub { }, sub { $ci .= "2\n\\.\n"; $cp->pump_nb for 1 .. 5; });
$cp->finish;
}
done_testing();
