On Thu, 12 Nov 2026 at 02:41, Greg Burd <[email protected]> wrote: > > Hello, > > Following up on the torn-page problem Shihao found and Vignesh > reproduced, and on Amit's comments. I reproduced it, tested the fixes, > and have reworked the series so the torn-page fix comes first. > > The case of the torn-page: The bug is on master today, independent of > the pin-only rework. I ran Vignesh's reproducer (adapted so it finds > the primary bucket page from a record that registers it, rather than > from the removed bucket_blkno field) against three trees: > > - plain master: recovery fails replaying the overflow DELETE, "invalid > page in block 7", before the later full-page image of block 7. > - the pin-only series: same failure, reached through the appended > block number instead of block 0. > - master + the fix: recovery succeeds and the index returns the right > rows. > > The hazard was spotted during the original hash WAL work. In 2017 > Robert raised it on the write-ahead-logging thread [1]: > > "the page is registered and therefore an FPI will be taken if this is > the first reference to this page since the checkpoint, but > hash_xlog_delete won't restore the FPI. I think that exposes us to a > torn-page hazard. _hash_freeovflpage/hash_xlog_squeeze_page seems to > have the same disease, as does > _hash_squeezebucket/hash_xlog_move_page_contents." > > Redo was then switched to XLogReadBufferForRedoExtended() so it restores > an image when the record has one, but these records register the primary > bucket page pin-only with REGBUF_NO_IMAGE, so there is nothing to > restore and the RBM_NORMAL read still reaches the torn page. So the > hazard stayed open; Shihao and Vignesh have now hit it. > > On the fix: RBM_ZERO_ON_ERROR (Vignesh) does fix the reproducer, but as > Amit noted it is documented for non-critical forks where the caller > repairs, and this is a main-fork index page. Guarding the read with > InHotStandby (Amit's suggestion) also fixes it and is the right one: > > - The read exists only to take the cleanup lock, which only matters to > serialize against a scan on a hot standby. When hot standby is not > active no such scan can exist, so neither the read nor the lock is > needed. > - That is exactly the window where the torn-page hazard lives. The > repairing image always precedes the point where hot standby becomes > active, so once InHotStandby is true any earlier image has already > been replayed and the page is no longer torn. > - It matches existing practice: nbtree and gist already gate their > redo cleanup-lock work on InHotStandby. > > Since the bug is pre-existing, the fix is patch 0001: it stands on its > own against master, applies with or without the rework, and (the > back-branch question aside) is separate from the refactor.
The fix skips reading and acquiring the cleanup lock on the primary bucket page unless 'InHotStandby' is true. This works for crash recovery, and the first test (068a) passes. However, 'InHotStandby' does not mean that hot standby is ready to accept queries. It becomes true as soon as the first 'XLOG_RUNNING_XACTS' record is replayed, which happens shortly after the redo pointer is reached, well before recovery reaches consistency. With 'hot_standby = on', redo can therefore still read the page during this window. A base backup can contain a torn copy of the page if it is written while the backup files are being copied. The full-page image for that page is then present in WAL generated after the backup starts. The DELETE record for the overflow page can precede that image, while the standby reaches consistency only after the backup ends, by which time the image should have been replayed. The attached test (068b) reproduces this scenario: a) Takes a low-level backup using 'pg_backup_start()'. b) While the backup is in progress, runs VACUUM to generate a DELETE record for the overflow page, followed by another VACUUM that modifies the primary page for the first time since the backup's checkpoint. The latter record therefore includes a full-page image of the primary page. c) Tears the backup copy of the primary page. d) Starts a standby with 'hot_standby = on' using that backup. With the fix, the standby fails to start: FATAL: invalid page in block 7 of relation "base/5/16387" CONTEXT: WAL redo at 0/020099A0 for Hash/DELETE: ... is_primary F; blkref #0: ... blk 7; blkref #1: ... blk 10 FPW The attached patch (0001) uses 'HotStandbyActiveInReplay()' instead. This becomes true only after recovery has reached consistency and the postmaster has been notified that connections can be accepted. Since no scans can be running before that point, there is no need to read or lock the primary bucket page earlier. By then, any full-page image needed to repair a torn page should also have been replayed. How about changing InHotStandby to HotStandbyActiveInReplay to fix this issue? Regards, Vignesh
# Copyright (c) 2026, PostgreSQL Global Development Group
# Test a standby that starts from a base backup whose copy of the primary
# bucket page of a hash bucket is torn, and the full-page image that repairs it
# comes after the record that reads it.
#
use strict;
use warnings FATAL => 'all';
use File::Path qw(rmtree);
use IPC::Run;
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
my $primary = PostgreSQL::Test::Cluster->new('primary');
$primary->init(allows_streaming => 1);
$primary->append_conf(
'postgresql.conf', qq{
autovacuum = off
fsync = off
full_page_writes = on
checkpoint_timeout = 1h
});
$primary->start;
is($primary->safe_psql('postgres', 'SHOW data_checksums'),
'on', 'premise: data checksums are on');
# One bucket receives every key: a primary page holding about 407 entries and
# an overflow page with the rest.
$primary->safe_psql(
'postgres', q{
CREATE TABLE t (id int, k int) WITH (autovacuum_enabled = false);
CREATE INDEX t_i ON t USING hash (k);
INSERT INTO t SELECT i, 1 FROM generate_series(1, 600) i;
});
my $relpath = $primary->safe_psql('postgres',
q{SELECT pg_relation_filepath('t_i')});
my $bs = $primary->safe_psql('postgres', 'SHOW block_size');
# Start a low-level backup, which does a checkpoint, and copy the files.
my $backup_name = 'torn';
my $backup_dir = $primary->backup_dir . '/' . $backup_name;
my $psql = $primary->background_psql('postgres');
$psql->query_safe("SET client_min_messages TO WARNING");
$psql->set_query_timer_restart;
$psql->query_safe("select pg_backup_start('torn primary page', true)");
my $redo = $primary->safe_psql('postgres',
'SELECT redo_lsn FROM pg_control_checkpoint()');
PostgreSQL::Test::RecursiveCopy::copypath($primary->data_dir, $backup_dir);
unlink("$backup_dir/postmaster.pid")
or BAIL_OUT("unable to unlink $backup_dir/postmaster.pid");
unlink("$backup_dir/postmaster.opts")
or BAIL_OUT("unable to unlink $backup_dir/postmaster.opts");
rmtree("$backup_dir/pg_wal")
or BAIL_OUT("unable to unlink contents of $backup_dir/pg_wal");
mkdir("$backup_dir/pg_wal");
# While the backup is in progress: the entries of ids 451-600 are on the
# overflow page, those of ids 1-50 on the primary page. The DELETE for the
# overflow page comes first; the VACUUM of the primary page then changes it for
# the first time since the checkpoint of the backup, so its record carries the
# full-page image of the page.
$primary->safe_psql('postgres', 'DELETE FROM t WHERE id > 450');
$primary->safe_psql('postgres', 'VACUUM t');
$primary->safe_psql('postgres', 'DELETE FROM t WHERE id <= 50');
$primary->safe_psql('postgres', 'VACUUM t');
$primary->safe_psql('postgres', 'SELECT pg_switch_wal()');
# Read the Hash records after the redo pointer of the backup.
my ($stdout, $stderr);
IPC::Run::run(
[
'pg_waldump',
'--path' => $primary->data_dir . '/pg_wal',
'--start' => $redo,
'--rmgr' => 'Hash',
],
'>' => \$stdout,
'2>' => \$stderr);
my @records = grep { /^rmgr: Hash/ } split(/^(?=rmgr: )/m, $stdout);
# The primary page: a DELETE or SQUEEZE_PAGE marked is_primary T has it as
# block 1, the page that is changed.
my $pblk;
for my $r (@records)
{
if ($r =~ /desc: (?:DELETE|SQUEEZE_PAGE) .*is_primary T/
&& $r =~ /blkref #1: rel \S+ blk (\d+)/)
{
$pblk = $1;
last;
}
}
ok(defined $pblk, 'premise: found the primary bucket page')
or diag $stdout;
# The first DELETE for an overflow page.
my ($idx_delete) =
grep { $records[$_] =~ /desc: DELETE .*is_primary F/ } 0 .. $#records;
ok(defined $idx_delete,
'premise: VACUUM wrote a DELETE for an overflow page')
or diag $stdout;
# Where the first full-page image of the primary page is, relative to the
# DELETE for the overflow page.
my ($idx_fpi) =
defined $pblk
? grep { $records[$_] =~ /blkref #\d+: rel \S+ blk $pblk FPW/ }
0 .. $#records
: ();
ok(defined $idx_fpi,
'premise: the primary page got a full-page image after the checkpoint')
or diag $stdout;
if (defined $idx_delete && defined $idx_fpi)
{
cmp_ok($idx_delete, '<', $idx_fpi,
'premise: the DELETE comes before the image of the primary page');
}
# The copy of the primary page in the backup is torn: keep the first half of
# its block and overwrite the second half.
if (defined $pblk)
{
my $path = "$backup_dir/$relpath";
open(my $fh, '+<:raw', $path) or die "could not open $path: $!";
sysseek($fh, $pblk * $bs + $bs / 2, 0) or die "seek: $!";
my $junk = join('', map { chr(int(rand(256))) } 1 .. $bs / 2);
syswrite($fh, $junk) == $bs / 2 or die "write: $!";
close($fh);
}
# End the backup, and save backup_label into the backup.
my $backup_label = $psql->query_safe("select labelfile from pg_backup_stop()");
$psql->quit;
open(my $fh, '>', "$backup_dir/backup_label")
or BAIL_OUT("unable to create $backup_dir/backup_label: $!");
print $fh $backup_label;
close($fh);
# Start a hot standby from the backup. Replay starts at the redo pointer of
# the backup, replays the DELETE for the overflow page first, and reaches the
# consistency point only after the end of the backup, after the full-page
# image of the primary page.
my $standby = PostgreSQL::Test::Cluster->new('standby');
$standby->init_from_backup($primary, $backup_name, has_streaming => 1);
$standby->append_conf('postgresql.conf', 'hot_standby = on');
my $started = $standby->start(fail_ok => 1);
ok($started, 'the standby starts');
if ($started)
{
$primary->wait_for_catchup($standby);
is( $standby->safe_psql(
'postgres',
q{SET enable_seqscan = off; SET enable_bitmapscan = off;
SELECT count(*) FROM t WHERE k = 1}),
'400',
'the hash index gives the right result on the standby');
}
else
{
my $log = slurp_file($standby->logfile);
diag("the standby did not start; the log has:\n"
. join('', grep { /invalid page|FATAL|WAL redo|consistent recovery/ } split(/^/m, $log)));
}
done_testing();
Use_HotStandbyActiveInReplay_to_skip_primary_bucket.patch
Description: Binary data
