On 1/20/25 04:24, Michael Paquier wrote:
And done this as well. The buildfarm looks OK with it for the moment.
I'll keep an eye on things, including the CF bot.
Hi Michael and Alexander!
I think I found potential instability in tap test 039_end_of_wal.pl
according to this thread activity,
please take a look at attached reproducing patch. Maybe reproducing
looks a bit tricky, but it seems
that occurrence of an error depends on the size of generated WAL. Plz
take into account that Postgres
was configured without compression:
./configure --prefix=/home/maxim/master --enable-tap-tests
--enable-cassert --with-libxml --enable-debug CFLAGS='-O0 -Wall'
--with-icu --enable-injection-points
Patch was applied to SHA cfdd09589b9, I am not sure that reproducing can
occur on another commits, because
it depends from WAL size and I tried to adjust it to fit the problematic
values.
Then test 039_end_of_wal.pl failed
make check PROVE_TESTS=t/039_end_of_wal.pl
...
# +++ tap check in src/test/recovery +++
t/039_end_of_wal.pl .. 3/?
# Failed test 'xl_prev bad'
# at t/039_end_of_wal.pl line 189.
# Looks like you failed 1 test of 4.
t/039_end_of_wal.pl .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/4 subtests
Test Summary Report
-------------------
t/039_end_of_wal.pl (Wstat: 256 (exited 1) Tests: 4 Failed: 1)
Failed test: 4
Non-zero exit status: 1
In logs instead of
record with incorrect prev-link 0/DEADBEEF at
I see
invalid magic number 0000 in WAL segment 000000010000000000000002, LSN
0/2018000, offset 98304
I've investigated it and found that insert_lsn in
advance_wal_out_of_record_splitting_zone can be on the
not initialized WAL page, then if we manually in perl exec write_wal to
this LSN, operation will be succeded
but postgres can't read rec, because it placed on not initialized page
and invalid magic number will be fired.
Now we don't see this issue, but in future if generated WAL will change,
this issue can fire.
As fix I suggest to control in advance_wal_out_of_record_splitting_zone
that end_lsn of last emitted record
should be on the same page with next insert lsn. Patch was attached, it
you have some other ideas, plz share it.
Best regards,
Maksim Melnikov
diff --git a/src/test/recovery/t/039_end_of_wal.pl
b/src/test/recovery/t/039_end_of_wal.pl
index 47f9bb15e03..0846e11f355 100644
--- a/src/test/recovery/t/039_end_of_wal.pl
+++ b/src/test/recovery/t/039_end_of_wal.pl
@@ -177,7 +177,8 @@ ok( $node->log_contains(
"xl_tot_len short at end-of-page");
# Need more pages, but xl_prev check fails first.
-$node->emit_wal(0);
+$end_lsn = $node->emit_wal(980);
+$end_lsn = $node->emit_wal(0);
$end_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
$node->stop('immediate');
$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
@@ -188,179 +189,5 @@ ok( $node->log_contains(
"record with incorrect prev-link 0/DEADBEEF at .*", $log_size),
"xl_prev bad");
-# xl_crc check fails.
-$node->emit_wal(0);
-$node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
-$end_lsn = $node->emit_wal(10);
-$node->stop('immediate');
-# Corrupt a byte in that record, breaking its CRC.
-$node->write_wal($TLI, $end_lsn - 8, $WAL_SEGMENT_SIZE, '!');
-$log_size = -s $node->logfile;
-$node->start;
-ok( $node->log_contains(
- "incorrect resource manager data checksum in record at .*",
$log_size
- ),
- "xl_crc bad");
-
-
-###########################################################################
-note "Multi-page end-of-WAL detection, header is not split";
-###########################################################################
-
-# This series of tests requires a valid xl_prev set in the record header
-# written to WAL.
-
-# Good xl_prev, we hit zero page next (zero magic).
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
-$end_lsn = $node->emit_wal(0);
-$node->stop('immediate');
-$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
- build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
-$log_size = -s $node->logfile;
-$node->start;
-ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
- "xlp_magic zero");
-
-# Good xl_prev, we hit garbage page next (bad magic).
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
-$end_lsn = $node->emit_wal(0);
-$node->stop('immediate');
-$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
- build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
-$node->write_wal($TLI, start_of_next_page($end_lsn),
- $WAL_SEGMENT_SIZE, build_page_header(0xcafe, 0, 1, 0));
-$log_size = -s $node->logfile;
-$node->start;
-ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
- "xlp_magic bad");
-
-# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
-# xlp_pageaddr).
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
-$end_lsn = $node->emit_wal(0);
-$node->stop('immediate');
-$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
- build_record_header(2 * 1024 * 1024 * 1024, 0, $prev_lsn));
-$node->write_wal($TLI, start_of_next_page($end_lsn),
- $WAL_SEGMENT_SIZE, build_page_header($XLP_PAGE_MAGIC, 0, 1,
0xbaaaaaad));
-$log_size = -s $node->logfile;
-$node->start;
-ok( $node->log_contains(
- "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
- "xlp_pageaddr bad");
-
-# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
-$end_lsn = $node->emit_wal(0);
-$node->stop('immediate');
-$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
- build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
-$node->write_wal(
- $TLI,
- start_of_next_page($end_lsn),
- $WAL_SEGMENT_SIZE,
- build_page_header(
- $XLP_PAGE_MAGIC, 0x1234, 1, start_of_next_page($end_lsn)));
-$log_size = -s $node->logfile;
-$node->start;
-ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
- "xlp_info bad");
-
-# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
-# continuation record.
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
-$end_lsn = $node->emit_wal(0);
-$node->stop('immediate');
-$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
- build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
-$node->write_wal($TLI, start_of_next_page($end_lsn),
- $WAL_SEGMENT_SIZE,
- build_page_header($XLP_PAGE_MAGIC, 0, 1, start_of_next_page($end_lsn)));
-$log_size = -s $node->logfile;
-$node->start;
-ok($node->log_contains("there is no contrecord flag at .*", $log_size),
- "xlp_info lacks XLP_FIRST_IS_CONTRECORD");
-
-# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
-# up.
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
-$end_lsn = $node->emit_wal(0);
-$node->stop('immediate');
-$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
- build_record_header(2 * 1024 * 1024 * 1024, 42, $prev_lsn));
-$node->write_wal(
- $TLI,
- start_of_next_page($end_lsn),
- $WAL_SEGMENT_SIZE,
- build_page_header(
- $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
- 1, start_of_next_page($end_lsn),
- 123456));
-$log_size = -s $node->logfile;
-$node->start;
-ok( $node->log_contains(
- "invalid contrecord length 123456 .* at .*", $log_size),
- "xlp_rem_len bad");
-
-
-###########################################################################
-note "Multi-page, but header is split, so page checks are done first";
-###########################################################################
-
-# xl_prev is bad and xl_tot_len is too big, but we'll check xlp_magic first.
-$node->emit_wal(0);
-$end_lsn = $node->advance_wal_to_record_splitting_zone($WAL_BLOCK_SIZE);
-$node->stop('immediate');
-$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
- build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
-$log_size = -s $node->logfile;
-$node->start;
-ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
- "xlp_magic zero (split record header)");
-
-# And we'll also check xlp_pageaddr before any header checks.
-$node->emit_wal(0);
-$end_lsn = $node->advance_wal_to_record_splitting_zone($WAL_BLOCK_SIZE);
-$node->stop('immediate');
-$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
- build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
-$node->write_wal(
- $TLI,
- start_of_next_page($end_lsn),
- $WAL_SEGMENT_SIZE,
- build_page_header(
- $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD, 1, 0xbaaaaaad));
-$log_size = -s $node->logfile;
-$node->start;
-ok( $node->log_contains(
- "unexpected pageaddr 0/BAAAAAAD in .*, LSN .*,", $log_size),
- "xlp_pageaddr bad (split record header)");
-
-# We'll also discover that xlp_rem_len doesn't add up before any
-# header checks,
-$node->emit_wal(0);
-$end_lsn = $node->advance_wal_to_record_splitting_zone($WAL_BLOCK_SIZE);
-$node->stop('immediate');
-$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
- build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
-$node->write_wal(
- $TLI,
- start_of_next_page($end_lsn),
- $WAL_SEGMENT_SIZE,
- build_page_header(
- $XLP_PAGE_MAGIC, $XLP_FIRST_IS_CONTRECORD,
- 1, start_of_next_page($end_lsn),
- 123456));
-$log_size = -s $node->logfile;
-$node->start;
-ok( $node->log_contains(
- "invalid contrecord length 123456 .* at .*", $log_size),
- "xlp_rem_len bad (split record header)");
done_testing();
From 86c718a44cccbacf307ae3935f4f98ac82196b8d Mon Sep 17 00:00:00 2001
From: Maksim Melnikov <[email protected]>
Date: Thu, 12 Nov 2026 10:18:21 +0300
Subject: [PATCH v1 3/3] Fix 039_end_of_wal.pl WAL manual inserts.
---
src/test/perl/PostgreSQL/Test/Cluster.pm | 24 ++++++++---
src/test/recovery/t/039_end_of_wal.pl | 40 +++++++++----------
.../recovery/t/043_no_contrecord_switch.pl | 6 +--
3 files changed, 41 insertions(+), 29 deletions(-)
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index a2930fd1f2f8..a5eefe298898 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -3143,18 +3143,30 @@ Returns the end LSN up to which WAL has advanced, in bytes.
sub advance_wal_out_of_record_splitting_zone
{
- my ($self, $wal_block_size) = @_;
+ my ($self, $wal_block_size, $end_lsn) = @_;
my $page_threshold = $wal_block_size / 4;
- my $end_lsn = $self->_get_insert_lsn();
- my $page_offset = $end_lsn % $wal_block_size;
+ my $insert_lsn = $self->_get_insert_lsn();
+
+ if (defined($end_lsn))
+ {
+ # both lsns should be on the same page to avoid edge cases when new WAL
+ # page hasn't been initialized yet.
+ while(int($insert_lsn / $wal_block_size) != int($end_lsn / $wal_block_size))
+ {
+ $end_lsn = $self->emit_wal(0);
+ $insert_lsn = $self->_get_insert_lsn();
+ }
+ }
+
+ my $page_offset = $insert_lsn % $wal_block_size;
while ($page_offset >= $wal_block_size - $page_threshold)
{
$self->emit_wal($page_threshold);
- $end_lsn = $self->_get_insert_lsn();
- $page_offset = $end_lsn % $wal_block_size;
+ $insert_lsn = $self->_get_insert_lsn();
+ $page_offset = $insert_lsn % $wal_block_size;
}
- return $end_lsn;
+ return $insert_lsn;
}
=pod
diff --git a/src/test/recovery/t/039_end_of_wal.pl b/src/test/recovery/t/039_end_of_wal.pl
index 47f9bb15e033..50008d70d8b1 100644
--- a/src/test/recovery/t/039_end_of_wal.pl
+++ b/src/test/recovery/t/039_end_of_wal.pl
@@ -141,8 +141,8 @@ note "Single-page end-of-WAL detection";
###########################################################################
# xl_tot_len is 0 (a common case, we hit trailing zeroes).
-$node->emit_wal(0);
-$end_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$end_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$node->stop('immediate');
my $log_size = -s $node->logfile;
$node->start;
@@ -152,8 +152,8 @@ ok( $node->log_contains(
"xl_tot_len zero");
# xl_tot_len is < 24 (presumably recycled garbage).
-$node->emit_wal(0);
-$end_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$end_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$node->stop('immediate');
$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE, build_record_header(23));
$log_size = -s $node->logfile;
@@ -177,8 +177,8 @@ ok( $node->log_contains(
"xl_tot_len short at end-of-page");
# Need more pages, but xl_prev check fails first.
-$node->emit_wal(0);
-$end_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$end_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$node->stop('immediate');
$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
build_record_header(2 * 1024 * 1024 * 1024, 0, 0xdeadbeef));
@@ -189,8 +189,8 @@ ok( $node->log_contains(
"xl_prev bad");
# xl_crc check fails.
-$node->emit_wal(0);
-$node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$end_lsn = $node->emit_wal(10);
$node->stop('immediate');
# Corrupt a byte in that record, breaking its CRC.
@@ -211,8 +211,8 @@ note "Multi-page end-of-WAL detection, header is not split";
# written to WAL.
# Good xl_prev, we hit zero page next (zero magic).
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$end_lsn = $node->emit_wal(0);
$node->stop('immediate');
$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
@@ -223,8 +223,8 @@ ok($node->log_contains("invalid magic number 0000 .* LSN .*", $log_size),
"xlp_magic zero");
# Good xl_prev, we hit garbage page next (bad magic).
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$end_lsn = $node->emit_wal(0);
$node->stop('immediate');
$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
@@ -238,8 +238,8 @@ ok($node->log_contains("invalid magic number CAFE .* LSN .*", $log_size),
# Good xl_prev, we hit typical recycled page (good xlp_magic, bad
# xlp_pageaddr).
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$end_lsn = $node->emit_wal(0);
$node->stop('immediate');
$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
@@ -253,8 +253,8 @@ ok( $node->log_contains(
"xlp_pageaddr bad");
# Good xl_prev, xlp_magic, xlp_pageaddr, but bogus xlp_info.
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$end_lsn = $node->emit_wal(0);
$node->stop('immediate');
$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
@@ -272,8 +272,8 @@ ok($node->log_contains("invalid info bits 1234 in .*, LSN .*,", $log_size),
# Good xl_prev, xlp_magic, xlp_pageaddr, but xlp_info doesn't mention
# continuation record.
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$end_lsn = $node->emit_wal(0);
$node->stop('immediate');
$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
@@ -288,8 +288,8 @@ ok($node->log_contains("there is no contrecord flag at .*", $log_size),
# Good xl_prev, xlp_magic, xlp_pageaddr, xlp_info but xlp_rem_len doesn't add
# up.
-$node->emit_wal(0);
-$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+$end_lsn = $node->emit_wal(0);
+$prev_lsn = $node->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
$end_lsn = $node->emit_wal(0);
$node->stop('immediate');
$node->write_wal($TLI, $end_lsn, $WAL_SEGMENT_SIZE,
diff --git a/src/test/recovery/t/043_no_contrecord_switch.pl b/src/test/recovery/t/043_no_contrecord_switch.pl
index ead41e6f40be..12072c5d64d6 100644
--- a/src/test/recovery/t/043_no_contrecord_switch.pl
+++ b/src/test/recovery/t/043_no_contrecord_switch.pl
@@ -79,9 +79,9 @@ $TLI = $primary->safe_psql('postgres',
# Get close to the end of the current WAL page, enough to fit the
# beginning of a record that spans on two pages, generating a
# continuation record.
-$primary->emit_wal(0);
-my $end_lsn =
- $primary->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE);
+my $end_lsn = $primary->emit_wal(0);
+$end_lsn =
+ $primary->advance_wal_out_of_record_splitting_zone($WAL_BLOCK_SIZE, $end_lsn);
# Do some math to find the record size that will overflow the page, and
# write it.
--
2.43.0