Hi Sai Pranav, Fujii-san,

Thanks for testing that, Sai Pranav!

I have attached v5 split into two patches so Fujii-san can easily pick
either option:
1. v5-0001 contains just the reconstruct.c fix (same as v4).
2. v5-0002 adds the deterministic zero-block append test to
011_ib_truncation.pl (with the primary stopped while appending to the file,
and a comment explaining why the zero page is appended directly).

I am okay with either patch (with or without the test) though I feel the
test adds value by catching any issues immediately.
Thanks.

Regards,
Rajesh Kumar Thandapani

On Tue, Nov 17, 2026 at 4:17 AM Raja Sai pranav <[email protected]>
wrote:

> Hi Rajesh,
>
> I tried the zero-block variant on current master (5ffc821037) with v4's
> reconstruct.c, adding the append plus a second incremental backup and a
> third node to
> https://www.google.com/url?q=http://011_ib_truncation.pl&source=gmail&ust=1794975425465000&sa=E,
> and ran it with shared_buffers at 8MB and 128MB. With the fix it passes at
> both sizes; without the fix it fails at both with 8 rows instead of 7. So
> unlike the v3 subtest it reproduces the bug deterministically, and nothing
> in it depends on COPY batching.
>
> I think it is worth adding. The only thing I would add is a comment saying
> why a zero page is appended to the heap file directly, since that is the
> exact shape the bug mishandles (an unwritten block past the truncation
> point).
>
> Regards,
> Sai Pranav
>
> On Mon, Nov 16, 2026 07:06 PM, Rajesh Kumar Thandapani <
> [email protected]> wrote:
>
>> Hi Fujii-san, Sai Pranav,
>>
>> Thank you both for checking that. I agree that relying on COPY batching and
>> shared_buffers makes the test too sensitive to internal heuristics, so I have
>> removed the test change in the attached v4 patch.
>>
>> One alternative I was wondering about: in 011_ib_truncation.pl, after taking
>> the first incremental backup (where table t is truncated from 8 blocks to
>> 7 blocks), we could append one block of zeroes directly to the relation file
>> on disk from Perl before taking a second incremental backup:
>>
>> my $relpath =
>>     $primary->safe_psql('postgres', "SELECT pg_relation_filepath('t');");
>>   my $segfile = $primary->data_dir . "/$relpath";
>>   $segfile .= ".1" if -f "$segfile.1";
>>   open(my $fh, '>>:raw', $segfile) or die "open $segfile: $!";
>>   print $fh ("\0" x $block_size);
>>   close($fh);
>>
>> That simulates an unwritten zero-filled page at block 7 without relying on
>> COPY or shared_buffers, and deterministically reproduces the bug (returning
>> 8 rows instead of 7 without the fix).
>>
>> Do you think a test like that is worth adding, or is it better to leave out
>> the test and go with v4 as-is?
>> I am okay with either option; I'm just sharing my thoughts on adding a
>> test without assuming any internal huristics.
>>
>> Thanks.
>>
>> Regards,
>> Rajesh Kumar
>>
>> On Mon, Nov 16, 2026 at 11:33 AM Raja Sai pranav <
>> [email protected]> wrote:
>>
>>> Hi Fujii-san,
>>>
>>> I checked this by running
>>> https://www.google.com/url?q=http://011_ib_truncation.pl&source=gmail&ust=1794895380937000&sa=E
>>> against current master without the reconstruct.c change, varying $copy_rows
>>> (1, 4, 16, 17, 18, 32, 100) and shared_buffers (8MB, 16MB, 128MB). The new
>>> subtest catches the bug in only 6 of the 21 combinations (copy_rows 16-17
>>> with 8MB, 17-18 with 16MB and 128MB); in all the others it passes without
>>> the fix. So I agree it depends too much on COPY's batching to be a reliable
>>> regression test, and dropping it makes sense, since the reproducer upthread
>>> already demonstrates the bug and the fix.
>>>
>>> Regards,
>>> Sai Pranav
>>>
>>> On Mon, Nov 16, 2026 10:42 AM, Fujii Masao <[email protected]>
>>> wrote:
>>>
>>>> On Sun, Nov 8, 2026 at 12:21 AM Raja Sai pranav
>>>> <[email protected]> wrote:
>>>> > - v3 now adds the bulk-extension regression test I'd flagged as
>>>> missing: the
>>>> > node3 / incr2-incr4 chain that bumps shared_buffers so COPY
>>>> bulk-extends the
>>>> > relation past an intermediate truncation point, leaving zero-filled
>>>> unwritten
>>>> > pages in the truncated range.
>>>>
>>>> I just wonder if the added test might be fragile. It relies on internal
>>>> logics, such as COPY's batching behavior ($copy_rows = 17), buffer pin
>>>> limits (shared_buffers = 8MB), and the bulk-extension strategy leaving
>>>> pages unwritten. Changes to any of these could easily make the test pass
>>>> even without the fix. No?
>>>>
>>>> If so, isn't it better to leave out the test for now? Also because
>>>> the reproducer has already demonstrated the bug and the fix.
>>>>
>>>> Regards,
>>>>
>>>>
>>>> --
>>>> Fujii Masao
>>>>
>>>
From 35770da48efe7cdb4d208e8f1a9c1b1096045d37 Mon Sep 17 00:00:00 2001
From: Rajesh Kumar Thandapani <[email protected]>
Date: Mon, 16 Nov 2026 13:12:36 +0000
Subject: [PATCH v5 1/2] Fix pg_combinebackup with truncated intermediate
 backups.

When reconstructing a relation from a chain of backups,
reconstruct_from_incremental_file() walks backward from the latest
incremental backup to the full backup, filling sourcemap[] for any
block b < latest_source->truncation_block_length that has not yet been
assigned a source.

However, if an intermediate incremental backup truncated the relation
(s->truncation_block_length < latest_source->truncation_block_length)
and a later backup extended the relation again without writing every
block in the truncated range (for example, via bulk relation extension),
those unwritten blocks still had sourcemap[b] == NULL when the loop
moved past the intermediate backup. As a result, they were sourced from
an earlier backup taken before the truncation, resurrecting stale rows.

Fix by tracking the current truncation_block_length upper bound as we
walk backward through the backup chain and tightening it whenever an
incremental backup has a smaller s->truncation_block_length.

Author: Rajesh Kumar Thandapani <[email protected]>
Reviewed-by: Raja Sai Pranav <[email protected]>
Reviewed-by: Fujii Masao <[email protected]>
Bug: #19776
Reported-by: Ke <[email protected]>
Discussion: https://postgr.es/m/[email protected]
---
 src/bin/pg_combinebackup/reconstruct.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/src/bin/pg_combinebackup/reconstruct.c b/src/bin/pg_combinebackup/reconstruct.c
index 5900c5b..f33ad2a 100644
--- a/src/bin/pg_combinebackup/reconstruct.c
+++ b/src/bin/pg_combinebackup/reconstruct.c
@@ -105,6 +105,7 @@ reconstruct_from_incremental_file(char *input_filename,
 	rfile	  **sourcemap;
 	off_t	   *offsetmap;
 	unsigned	block_length;
+	unsigned	truncation_block_length;
 	unsigned	sidx = n_prior_backups;
 	bool		full_copy_possible = true;
 	int			copy_source_index = -1;
@@ -128,6 +129,7 @@ reconstruct_from_incremental_file(char *input_filename,
 	latest_source = make_incremental_rfile(input_filename);
 	source[n_prior_backups] = latest_source;
 	block_length = find_reconstructed_block_length(latest_source);
+	truncation_block_length = latest_source->truncation_block_length;
 
 	/*
 	 * For each block in the output file, we need to know from which file we
@@ -224,7 +226,7 @@ reconstruct_from_incremental_file(char *input_filename,
 			 * later backup is one that we can't validate.
 			 */
 			blocklength = sb.st_size / BLCKSZ;
-			for (b = 0; b < latest_source->truncation_block_length; ++b)
+			for (b = 0; b < truncation_block_length; ++b)
 			{
 				if (sourcemap[b] == NULL && b < blocklength)
 				{
@@ -264,7 +266,7 @@ reconstruct_from_incremental_file(char *input_filename,
 		{
 			BlockNumber b = s->relative_block_numbers[i];
 
-			if (b < latest_source->truncation_block_length &&
+			if (b < truncation_block_length &&
 				sourcemap[b] == NULL)
 			{
 				sourcemap[b] = s;
@@ -278,6 +280,19 @@ reconstruct_from_incremental_file(char *input_filename,
 				full_copy_possible = false;
 			}
 		}
+
+		/*
+		 * Blocks at or beyond this incremental file's truncation_block_length
+		 * were either not present or truncated away before this backup, so
+		 * their contents in any earlier backup are stale and must not be
+		 * used. This also prevents copying an earlier full file whole even if
+		 * its size happens to match latest_source->truncation_block_length.
+		 */
+		if (s->truncation_block_length < truncation_block_length)
+		{
+			truncation_block_length = s->truncation_block_length;
+			full_copy_possible = false;
+		}
 	}
 
 	/*
-- 
2.56.0.679.gc527c65306-goog

From 31114c4bc6235ac8e5046946c03dcf08cff27be7 Mon Sep 17 00:00:00 2001
From: Rajesh Kumar Thandapani <[email protected]>
Date: Tue, 17 Nov 2026 05:09:03 +0000
Subject: [PATCH v5 2/2] Add regression test for truncated intermediate
 backups.

Extend 011_ib_truncation.pl to cover a backup chain where an
intermediate incremental backup truncates a relation and a subsequent
incremental backup sees the relation extended again with an unwritten
zero-filled block (appended directly to the stopped primary's heap file
to avoid relying on COPY or bulk-extension heuristics).

Author: Rajesh Kumar Thandapani <[email protected]>
Reviewed-by: Raja Sai Pranav <[email protected]>
Bug: #19776
Discussion: https://postgr.es/m/[email protected]
---
 .../pg_combinebackup/t/011_ib_truncation.pl   | 38 +++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/src/bin/pg_combinebackup/t/011_ib_truncation.pl b/src/bin/pg_combinebackup/t/011_ib_truncation.pl
index 1174928..999560b 100644
--- a/src/bin/pg_combinebackup/t/011_ib_truncation.pl
+++ b/src/bin/pg_combinebackup/t/011_ib_truncation.pl
@@ -115,7 +115,45 @@ my $restored_count =
 is($restored_count, $rows_after_truncation,
 	'Restored backup has correct row count');
 
+my $relpath =
+  $primary->safe_psql('postgres', "SELECT pg_relation_filepath('t');");
+
 $primary->stop;
 $restored->stop;
 
+# Test that blocks truncated away in an intermediate incremental backup are
+# not resurrected from an earlier backup when a later backup extends the
+# relation with unwritten zero-filled pages (for example, via bulk relation
+# extension).  Append a zero-filled block directly to the heap file on disk
+# rather than relying on COPY or bulk-extension heuristics so that block 7 is
+# guaranteed to exist past incr's truncation point without any WAL record
+# referencing it.
+my $segfile = $primary->data_dir . "/$relpath";
+$segfile .= ".1" if -f "$segfile.1";
+open(my $fh, '>>:raw', $segfile) or die "open $segfile: $!";
+print $fh ("\0" x $block_size);
+close($fh) or die "close $segfile: $!";
+
+$primary->start;
+
+my $incr_backup = "$backup_path/incr";
+$primary->backup('incr2',
+	backup_options => [ '--incremental', "$incr_backup/backup_manifest" ]);
+
+$primary->stop;
+
+my $restored2 = PostgreSQL::Test::Cluster->new('node3');
+$restored2->init_from_backup(
+	$primary, 'incr2',
+	combine_with_prior => [ 'full', 'incr' ]);
+$restored2->start();
+
+$restored_count =
+  $restored2->safe_psql('postgres', "SELECT count(*) FROM t;");
+is($restored_count, $rows_after_truncation,
+	'Restored backup does not resurrect blocks truncated in an intermediate backup'
+);
+
+$restored2->stop;
+
 done_testing();
-- 
2.56.0.679.gc527c65306-goog

Reply via email to