On Tue, Nov 3, 2026 at 1:40 AM Bertrand Drouvot <[email protected]> wrote: > > 0001 takes a different approach (than [2]) and recreates the ReadStream when > initscan() changes whether a strategy is active. Ordinary rescans continue to > reset and reuse the existing stream. It uses the existing ReadStream APIs, > which > could also make the backpatches easier.
Now that initscan() has to concern itself with the read stream because it has to reset or end the stream before changing the strategy, then change the strategy, and then make the new read stream, I think the separation between initscan() and making the read stream is no longer useful. In fact, the ping-ponging has become confusing. I think your version might actually have read_stream_reset() getting called twice in some cases since read_stream_end() also calls it. Attached v2 just moves all the read stream setup into initscan() and adds some flags to make it more clear when we want to rebuild the stream/strategy. > 0002 contains a test. I'm not sure it is worth adding to the tree given the > amount > of setup, but it demonstrates the issue. I agree that zero test coverage of this is not good. I really would have liked to find somewhere to put the test that wasn't a new TAP test file that makes yet another new cluster. Alas, neither I nor my LLM could come up with a good place. I did however, change the test to use a cursor instead of a query that does a rescan. That takes away the need for the plpgsql function and makes the test simpler IMO. pg_stat_get_backend_io() doesn't exist in 17, so we'll have to modify the test to backpatch there. - Melanie
From 3888a3a2811639983a5f487da04e41e9c10a6dac Mon Sep 17 00:00:00 2001 From: Melanie Plageman <[email protected]> Date: Tue, 17 Nov 2026 15:57:28 -0500 Subject: [PATCH v2] Recreate read stream across rescans Heap scan descriptors persist across rescans, but initscan() can add or remove the buffer access strategy when the relation size or caller flags change. The ReadStream stored the strategy pointer and sized its queue capacity based on the strategy at ReadStream creation, which was only before the first scan. This meant that on rescan, enabling a bulk-read strategy left the existing stream using its NULL strategy and old capacity. And disabling the strategy left the stream pointing to freed memory. Reset and reuse the stream when strategy use is unchanged. Otherwise, end it before changing the strategy and create a correctly sized replacement. Reported-by: Bertrand Drouvot <[email protected]> Reported-by: Nazir Bilal Yavuz <[email protected]> Backpatch-through: 17 --- src/backend/access/heap/heapam.c | 155 ++++++++++-------- src/backend/storage/aio/read_stream.c | 4 +- src/test/modules/test_misc/meson.build | 1 + .../test_misc/t/016_heap_scan_stream.pl | 54 ++++++ 4 files changed, 142 insertions(+), 72 deletions(-) create mode 100644 src/test/modules/test_misc/t/016_heap_scan_stream.pl diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index 3bb33699349..67073fff6b7 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -360,8 +360,11 @@ static void initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) { ParallelBlockTableScanDesc bpscan = NULL; - bool allow_strat; + BlockNumber nblocks; + bool had_strat = scan->rs_strategy != NULL; + bool use_strat; bool allow_sync; + bool rebuild_stream; /* * Determine the number of blocks we have to scan. @@ -377,10 +380,10 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) if (scan->rs_base.rs_parallel != NULL) { bpscan = (ParallelBlockTableScanDesc) scan->rs_base.rs_parallel; - scan->rs_nblocks = bpscan->phs_nblocks; + nblocks = bpscan->phs_nblocks; } else - scan->rs_nblocks = RelationGetNumberOfBlocks(scan->rs_base.rs_rd); + nblocks = RelationGetNumberOfBlocks(scan->rs_base.rs_rd); /* * If the table is large relative to NBuffers, use a bulk-read access @@ -395,27 +398,47 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) * if you change this, consider changing that one, too. */ if (!RelationUsesLocalBuffers(scan->rs_base.rs_rd) && - scan->rs_nblocks > NBuffers / 4) + nblocks > NBuffers / 4) { - allow_strat = (scan->rs_base.rs_flags & SO_ALLOW_STRAT) != 0; + use_strat = (scan->rs_base.rs_flags & SO_ALLOW_STRAT) != 0; allow_sync = (scan->rs_base.rs_flags & SO_ALLOW_SYNC) != 0; } else - allow_strat = allow_sync = false; + use_strat = allow_sync = false; - if (allow_strat) + /* + * Reset the stream before reinitializing state used by its callback. A + * stream keeps the strategy passed at creation and sizes its buffer queue + * accordingly, so end it instead when strategy use changes. + */ + rebuild_stream = scan->rs_read_stream && had_strat != use_strat; + if (scan->rs_read_stream) + { + if (rebuild_stream) + { + read_stream_end(scan->rs_read_stream); + scan->rs_read_stream = NULL; + } + else + read_stream_reset(scan->rs_read_stream); + } + + if (use_strat) { /* During a rescan, keep the previous strategy object. */ - if (scan->rs_strategy == NULL) + if (!had_strat) scan->rs_strategy = GetAccessStrategy(BAS_BULKREAD); } else { - if (scan->rs_strategy != NULL) + if (had_strat) FreeAccessStrategy(scan->rs_strategy); scan->rs_strategy = NULL; } + /* Publish the new size only after the old stream has been reset. */ + scan->rs_nblocks = nblocks; + if (scan->rs_base.rs_parallel != NULL) { /* For parallel scan, believe whatever ParallelTableScanDesc says. */ @@ -491,6 +514,53 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock) */ if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN) pgstat_count_heap_scan(scan->rs_base.rs_rd); + + if (scan->rs_read_stream == NULL) + { + if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN || + scan->rs_base.rs_flags & SO_TYPE_TIDRANGESCAN) + { + ReadStreamBlockNumberCB cb; + + if (scan->rs_base.rs_parallel) + cb = heap_scan_stream_read_next_parallel; + else + cb = heap_scan_stream_read_next_serial; + + /* --- + * It is safe to use batchmode as the only locks taken by `cb` + * are never taken while waiting for IO: + * - SyncScanLock is used in the non-parallel case + * - in the parallel case, only spinlocks and atomics are used + * --- + */ + scan->rs_read_stream = + read_stream_begin_relation(READ_STREAM_SEQUENTIAL | + READ_STREAM_USE_BATCHING, + scan->rs_strategy, + scan->rs_base.rs_rd, + MAIN_FORKNUM, + cb, + scan, + 0); + } + else if (scan->rs_base.rs_flags & SO_TYPE_BITMAPSCAN) + { + scan->rs_read_stream = + read_stream_begin_relation(READ_STREAM_DEFAULT | + READ_STREAM_USE_BATCHING, + scan->rs_strategy, + scan->rs_base.rs_rd, + MAIN_FORKNUM, + bitmapheap_stream_read_next, + scan, + sizeof(TBMIterateResult)); + } + + if (scan->rs_read_stream && scan->rs_base.rs_instrument) + read_stream_enable_stats(scan->rs_read_stream, + &scan->rs_base.rs_instrument->io); + } } /* @@ -1204,7 +1274,10 @@ heap_beginscan(Relation relation, Snapshot snapshot, scan->rs_base.rs_nkeys = nkeys; scan->rs_base.rs_flags = flags; scan->rs_base.rs_parallel = parallel_scan; - scan->rs_base.rs_instrument = NULL; + if (flags & SO_SCAN_INSTRUMENT) + scan->rs_base.rs_instrument = palloc0_object(TableScanInstrumentation); + else + scan->rs_base.rs_instrument = NULL; scan->rs_strategy = NULL; /* set in initscan */ scan->rs_cbuf = InvalidBuffer; @@ -1270,60 +1343,8 @@ heap_beginscan(Relation relation, Snapshot snapshot, else scan->rs_base.rs_key = NULL; - initscan(scan, key, false); - scan->rs_read_stream = NULL; - - /* - * Set up a read stream for sequential scans and TID range scans. This - * should be done after initscan() because initscan() allocates the - * BufferAccessStrategy object passed to the read stream API. - */ - if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN || - scan->rs_base.rs_flags & SO_TYPE_TIDRANGESCAN) - { - ReadStreamBlockNumberCB cb; - - if (scan->rs_base.rs_parallel) - cb = heap_scan_stream_read_next_parallel; - else - cb = heap_scan_stream_read_next_serial; - - /* --- - * It is safe to use batchmode as the only locks taken by `cb` - * are never taken while waiting for IO: - * - SyncScanLock is used in the non-parallel case - * - in the parallel case, only spinlocks and atomics are used - * --- - */ - scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL | - READ_STREAM_USE_BATCHING, - scan->rs_strategy, - scan->rs_base.rs_rd, - MAIN_FORKNUM, - cb, - scan, - 0); - } - else if (scan->rs_base.rs_flags & SO_TYPE_BITMAPSCAN) - { - scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_DEFAULT | - READ_STREAM_USE_BATCHING, - scan->rs_strategy, - scan->rs_base.rs_rd, - MAIN_FORKNUM, - bitmapheap_stream_read_next, - scan, - sizeof(TBMIterateResult)); - } - - /* enable read stream instrumentation */ - if ((flags & SO_SCAN_INSTRUMENT) && (scan->rs_read_stream != NULL)) - { - scan->rs_base.rs_instrument = palloc0_object(TableScanInstrumentation); - read_stream_enable_stats(scan->rs_read_stream, - &scan->rs_base.rs_instrument->io); - } + initscan(scan, key, false); scan->rs_vmbuffer = InvalidBuffer; @@ -1375,14 +1396,6 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, * additional data vs a normal HeapScan */ - /* - * The read stream is reset on rescan. This must be done before - * initscan(), as some state referred to by read_stream_reset() is reset - * in initscan(). - */ - if (scan->rs_read_stream) - read_stream_reset(scan->rs_read_stream); - /* * reinitialize scan descriptor */ diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c index fddf6ef441d..f005a93aa5b 100644 --- a/src/backend/storage/aio/read_stream.c +++ b/src/backend/storage/aio/read_stream.c @@ -255,7 +255,9 @@ void read_stream_enable_stats(ReadStream *stream, IOStats *stats) { stream->stats = stats; - if (stream->stats) + /* The same accumulator may span recreated streams of different sizes. */ + if (stream->stats && + stream->stats->distance_capacity < stream->max_pinned_buffers) stream->stats->distance_capacity = stream->max_pinned_buffers; } diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build index 5d81f5b13be..5225aa280ca 100644 --- a/src/test/modules/test_misc/meson.build +++ b/src/test/modules/test_misc/meson.build @@ -24,6 +24,7 @@ tests += { 't/013_temp_obj_multisession.pl', 't/014_log_statement_max_length.pl', 't/015_temp_schema_exit_deferrable.pl', + 't/016_heap_scan_stream.pl', ], # The injection points are cluster-wide, so disable installcheck 'runningcheck': false, diff --git a/src/test/modules/test_misc/t/016_heap_scan_stream.pl b/src/test/modules/test_misc/t/016_heap_scan_stream.pl new file mode 100644 index 00000000000..a659b5e1eae --- /dev/null +++ b/src/test/modules/test_misc/t/016_heap_scan_stream.pl @@ -0,0 +1,54 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +# Set a low threshold for selecting a bulk-read strategy. +my $node = PostgreSQL::Test::Cluster->new('main'); +$node->init; +$node->append_conf( + 'postgresql.conf', qq{ +shared_buffers = '1MB' +}); +$node->start; + +$node->safe_psql( + 'postgres', q{ +CREATE TABLE growing AS SELECT 1 AS i; +}); + +# Start without a strategy, grow the relation, then rewind the same scan. The +# rescan must recreate its read stream with a bulk-read strategy. +my $bulkread = $node->safe_psql( + 'postgres', q{ +BEGIN; +DECLARE c SCROLL CURSOR FOR SELECT FROM growing; +MOVE FORWARD ALL IN c; + +INSERT INTO growing SELECT generate_series(2, 10000); + +MOVE ABSOLUTE 0 IN c; +MOVE FORWARD ALL IN c; +COMMIT; + +SELECT pg_stat_force_next_flush() +\gset + +SELECT EXISTS ( + SELECT + FROM pg_stat_get_backend_io(pg_backend_pid()) + WHERE object = 'relation' + AND context = 'bulkread' + AND reads + hits > 0 +); +}); + +is($bulkread, 't', + 'a rescanned sequential scan uses its newly selected bulk-read strategy'); + +$node->stop; +done_testing(); -- 2.43.0
