pgsql: Remove long-unused TransactionIdIsActive()

2025-07-12 Thread Andres Freund
Remove long-unused TransactionIdIsActive() TransactionIdIsActive() has not been used since bb38fb0d43c, in 2014. There are no known uses in extensions either and it's hard to see valid uses for it. Therefore remove TransactionIdIsActive(). Discussion: https://postgr.es/m/odgftbtwp5oq7cxjgf4kjkmy

pgsql: Add pg_assume(expr) macro

2025-07-09 Thread Andres Freund
Add pg_assume(expr) macro This macro can be used to avoid compiler warnings, particularly when using -O3 and not using assertions, and to get the compiler to generate better code. A subsequent commit introduces a first user. Reviewed-by: Tom Lane Discussion: https://postgr.es/m/3prdb6hkep3dugl

pgsql: Use pg_assume() to avoid compiler warning below exec_set_found()

2025-07-09 Thread Andres Freund
Use pg_assume() to avoid compiler warning below exec_set_found() The warning, visible when building with -O3 and a recent-ish gcc, is due to gcc not realizing that found is a byvalue type and therefore will never be interpreted as a varlena type. Discussion: https://postgr.es/m/3prdb6hkep3duglhs

pgsql: aio: Combine io_uring memory mappings, if supported

2025-07-07 Thread Andres Freund
aio: Combine io_uring memory mappings, if supported By default io_uring creates a shared memory mapping for each io_uring instance, leading to a large number of memory mappings. Unfortunately a large number of memory mappings slows things down, backend exit is particularly affected. To address th

pgsql: aio: Combine io_uring memory mappings, if supported

2025-07-07 Thread Andres Freund
aio: Combine io_uring memory mappings, if supported By default io_uring creates a shared memory mapping for each io_uring instance, leading to a large number of memory mappings. Unfortunately a large number of memory mappings slows things down, backend exit is particularly affected. To address th

pgsql: aio: Fix reference to outdated name

2025-06-30 Thread Andres Freund
aio: Fix reference to outdated name Reported-by: Antonin Houska Author: Antonin Houska Discussion: https://postgr.es/m/5250.1751266701@localhost Backpatch-through: 18, where da7226993fd4 introduced this Branch -- REL_18_STABLE Details --- https://git.postgresql.org/pg/commitdiff/95163c

pgsql: aio: Fix reference to outdated name

2025-06-30 Thread Andres Freund
aio: Fix reference to outdated name Reported-by: Antonin Houska Author: Antonin Houska Discussion: https://postgr.es/m/5250.1751266701@localhost Backpatch-through: 18, where da7226993fd4 introduced this Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/f20a347e1a613

pgsql: aio: Add missing memory barrier when waiting for IO handle

2025-06-16 Thread Andres Freund
a separate patch. Thanks a lot to Alexander and Konstantin for all the debugging help. Reported-by: Tom Lane Reported-by: Alexander Lakhin Investigated-by: Andres Freund Investigated-by: Alexander Lakhin Investigated-by: Konstantin Knizhnik Discussion: htt

pgsql: Remove accidentally added meson.build

2025-05-22 Thread Andres Freund
Remove accidentally added meson.build Commit b4363fc66e6 added an empty meson.build to 13, presumably due to a conflict resolution while backpatching. Remove it. Discussion: https://postgr.es/m/b7angm4wz2ln2wsqvsb3bwl4pqa4xxgcawn36o5kweu5pgpl6s@ofhhrgl5ouwb Branch -- REL_13_STABLE Details

pgsql: aio: Fix possible state confusions due to interrupt processing

2025-05-19 Thread Andres Freund
aio: Fix possible state confusions due to interrupt processing elog()/ereport() process interrupts, iff the log message is < ERROR and the log message will be emitted. aio's debug messages are emitted via ereport(), but in some places the code is not ready for interrupts to be processed. Fix the

Re: pgsql: doc: clarify new behavior of max_files_per_process

2025-04-29 Thread Andres Freund
Hi, On 2025-04-29 13:45:44 +, Bruce Momjian wrote: > doc: clarify new behavior of max_files_per_process I don't think this really is an improvement - if postmaster is called with a bunch of files already opened, they aren't opened by postmaster, but still are ignored... Greetings, Andres

pgsql: aio: Fix crash potential for pg_aios views due to late state upd

2025-04-25 Thread Andres Freund
aio: Fix crash potential for pg_aios views due to late state update pgaio_io_reclaim() reset the fields in PgAioHandle before updating the state to IDLE or incrementing the generation. For most things that's OK, but for pg_get_aios() it is not - if it copied the PgAioHandle while fields were being

pgsql: aio: Improve debug logging around waiting for IOs

2025-04-25 Thread Andres Freund
aio: Improve debug logging around waiting for IOs Trying to investigate a bug report by Alexander Lakhin made it apparent that the debug logging around waiting for IO completion is insufficient. Fix that. Discussion: https://postgr.es/m/h4in2db37vepagmi2oz5vvqymjasc5gyb4lpqkunj4eusu274i@37jpd3c2

pgsql: Fix bug allowing io_combine_limit > io_max_combine_combine limit

2025-04-25 Thread Andres Freund
Fix bug allowing io_combine_limit > io_max_combine_combine limit 10f66468475 intended to limit the value of io_combine_limit to the minimum of io_combine_limit and io_max_combine_limit. To avoid issues with interdependent GUCs, it introduced io_combine_limit_guc and set io_combine_limit in assign

Re: pgsql: Add support for basic NUMA awareness

2025-04-10 Thread Andres Freund
Hi, Walther, adding you because of the failure on buildfarm animal dogfish. The main issue is something unrelated to your animal (except it was the only one to catch it), but there's one oddity: On 2025-04-08 10:36:32 -0400, Tom Lane wrote: > Andres Freund writes: > > On April 8,

Re: pgsql: Add support for basic NUMA awareness

2025-04-08 Thread Andres Freund
Hi, On 2025-04-08 17:17:21 +0200, Wolfgang Walther wrote: > Andres Freund: > > But I'm rather bewildered - I can't actually build with the professed > > version > > of ninja, it errors out: > > > > ninja-1.9.0 src/bin/psql/sql_help.c > > ninja:

Re: pgsql: Add support for basic NUMA awareness

2025-04-08 Thread Andres Freund
Hi, On April 8, 2025 10:05:48 AM EDT, Tom Lane wrote: >Daniel Gustafsson writes: >>> On 7 Apr 2025, at 23:18, Tomas Vondra wrote: >>> Add support for basic NUMA awareness > >> dogfish is a little bit upset it seems. > >Hm, but why isn't anything else? I assume it's a question of degree of para

pgsql: Increase BAS_BULKREAD based on effective_io_concurrency

2025-04-07 Thread Andres Freund
Increase BAS_BULKREAD based on effective_io_concurrency Before, BAS_BULKREAD was always of size 256kB. With the default io_combine_limit of 16, that only allowed 1-2 IOs to be in flight - insufficient even on very low latency storage. We don't just want to increase the size to a much larger hardc

pgsql: Add pg_buffercache_evict_{relation,all} functions

2025-04-07 Thread Andres Freund
extension version is needed, as it was already increased this release by ba2a3c2302f. Author: Nazir Bilal Yavuz Reviewed-by: Andres Freund Reviewed-by: Aidar Imamov Reviewed-by: Joseph Koshakow Discussion: https://postgr.es/m/CAN55FZ0h_YoSqqutxV6DES1RW8ig6wcA8CR9rJk358YRMxZFmw

pgsql: aio: Make AIO more compatible with valgrind

2025-04-07 Thread Andres Freund
aio: Make AIO more compatible with valgrind In some edge cases valgrind flags issues with the memory referenced by IOs. All of the cases addressed in this change are false positives. Most of the false positives are caused by UnpinBuffer[NoOwner] marking buffer data as inaccessible. This happens e

pgsql: localbuf: Add Valgrind buffer access instrumentation

2025-04-07 Thread Andres Freund
localbuf: Add Valgrind buffer access instrumentation This mirrors 1e0dfd166b3 (+ 46ef520b9566), for temporary table buffers. This is mainly interesting right now because the AIO work currently triggers spurious valgrind errors, and the fix for that is cleaner if temp buffers behave the same as sha

Re: pgsql: Allow NOT NULL constraints to be added as NOT VALID

2025-04-07 Thread Andres Freund
6%3A49 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hippopotamus&dt=2025-04-07%2018%3A28%3A12 Greetings, Andres Freund

pgsql: read_stream: Fix overflow hazard with large shared buffers

2025-04-07 Thread Andres Freund
read_stream: Fix overflow hazard with large shared buffers If the limit returned by GetAdditionalPinLimit() is large, the buffer_limit variable in read_stream_start_pending_read() can overflow. While the code is careful to limit buffer_limit PG_INT16_MAX, we subsequently add the number of forwarde

pgsql: aio: Avoid spurious coverity warning

2025-04-06 Thread Andres Freund
aio: Avoid spurious coverity warning PgAioResult.result is never accessed in the relevant path, but coverity complains about an uninitialized access anyway. So just zero-initialize the whole thing. While at it, reduce the scope of the variable. Reported-by: Ranier Vilela Reviewed-by: Noah Misch

pgsql: docs: Reframe track_io_timing related docs as wait time

2025-04-05 Thread Andres Freund
docs: Reframe track_io_timing related docs as wait time With AIO it does not make sense anymore to track the time for each individual IO, as multiple IOs can be in-flight at the same time. Instead we now track the time spent *waiting* for IOs. This should be reflected in the docs. While, so far,

pgsql: meson: Flush stdout in testwrap

2025-04-05 Thread Andres Freund
meson: Flush stdout in testwrap Otherwise the progress won't reliably be displayed during a test. Reviewed-by: Noah Misch Discussion: https://postgr.es/m/kx6xu7suexal5vwsxpy7ybgkcznx6hgywbuhkr6qabcwxjqax2@i4pcpk75jvaa Backpatch-through: 16 Branch -- REL_17_STABLE Details --- https://g

Re: pgsql: aio: Add core asynchronous I/O infrastructure

2025-04-05 Thread Andres Freund
Hi, On 2025-03-18 11:03:36 -0400, Tom Lane wrote: > Andres Freund writes: > > aio: Add core asynchronous I/O infrastructure > > Some of the buildfarm is mildly unhappy with this. > So far I see > > ayu | 2025-03-18 13:08:04 | aio_callback.c:83:12: warning:

pgsql: Remove HeapBitmapScan's skip_fetch optimization

2025-04-05 Thread Andres Freund
Remove HeapBitmapScan's skip_fetch optimization The optimization does not take the removal of TIDs by a concurrent vacuum into account. The concurrent vacuum can remove dead TIDs and make pages ALL_VISIBLE while those dead TIDs are referenced in the bitmap. This can lead to a skip_fetch scan retur

pgsql: Add errhint_internal()

2025-04-04 Thread Andres Freund
Add errhint_internal() We have errmsg_internal(), errdetail_internal(), but not errhint_internal(). Sometimes it is useful to output a hint with already translated format string (e.g. because there different messages depending on the condition). For message/detail we do that with the _internal()

pgsql: bufmgr: Improve stats when a buffer is read in concurrently

2025-04-04 Thread Andres Freund
bufmgr: Improve stats when a buffer is read in concurrently Previously we would have the following inaccuracies when a backend tried to read in a buffer, but that buffer was read in concurrently by another backend: - the read IO was double-counted in the global buffer access stats (pgBufferUsage)

pgsql: Remove HeapBitmapScan's skip_fetch optimization

2025-04-02 Thread Andres Freund
Remove HeapBitmapScan's skip_fetch optimization The optimization does not take the removal of TIDs by a concurrent vacuum into account. The concurrent vacuum can remove dead TIDs and make pages ALL_VISIBLE while those dead TIDs are referenced in the bitmap. This can lead to a skip_fetch scan retur

pgsql: Remove HeapBitmapScan's skip_fetch optimization

2025-04-02 Thread Andres Freund
Remove HeapBitmapScan's skip_fetch optimization The optimization does not take the removal of TIDs by a concurrent vacuum into account. The concurrent vacuum can remove dead TIDs and make pages ALL_VISIBLE while those dead TIDs are referenced in the bitmap. This can lead to a skip_fetch scan retur

pgsql: Remove HeapBitmapScan's skip_fetch optimization

2025-04-02 Thread Andres Freund
Remove HeapBitmapScan's skip_fetch optimization The optimization does not take the removal of TIDs by a concurrent vacuum into account. The concurrent vacuum can remove dead TIDs and make pages ALL_VISIBLE while those dead TIDs are referenced in the bitmap. This can lead to a skip_fetch scan retur

pgsql: Add test for HeapBitmapScan's broken skip_fetch optimization

2025-04-02 Thread Andres Freund
Add test for HeapBitmapScan's broken skip_fetch optimization In the previous commit HeapBitmapScan's skip_fetch optimization was removed, due to being broken in not easily fixable ways. Add a test that verifies we don't re-introduce this bug if somebody tries to re-add the feature. Only add the t

pgsql: Remove HeapBitmapScan's skip_fetch optimization

2025-04-02 Thread Andres Freund
Remove HeapBitmapScan's skip_fetch optimization The optimization does not take the removal of TIDs by a concurrent vacuum into account. The concurrent vacuum can remove dead TIDs and make pages ALL_VISIBLE while those dead TIDs are referenced in the bitmap. This can lead to a skip_fetch scan retur

pgsql: Remove HeapBitmapScan's skip_fetch optimization

2025-04-02 Thread Andres Freund
Remove HeapBitmapScan's skip_fetch optimization The optimization does not take the removal of TIDs by a concurrent vacuum into account. The concurrent vacuum can remove dead TIDs and make pages ALL_VISIBLE while those dead TIDs are referenced in the bitmap. This can lead to a skip_fetch scan retur

pgsql: tests: Fix incompatibility of test_aio with *_FORCE_RELEASE

2025-04-02 Thread Andres Freund
tests: Fix incompatibility of test_aio with *_FORCE_RELEASE The test added in 93bc3d75d8e failed in a build with RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE defined. The test intentionally forgets to exit batchmode - normally that would trigger an error at the end of the transaction, which t

pgsql: tests: Cope with WARNINGs during failed CREATE DB on windows

2025-04-02 Thread Andres Freund
tests: Cope with WARNINGs during failed CREATE DB on windows The test added in 93bc3d75d8e sometimes fails on windows, due to warnings like WARNING: some useless files may be left behind in old database directory "base/16514" The reason for that is createdb_failure_callback() does not ensure th

pgsql: tests: Cope with io_method in TEMP_CONFIG in test_aio

2025-04-02 Thread Andres Freund
tests: Cope with io_method in TEMP_CONFIG in test_aio If io_method is set in TEMP_CONFIG the test added in 93bc3d75d8e fails, because it assumes the io_method specified at initdb is actually used. Fix that by appending the io_method again, after initdb (and thus after TEMP_CONFIG has been added b

pgsql: aio: Minor comment improvements

2025-04-01 Thread Andres Freund
aio: Minor comment improvements Reviewed-by: Noah Misch Discussion: https://postgr.es/m/usbwzckj7q3jhfx3ann3nrfnukmupbs35axvq5zfyeo6nvrzrm@onjhxs2du4st Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/e19dc74491e68e1df05c99c4937929cef3092a3b Modified Files ---

pgsql: aio: Add README.md explaining higher level design

2025-04-01 Thread Andres Freund
aio: Add README.md explaining higher level design Reviewed-by: Noah Misch Reviewed-by: Heikki Linnakangas Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt Discussion: https://postgr.es/m/20210223100344.llw5an2akleng...@alap3.anarazel.de Discussi

pgsql: aio: Add test_aio module

2025-04-01 Thread Andres Freund
aio: Add test_aio module To make the tests possible, a few functions from bufmgr.c/localbuf.c had to be exported, via buf_internals.h. Reviewed-by: Noah Misch Co-authored-by: Andres Freund Co-authored-by: Nazir Bilal Yavuz Discussion: https://postgr.es/m

pgsql: md: Add comment & assert to buffer-zeroing path in md[start]read

2025-04-01 Thread Andres Freund
md: Add comment & assert to buffer-zeroing path in md[start]readv() mdreadv() has a codepath to zero out buffers when a read returns zero bytes, guarded by a check for zero_damaged_pages || InRecovery. The InRecovery codepath to zero out buffers in mdreadv() appears to be unreachable. The only kn

pgsql: docs: Add acronym and glossary entries for I/O and AIO

2025-04-01 Thread Andres Freund
docs: Add acronym and glossary entries for I/O and AIO These are fairly basic, but better than nothing. While there are several opportunities to link to these entries, this patch does not add any. They will however be referenced by future patches. Reviewed-by: Noah Misch Discussion: https://pos

pgsql: aio: Add pg_aios view

2025-04-01 Thread Andres Freund
aio: Add pg_aios view The new view lists all IO handles that are currently in use and is mainly useful for PG developers, but may also be useful when tuning PG. Bumps catversion. Reviewed-by: Noah Misch Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62

Re: pgsql: aio: Add core asynchronous I/O infrastructure

2025-03-31 Thread Andres Freund
Hi, On 2025-03-31 19:23:57 +0300, Aleksander Alekseev wrote: > Hi Andres, > > > aio: Add core asynchronous I/O infrastructure > > > > [...] > > I noticed that aio.c references README.md: > > ``` > * aio.c > *AIO - Core Logic > * > * For documentation about how AIO works on a higher leve

pgsql: bufmgr: Use AIO in StartReadBuffers()

2025-03-30 Thread Andres Freund
Noah Misch Reviewed-by: Nazir Bilal Yavuz Co-authored-by: Andres Freund Co-authored-by: Thomas Munro Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt Discussion: https://postgr.es/m/20210223100344.llw5an2akleng...@alap3.anarazel.de Discussion:

pgsql: aio: Basic read_stream adjustments for real AIO

2025-03-30 Thread Andres Freund
tely. - It can be beneficial to defer starting new IOs until we can issue multiple IOs at once. That however requires non-trivial heuristics to decide when to do so. Reviewed-by: Noah Misch Co-authored-by: Andres Freund Co-authored-by: Thomas Munro Branch -- master Details ---

pgsql: Enable IO concurrency on all systems

2025-03-30 Thread Andres Freund
Enable IO concurrency on all systems Previously effective_io_concurrency and maintenance_io_concurrency could not be set above 0 on machines without fadvise support. AIO enables IO concurrency without such support, via io_method=worker. Currently only subsystems using the read stream API will tak

pgsql: read_stream: Introduce and use optional batchmode support

2025-03-30 Thread Andres Freund
read_stream: Introduce and use optional batchmode support Submitting IO in larger batches can be more efficient than doing so one-by-one, particularly for many small reads. It does, however, require the ReadStreamBlockNumberCB callback to abide by the restrictions of AIO batching (c.f. pgaio_enter

pgsql: bufmgr: Implement AIO read support

2025-03-30 Thread Andres Freund
bufmgr: Implement AIO read support This commit implements the infrastructure to perform asynchronous reads into the buffer pool. To do so, it: - Adds readv AIO callbacks for shared and local buffers It may be worth calling out that shared buffer completions may be run in a different backend

pgsql: Let caller of PageIsVerified() control ignore_checksum_failure

2025-03-30 Thread Andres Freund
Let caller of PageIsVerified() control ignore_checksum_failure For AIO the completion of a read into shared buffers (i.e. verifying the page including the checksum, updating the BufferDesc to reflect the IO) can happen in a different backend than the backend that started the IO. As ignore_checksum

pgsql: aio: Add WARNING result status

2025-03-30 Thread Andres Freund
aio: Add WARNING result status If an IO succeeds, but issues a warning, e.g. due to a page verification failure with zero_damaged_pages, we want to issue that warning in the context of the issuer of the IO, not the process that executes the completion (always the case for worker). It's already po

pgsql: pgstat: Allow checksum errors to be reported in critical section

2025-03-30 Thread Andres Freund
pgstat: Allow checksum errors to be reported in critical sections For AIO we execute completion callbacks in critical sections (to ensure that AIO can in the future be used for WAL, which in turn requires that we can call completion callbacks in critical sections, to get the resources for WAL io).

pgsql: Fix mis-attribution of checksum failure stats to the wrong datab

2025-03-29 Thread Andres Freund
Fix mis-attribution of checksum failure stats to the wrong database Checksum failure stats could be attributed to the wrong database in two cases: - when a read of a shared relation encountered a checksum error , it would be attributed to the current database, instead of the "database" represen

pgsql: localbuf: Track pincount in BufferDesc as well

2025-03-29 Thread Andres Freund
localbuf: Track pincount in BufferDesc as well For AIO on temporary table buffers the AIO subsystem needs to be able to ensure a pin on a buffer while AIO is going on, even if the IO issuing query errors out. Tracking the buffer in LocalRefCount does not work, as it would cause CheckForLocalBuffer

pgsql: aio, bufmgr: Comment fixes/improvements

2025-03-29 Thread Andres Freund
aio, bufmgr: Comment fixes/improvements Some of these comments have been wrong for a while (12f3867f5534), some I recently introduced (da7226993fd, 55b454d0e14). This includes an update to a comment in FlushBuffer(), which will be copied in a future commit. These changes seem big enough to be wor

pgsql: aio: Implement support for reads in smgr/md/fd

2025-03-29 Thread Andres Freund
aio: Implement support for reads in smgr/md/fd This implements the following: 1) An smgr AIO target, for AIO on smgr files. This should be usable not just for md.c but also other SMGR implementation if we ever get them. 2) readv support in fd.c, which requires a small bit of infrastructure wor

pgsql: Fix "‘static’ is not at beginning of declaration" warning

2025-03-29 Thread Andres Freund
Fix "‘static’ is not at beginning of declaration" warning b98be8a2a2a used "const static" instead of "static const". We normally use the latter form. Discussion: https://postgr.es/m/z4mc2hzecahyq3paupfsouhuupmzmgum45md3k5my6bmo7gvn7@z5j26doqamqy Branch -- master Details --- https://git

Re: pgsql: aio: Add io_method=io_uring

2025-03-26 Thread Andres Freund
Hi, On 2025-03-27 00:15:53 +, Andres Freund wrote: > aio: Add io_method=io_uring FWIW, the BF failure on culicidae looks to be independent to me https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=culicidae&dt=2025-03-27%2000%3A16%3A30 The relevant line from the log is: 2025-03-2

pgsql: aio: Add io_method=io_uring

2025-03-26 Thread Andres Freund
aio: Add io_method=io_uring Performing AIO using io_uring can be considerably faster than io_method=worker, particularly when lots of small IOs are issued, as a) the context-switch overhead for worker based AIO becomes more significant b) the number of IO workers can become limiting io_uring, how

pgsql: aio: Add liburing dependency

2025-03-26 Thread Andres Freund
aio: Add liburing dependency Will be used in a subsequent commit, to implement io_method=io_uring. Kept separate for easier review. Reviewed-by: Noah Misch Discussion: https://postgr.es/m/uvrtrknj4kdytuboidbhwclo4gxhswwcpgadptsjvjqcluzmah%40brqs62irg4dt Branch -- master Details --- ht

pgsql: aio: Rename pgaio_io_prep_* to pgaio_io_start_*

2025-03-26 Thread Andres Freund
aio: Rename pgaio_io_prep_* to pgaio_io_start_* The old naming pattern (mirroring liburing's naming) was inconsistent with the (not yet introduced) callers. It seems better to get rid of the inconsistency now than to grow more users of the odd naming. Reported-by: Noah Misch Discussion: https://

pgsql: aio: Be more paranoid about interrupts

2025-03-26 Thread Andres Freund
aio: Be more paranoid about interrupts As reported by Noah, it's possible, although practically very unlikely, that interrupts could be processed in between pgaio_io_reopen() and pgaio_io_perform_synchronously(). Prevent that by explicitly holding interrupts. It also seems good to add an assertio

pgsql: aio: Pass result of local callbacks to ->report_return

2025-03-26 Thread Andres Freund
aio: Pass result of local callbacks to ->report_return Otherwise the results of e.g. temp table buffer verification errors will not reach bufmgr.c. Obviously that's not right. Found while expanding the tests for invalid buffer contents. Reviewed-by: Noah Misch Discussion: https://postgr.es/m/202

pgsql: Redefine max_files_per_process to control additionally opened fi

2025-03-24 Thread Andres Freund
Redefine max_files_per_process to control additionally opened files Until now max_files_per_process=N limited each backend to open N files in total (minus a safety factor), even if there were already more files opened in postmaster and inherited by backends. Change max_files_per_process to contro

pgsql: aio: Change prefix of PgAioResultStatus values to PGAIO_RS_

2025-03-22 Thread Andres Freund
aio: Change prefix of PgAioResultStatus values to PGAIO_RS_ The previous prefix wasn't consistent with the naming of other AIO related enum values. It seems best to rename it before the users are introduced. Reported-by: Melanie Plageman Discussion: https://postgr.es/m/caakru_yb+jzqpnsguxcb0gbi

pgsql: smgr: Hold interrupts in most smgr functions

2025-03-20 Thread Andres Freund
smgr: Hold interrupts in most smgr functions We need to hold interrupts across most of the smgr.c/md.c functions, as otherwise interrupt processing, e.g. due to a < ERROR elog/ereport, can trigger procsignal processing, which in turn can trigger smgrreleaseall(). As the relevant code is not reentr

Re: pgsql: aio: Add core asynchronous I/O infrastructure

2025-03-19 Thread Andres Freund
Hi, On 2025-03-18 11:26:15 -0400, Tom Lane wrote: > Andres Freund writes: > > I wonder if we should instead either ask those buildfarm animals to be > > disabled or have the warning manually disabled. I don't think it's a good > > investment on our part to work

pgsql: meson: Flush stdout in testwrap

2025-03-19 Thread Andres Freund
meson: Flush stdout in testwrap Otherwise the progress won't reliably be displayed during a test. Reviewed-by: Noah Misch Discussion: https://postgr.es/m/kx6xu7suexal5vwsxpy7ybgkcznx6hgywbuhkr6qabcwxjqax2@i4pcpk75jvaa Backpatch-through: 16 Branch -- master Details --- https://git.post

pgsql: meson: Flush stdout in testwrap

2025-03-19 Thread Andres Freund
meson: Flush stdout in testwrap Otherwise the progress won't reliably be displayed during a test. Reviewed-by: Noah Misch Discussion: https://postgr.es/m/kx6xu7suexal5vwsxpy7ybgkcznx6hgywbuhkr6qabcwxjqax2@i4pcpk75jvaa Backpatch-through: 16 Branch -- REL_16_STABLE Details --- https://g

Re: pgsql: Increase default maintenance_io_concurrency to 16

2025-03-18 Thread Andres Freund
Greetings, Andres Freund

pgsql: smgr: Make SMgrRelation initialization safer against errors

2025-03-18 Thread Andres Freund
smgr: Make SMgrRelation initialization safer against errors In case the smgr_open callback failed, the ->pincount field would not be initialized and the relation would not be put onto the unpinned_relns list. This buglet was introduced in 21d9c3ee4ef7, in 17. Discussion: https://postgr.es/m/3va

pgsql: smgr: Make SMgrRelation initialization safer against errors

2025-03-18 Thread Andres Freund
smgr: Make SMgrRelation initialization safer against errors In case the smgr_open callback failed, the ->pincount field would not be initialized and the relation would not be put onto the unpinned_relns list. This buglet was introduced in 21d9c3ee4ef7, in 17. Discussion: https://postgr.es/m/3va

pgsql: aio: Infrastructure for io_method=worker

2025-03-18 Thread Andres Freund
down checkpoint has been performed and walsender/archiver have shut down, but before the checkpointer itself shuts down. See also 87a6690cc69. Updates PGSTAT_FILE_FORMAT_ID due to the addition of a new BackendType. Reviewed-by: Noah Misch Co-authored-by: Thomas Munro Co-authored-by: Andres Fre

pgsql: aio: Add io_method=worker

2025-03-18 Thread Andres Freund
when submitted. For now the default io_method is changed to "worker". We should re-evaluate that around beta1, we might want to be careful and set the default to "sync" for 18. Reviewed-by: Noah Misch Co-authored-by: Thomas Munro Co-authored-by: Andres Freund Discussion:

pgsql: aio: Add core asynchronous I/O infrastructure

2025-03-17 Thread Andres Freund
rototype, the current version of the AIO infrastructure would not exist. Similarly, the reviewers below have not necessarily looked at the current design or the whole infrastructure, but have provided very valuable input. I am to blame for problems, not they. Author: Andres Freund Co-authored-by: Thomas

pgsql: aio: Basic subsystem initialization

2025-03-17 Thread Andres Freund
aio: Basic subsystem initialization This commit just does the minimal wiring up of the AIO subsystem, added in the next commit, to the rest of the system. The next commit contains more details about motivation and architecture. This commit is kept separate to make it easier to review, separating

pgsql: tests: Expand temp table tests to some pin related matters

2025-03-17 Thread Andres Freund
tests: Expand temp table tests to some pin related matters Added tests: - recovery from running out of unpinned local buffers - that we don't run out of unpinned buffers due to read stream (only recently fixed, in 92fc6856cb4) - temp tables can't be dropped while in use by cursors Discussion: w

pgsql: localbuf: Introduce TerminateLocalBufferIO()

2025-03-15 Thread Andres Freund
localbuf: Introduce TerminateLocalBufferIO() Previously TerminateLocalBufferIO() was open-coded in multiple places, which doesn't seem like a great idea. While TerminateLocalBufferIO() currently is rather simple, an upcoming patch requires additional code to be added to TerminateLocalBufferIO(), m

pgsql: localbuf: Introduce InvalidateLocalBuffer()

2025-03-15 Thread Andres Freund
localbuf: Introduce InvalidateLocalBuffer() Previously, there were three copies of this code, two of them identical. There's no good reason for that. This change is nice on its own, but the main motivation is the AIO patchset, which needs to add extra checks the deduplicated code, which of course

pgsql: localbuf: Fix dangerous coding pattern in GetLocalVictimBuffer()

2025-03-15 Thread Andres Freund
localbuf: Fix dangerous coding pattern in GetLocalVictimBuffer() If PinLocalBuffer() were to modify the buf_state, the buf_state in GetLocalVictimBuffer() would be out of date. Currently that does not happen, as PinLocalBuffer() only modifies the buf_state if adjust_usagecount=true and GetLocalVic

pgsql: localbuf: Introduce StartLocalBufferIO()

2025-03-15 Thread Andres Freund
localbuf: Introduce StartLocalBufferIO() To initiate IO on a shared buffer we have StartBufferIO(). For temporary table buffers no similar function exists - likely because the code for that currently is very simple due to the lack of concurrency. However, the upcoming AIO support will make it pos

pgsql: localbuf: Introduce FlushLocalBuffer()

2025-03-15 Thread Andres Freund
localbuf: Introduce FlushLocalBuffer() Previously we had two paths implementing writing out temporary table buffers. For shared buffers, the logic for that is centralized in FlushBuffer(). Introduce FlushLocalBuffer() to do the same for local buffers. Besides being a nice cleanup on its own, it a

Re: pgsql: tests: Fix race condition in postmaster/002_connection_limits

2025-03-10 Thread Andres Freund
Hi, On 2025-03-10 18:46:12 -0400, Andrew Dunstan wrote: > On 2025-03-07 Fr 1:22 PM, Andres Freund wrote: > > tests: Fix race condition in postmaster/002_connection_limits > > > > The test occasionally failed due to unexpected connection limit errors being > > encounte

pgsql: tests: Make postmaster/002_connection_limits deal verbose logs

2025-03-10 Thread Andres Freund
tests: Make postmaster/002_connection_limits deal verbose logs When log_error_verbosity=verbose is configured the test would hand (and then fail), because of the sqlstate being added between log level and message. Make regex cope. Reported-by: Andrew Dunstan Discussion: https://postgr.es/m/c7ba

pgsql: tests: Don't fail due to high default timeout in postmaster/003_

2025-03-07 Thread Andres Freund
tests: Don't fail due to high default timeout in postmaster/003_start_stop Some BF animals use very high timeouts due to their slowness. Unfortunately postmaster/003_start_stop fails if a high timeout is configured, due to authentication_timeout having a fairly low max. As this test is reasonably

pgsql: tests: Fix race condition in postmaster/002_connection_limits

2025-03-07 Thread Andres Freund
tests: Fix race condition in postmaster/002_connection_limits The test occasionally failed due to unexpected connection limit errors being encountered after having waited for FATAL errors on another connection. These spurious failures were caused by the the backend reporting FATAL errors to the cl

pgsql: valgrind: Adjust suppressions to handle glibc changes

2025-03-06 Thread Andres Freund
valgrind: Adjust suppressions to handle glibc changes In newer glibc versions two additional functions appear between send() and socketcall.send(msg): fun:__internal_syscall_cancel fun:__syscall_cancel Due to that our existing suppression do not work anymore. Use '...', like aleady used in

pgsql: valgrind: Adjust suppressions to handle glibc changes

2025-03-06 Thread Andres Freund
valgrind: Adjust suppressions to handle glibc changes In newer glibc versions two additional functions appear between send() and socketcall.send(msg): fun:__internal_syscall_cancel fun:__syscall_cancel Due to that our existing suppression do not work anymore. Use '...', like aleady used in

pgsql: ci: freebsd: Specify debug_parallel_query=regress

2025-03-05 Thread Andres Freund
ci: freebsd: Specify debug_parallel_query=regress A lot of buildfarm animals run with debug_parallel_query=regress, while CI didn't test that. That lead to the annoying situation of only noticing related test instabilities after merging changes upstream. FreeBSD was chosen because it's a relative

pgsql: ci: Document what makes certain tasks special

2025-03-05 Thread Andres Freund
ci: Document what makes certain tasks special To increase coverage without drastically increasing CI resource usage, we have different CI tasks test different things (e.g. the linux tasks use sanitizers). Unfortunately that can create confusing situations where CI fails on some OS, but not others

pgsql: ci: Upgrade FreeBSD image

2025-03-05 Thread Andres Freund
ci: Upgrade FreeBSD image Upgrade to the current stable version. To avoid needing commits like this in the future, the CI image name now doesn't contain the OS version number anymore. Backpatch to all versions with CI support, we don't want to generate CI images for multiple FreeBSD versions. Au

pgsql: ci: Upgrade FreeBSD image

2025-03-05 Thread Andres Freund
ci: Upgrade FreeBSD image Upgrade to the current stable version. To avoid needing commits like this in the future, the CI image name now doesn't contain the OS version number anymore. Backpatch to all versions with CI support, we don't want to generate CI images for multiple FreeBSD versions. Au

pgsql: ci: Upgrade FreeBSD image

2025-03-05 Thread Andres Freund
ci: Upgrade FreeBSD image Upgrade to the current stable version. To avoid needing commits like this in the future, the CI image name now doesn't contain the OS version number anymore. Backpatch to all versions with CI support, we don't want to generate CI images for multiple FreeBSD versions. Au

pgsql: ci: Upgrade FreeBSD image

2025-03-05 Thread Andres Freund
ci: Upgrade FreeBSD image Upgrade to the current stable version. To avoid needing commits like this in the future, the CI image name now doesn't contain the OS version number anymore. Backpatch to all versions with CI support, we don't want to generate CI images for multiple FreeBSD versions. Au

Re: pgsql: Set amcancrosscompare to true for hash

2025-03-03 Thread Andres Freund
, independent of that concrete breakage it looks like skink is running far fewer things under valgrind than it used to. Looks like it's related to a buildfarm client upgrade... The next run after this one will probably take a good bit longer. Greetings, Andres Freund

Re: pgsql: Add relallfrozen to pg_class

2025-03-03 Thread Andres Freund
er rapidly. That's not the case here. Having reliable information after a crash (where we loose pgstats) that 95% of the table is actually frozen, rather than not seems good. Greetings, Andres Freund

pgsql: Change relpath() et al to return path by value

2025-02-25 Thread Andres Freund
Change relpath() et al to return path by value For AIO, and also some other recent patches, we need the ability to call relpath() in a critical section. Until now that was not feasible, as it allocated memory. The fact that relpath() allocated memory also made it awkward to use in log messages be

pgsql: Change _mdfd_segpath() to return paths by value

2025-02-25 Thread Andres Freund
Change _mdfd_segpath() to return paths by value This basically mirrors the changes done in the predecessor commit. While there isn't currently a need to get these paths in critical sections, it seems a shame to unnecessarily allocate memory in these paths now that relpath() doesn't allocate anymor

  1   2   3   4   5   6   7   8   9   10   >