pgsql: pgstat: fix subscription stats entry leak.
pgstat: fix subscription stats entry leak. Commit 7b64e4b3 taught DropSubscription() to drop stats entry of subscription that is not associated with a replication slot for apply worker at DROP SUBSCRIPTION but missed covering the case where the subscription is not associated with replication slots for both apply worker and tablesync worker. Also add a test to verify that the stats for slot-less subscription is removed at DROP SUBSCRIPTION time. Backpatch down to 15. Author: Masahiko Sawada Reviewed-by: Nathan Bossart, Hayato Kuroda, Melih Mutlu, Amit Kapila Discussion: https://postgr.es/m/cad21aob71zkp7upt7jdpszcvp0749exeqnojxenkpdfisha...@mail.gmail.com Backpatch-through: 15 Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/68a59f9e99831cf05a746928a955781503232bbb Modified Files -- src/backend/commands/subscriptioncmds.c | 12 ++-- src/test/subscription/t/026_stats.pl| 20 2 files changed, 26 insertions(+), 6 deletions(-)
pgsql: pgstat: fix subscription stats entry leak.
pgstat: fix subscription stats entry leak. Commit 7b64e4b3 taught DropSubscription() to drop stats entry of subscription that is not associated with a replication slot for apply worker at DROP SUBSCRIPTION but missed covering the case where the subscription is not associated with replication slots for both apply worker and tablesync worker. Also add a test to verify that the stats for slot-less subscription is removed at DROP SUBSCRIPTION time. Backpatch down to 15. Author: Masahiko Sawada Reviewed-by: Nathan Bossart, Hayato Kuroda, Melih Mutlu, Amit Kapila Discussion: https://postgr.es/m/cad21aob71zkp7upt7jdpszcvp0749exeqnojxenkpdfisha...@mail.gmail.com Backpatch-through: 15 Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/be8cae7e2f03f35a049e2b1f1113860fea6d0212 Modified Files -- src/backend/commands/subscriptioncmds.c | 12 ++-- src/test/subscription/t/026_stats.pl| 20 2 files changed, 26 insertions(+), 6 deletions(-)
pgsql: pgstat: fix subscription stats entry leak.
pgstat: fix subscription stats entry leak. Commit 7b64e4b3 taught DropSubscription() to drop stats entry of subscription that is not associated with a replication slot for apply worker at DROP SUBSCRIPTION but missed covering the case where the subscription is not associated with replication slots for both apply worker and tablesync worker. Also add a test to verify that the stats for slot-less subscription is removed at DROP SUBSCRIPTION time. Backpatch down to 15. Author: Masahiko Sawada Reviewed-by: Nathan Bossart, Hayato Kuroda, Melih Mutlu, Amit Kapila Discussion: https://postgr.es/m/cad21aob71zkp7upt7jdpszcvp0749exeqnojxenkpdfisha...@mail.gmail.com Backpatch-through: 15 Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/66f8a139731957a7d516ade70cd3aa43d12d3c4f Modified Files -- src/backend/commands/subscriptioncmds.c | 12 ++-- src/test/subscription/t/026_stats.pl| 20 2 files changed, 26 insertions(+), 6 deletions(-)
pgsql: Clean up command argument assembly
Clean up command argument assembly Several commands internally assemble command lines to call other commands. This includes initdb, pg_dumpall, and pg_regress. (Also pg_ctl, but that is different enough that I didn't consider it here.) This has all evolved a bit organically, with fixed-size buffers, and various optional command-line arguments being injected with confusing-looking code, and the spacing between options handled in inconsistent ways. Clean all this up a bit to look clearer and be more easily extensible with new arguments and options. We start each command with printfPQExpBuffer(), and then append arguments as necessary with appendPQExpBuffer(). Also standardize on using initPQExpBuffer() over createPQExpBuffer() where possible. pg_regress uses StringInfo instead of PQExpBuffer, but many of the same ideas apply. Reviewed-by: Heikki Linnakangas Discussion: https://www.postgresql.org/message-id/flat/16d0beac-a141-e5d3-60e9-323da75f4...@eisentraut.org Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/cccdbc5d95aca276789bbe28783f3e06f9a3ba9c Modified Files -- src/bin/initdb/initdb.c | 56 --- src/bin/pg_dump/pg_dumpall.c | 27 +++-- src/test/regress/pg_regress.c | 26 3 files changed, 62 insertions(+), 47 deletions(-)
pgsql: Generate automatically code and documentation related to wait ev
Generate automatically code and documentation related to wait events The documentation and the code is generated automatically from a new file called wait_event_names.txt, formatted in sections dedicated to each wait event class (Timeout, Lock, IO, etc.) with three tab-separated fields: - C symbol in enums - Format in the system views - Description in the docs Using this approach has several advantages, as we have proved to be rather bad in maintaining this area of the tree across the years: - The order of each item in the documentation and the code, which should be alphabetical, has become incorrect multiple times, and the script generating the code and documentation has a few rules to enforce that, making the maintenance a no-brainer. - Some wait events were added to the code, but not documented, so this cannot be missed now. - The order of the tables for each wait event class is enforced in the documentation (the input .txt file does so as well for clarity, though this is not mandatory). - Less code, shaving 1.2k lines from the tree, with 1/3 of the savings coming from the code, the rest from the documentation. The wait event types "Lock" and "LWLock" still have their own code path for their code, hence only the documentation is created for them. These classes are listed with a special marker called WAIT_EVENT_DOCONLY in the input file. Adding a new wait event now requires only an update of wait_event_names.txt, with "Lock" and "LWLock" treated as exceptions. This commit has been tested with configure/Makefile, the CI and VPATH build. clean, distclean and maintainer-clean were working fine. Author: Bertrand Drouvot, Michael Paquier Discussion: https://postgr.es/m/77a86b3a-c4a8-5f5d-69b9-d70bbf2e9...@gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/fa88928470b538c0ec0289e4d69ee12356c5a8ce Modified Files -- doc/src/sgml/.gitignore|1 + doc/src/sgml/Makefile |4 +- doc/src/sgml/filelist.sgml |1 + doc/src/sgml/meson.build | 12 + doc/src/sgml/monitoring.sgml | 1271 +--- src/backend/Makefile | 13 +- src/backend/storage/lmgr/lwlocknames.txt |4 +- src/backend/utils/activity/.gitignore |2 + src/backend/utils/activity/Makefile| 12 + .../utils/activity/generate-wait_event_types.pl| 263 src/backend/utils/activity/meson.build | 14 + src/backend/utils/activity/wait_event.c| 611 +- src/backend/utils/activity/wait_event_names.txt| 371 ++ src/backend/utils/adt/lockfuncs.c |3 +- src/common/meson.build |7 +- src/include/Makefile |2 +- src/include/utils/.gitignore |1 + src/include/utils/meson.build | 19 + src/include/utils/wait_event.h | 232 +--- src/test/ssl/t/002_scram.pl|3 +- src/tools/msvc/Solution.pm | 19 + src/tools/msvc/clean.bat |3 + 22 files changed, 757 insertions(+), 2111 deletions(-)
pgsql: Fix assertion failure in snapshot building
Fix assertion failure in snapshot building
Clear any potential stale next_phase_at value from the snapshot
builder which otherwise may trip an assertion check ensuring
that there is no next_phase_at value.
This can be reproduced by running 80 concurrent sessions like
the below where $c is a loop counter (assumes there has been
1..$c databases created) :
echo "
CREATE TABLE replication_example(id SERIAL PRIMARY KEY,
somedata int,
text varchar(120));
SELECT 'init' FROM
pg_create_logical_replication_slot('regression_slot_$c',
'test_decoding');
SELECT data FROM
pg_logical_slot_get_changes('regression_slot_$c', NULL,
NULL, 'include-xids', '0',
'skip-empty-xacts', '1');
" | psql -d regress_$c >>psql.log &
Backpatch down to v16.
Bug: #17695
Author: Masahiko Sawada
Reviewed-by: Alexander Lakhin
Reported-by: bowenshi
Discussion: https://postgr.es/m/17695-6be9277c92959...@postgresql.org
Backpatch-through: v16
Branch
--
master
Details
---
https://git.postgresql.org/pg/commitdiff/48efb2302bea3b2af8b4bb865edcc70132ee4250
Modified Files
--
src/backend/replication/logical/snapbuild.c | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
pgsql: Fix assertion failure in snapshot building
Fix assertion failure in snapshot building
Clear any potential stale next_phase_at value from the snapshot
builder which otherwise may trip an assertion check ensuring
that there is no next_phase_at value.
This can be reproduced by running 80 concurrent sessions like
the below where $c is a loop counter (assumes there has been
1..$c databases created) :
echo "
CREATE TABLE replication_example(id SERIAL PRIMARY KEY,
somedata int,
text varchar(120));
SELECT 'init' FROM
pg_create_logical_replication_slot('regression_slot_$c',
'test_decoding');
SELECT data FROM
pg_logical_slot_get_changes('regression_slot_$c', NULL,
NULL, 'include-xids', '0',
'skip-empty-xacts', '1');
" | psql -d regress_$c >>psql.log &
Backpatch down to v16.
Bug: #17695
Author: Masahiko Sawada
Reviewed-by: Alexander Lakhin
Reported-by: bowenshi
Discussion: https://postgr.es/m/17695-6be9277c92959...@postgresql.org
Backpatch-through: v16
Branch
--
REL_16_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/eb27d3dc88879cb148ae0365394aae0ab84fa1a7
Modified Files
--
src/backend/replication/logical/snapbuild.c | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
pgsql: Ensure that creation of an empty relfile is fsync'd at checkpoin
Ensure that creation of an empty relfile is fsync'd at checkpoint. If you create a table and don't insert any data into it, the relation file is never fsync'd. You don't lose data, because an empty table doesn't have any data to begin with, but if you crash and lose the file, subsequent operations on the table will fail with "could not open file" error. To fix, register an fsync request in mdcreate(), like we do for mdwrite(). Per discussion, we probably should also fsync the containing directory after creating a new file. But that's a separate and much wider issue. Backpatch to all supported versions. Reviewed-by: Andres Freund, Thomas Munro Discussion: https://www.postgresql.org/message-id/d47d8122-415e-425c-d0a2-e0160829702d%40iki.fi Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/e24c02e4d258f8cc0305f5ec6708bceb57403276 Modified Files -- src/backend/storage/smgr/md.c | 3 +++ 1 file changed, 3 insertions(+)
pgsql: Ensure that creation of an empty relfile is fsync'd at checkpoin
Ensure that creation of an empty relfile is fsync'd at checkpoint. If you create a table and don't insert any data into it, the relation file is never fsync'd. You don't lose data, because an empty table doesn't have any data to begin with, but if you crash and lose the file, subsequent operations on the table will fail with "could not open file" error. To fix, register an fsync request in mdcreate(), like we do for mdwrite(). Per discussion, we probably should also fsync the containing directory after creating a new file. But that's a separate and much wider issue. Backpatch to all supported versions. Reviewed-by: Andres Freund, Thomas Munro Discussion: https://www.postgresql.org/message-id/d47d8122-415e-425c-d0a2-e0160829702d%40iki.fi Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/d85bf0719e71f483cdeb537cd0045913f587f743 Modified Files -- src/backend/storage/smgr/md.c | 3 +++ 1 file changed, 3 insertions(+)
pgsql: Ensure that creation of an empty relfile is fsync'd at checkpoin
Ensure that creation of an empty relfile is fsync'd at checkpoint. If you create a table and don't insert any data into it, the relation file is never fsync'd. You don't lose data, because an empty table doesn't have any data to begin with, but if you crash and lose the file, subsequent operations on the table will fail with "could not open file" error. To fix, register an fsync request in mdcreate(), like we do for mdwrite(). Per discussion, we probably should also fsync the containing directory after creating a new file. But that's a separate and much wider issue. Backpatch to all supported versions. Reviewed-by: Andres Freund, Thomas Munro Discussion: https://www.postgresql.org/message-id/d47d8122-415e-425c-d0a2-e0160829702d%40iki.fi Branch -- REL_12_STABLE Details --- https://git.postgresql.org/pg/commitdiff/1b4f1c6f8a6c0b764636f25e17d37d8cab7e82be Modified Files -- src/backend/storage/smgr/md.c | 3 +++ 1 file changed, 3 insertions(+)
pgsql: Ensure that creation of an empty relfile is fsync'd at checkpoin
Ensure that creation of an empty relfile is fsync'd at checkpoint. If you create a table and don't insert any data into it, the relation file is never fsync'd. You don't lose data, because an empty table doesn't have any data to begin with, but if you crash and lose the file, subsequent operations on the table will fail with "could not open file" error. To fix, register an fsync request in mdcreate(), like we do for mdwrite(). Per discussion, we probably should also fsync the containing directory after creating a new file. But that's a separate and much wider issue. Backpatch to all supported versions. Reviewed-by: Andres Freund, Thomas Munro Discussion: https://www.postgresql.org/message-id/d47d8122-415e-425c-d0a2-e0160829702d%40iki.fi Branch -- REL_11_STABLE Details --- https://git.postgresql.org/pg/commitdiff/6377f705cab2f8f3e4017cd9852887906c8ac053 Modified Files -- src/backend/storage/smgr/md.c | 3 +++ 1 file changed, 3 insertions(+)
pgsql: Ensure that creation of an empty relfile is fsync'd at checkpoin
Ensure that creation of an empty relfile is fsync'd at checkpoint. If you create a table and don't insert any data into it, the relation file is never fsync'd. You don't lose data, because an empty table doesn't have any data to begin with, but if you crash and lose the file, subsequent operations on the table will fail with "could not open file" error. To fix, register an fsync request in mdcreate(), like we do for mdwrite(). Per discussion, we probably should also fsync the containing directory after creating a new file. But that's a separate and much wider issue. Backpatch to all supported versions. Reviewed-by: Andres Freund, Thomas Munro Discussion: https://www.postgresql.org/message-id/d47d8122-415e-425c-d0a2-e0160829702d%40iki.fi Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/4b4798e1384ce12e19a558708e66c829276cf8d7 Modified Files -- src/backend/storage/smgr/md.c | 3 +++ 1 file changed, 3 insertions(+)
pgsql: Ensure that creation of an empty relfile is fsync'd at checkpoin
Ensure that creation of an empty relfile is fsync'd at checkpoint. If you create a table and don't insert any data into it, the relation file is never fsync'd. You don't lose data, because an empty table doesn't have any data to begin with, but if you crash and lose the file, subsequent operations on the table will fail with "could not open file" error. To fix, register an fsync request in mdcreate(), like we do for mdwrite(). Per discussion, we probably should also fsync the containing directory after creating a new file. But that's a separate and much wider issue. Backpatch to all supported versions. Reviewed-by: Andres Freund, Thomas Munro Discussion: https://www.postgresql.org/message-id/d47d8122-415e-425c-d0a2-e0160829702d%40iki.fi Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/acc8cdff424529ce34e93a7ae51eb410b02c7b30 Modified Files -- src/backend/storage/smgr/md.c | 3 +++ 1 file changed, 3 insertions(+)
pgsql: Ensure that creation of an empty relfile is fsync'd at checkpoin
Ensure that creation of an empty relfile is fsync'd at checkpoint. If you create a table and don't insert any data into it, the relation file is never fsync'd. You don't lose data, because an empty table doesn't have any data to begin with, but if you crash and lose the file, subsequent operations on the table will fail with "could not open file" error. To fix, register an fsync request in mdcreate(), like we do for mdwrite(). Per discussion, we probably should also fsync the containing directory after creating a new file. But that's a separate and much wider issue. Backpatch to all supported versions. Reviewed-by: Andres Freund, Thomas Munro Discussion: https://www.postgresql.org/message-id/d47d8122-415e-425c-d0a2-e0160829702d%40iki.fi Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/d431644b43fcce9b7c2f677db18452b0ee216b42 Modified Files -- src/backend/storage/smgr/md.c | 3 +++ 1 file changed, 3 insertions(+)
pgsql: Remove incidental md5() function uses from several tests
Remove incidental md5() function uses from several tests This removes md5() function calls from these test suites: - bloom - test_decoding - isolation - recovery - subscription This covers all remaining test suites where md5() calls were just used to generate some random data and can be replaced by appropriately adapted sha256() calls. This will eventually allow these tests to pass in OpenSSL FIPS mode (which does not allow MD5 use). See also 208bf364a9. Unlike for the main regression tests, I didn't write a fipshash() wrapper here, because that would have been too repetitive and wouldn't really save much here. In some cases it was easier to remove one layer of indirection by changing column types from text to bytea. Reviewed-by: Daniel Gustafsson Discussion: https://www.postgresql.org/message-id/flat/f9b480b5-e473-d2d1-223a-4b9db30a2...@eisentraut.org Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/657f5f223e8fd9aa12d8d9fc473faecc267b9b7a Modified Files -- contrib/bloom/expected/bloom.out | 32 ++-- contrib/bloom/sql/bloom.sql| 8 ++--- contrib/bloom/t/001_wal.pl | 4 +-- .../test_decoding/expected/concurrent_stream.out | 6 ++-- contrib/test_decoding/specs/concurrent_stream.spec | 8 ++--- .../specs/insert-conflict-specconflict.spec| 2 +- src/test/recovery/t/015_promotion_pages.pl | 4 +-- src/test/recovery/t/026_overwrite_contrecord.pl| 2 +- src/test/subscription/t/008_diff_schema.pl | 4 +-- src/test/subscription/t/015_stream.pl | 16 +- src/test/subscription/t/016_stream_subxact.pl | 24 +++ src/test/subscription/t/017_stream_ddl.pl | 18 ++-- .../subscription/t/018_stream_subxact_abort.pl | 34 +++--- .../subscription/t/019_stream_subxact_ddl_abort.pl | 14 - src/test/subscription/t/022_twophase_cascade.pl| 14 - src/test/subscription/t/023_twophase_stream.pl | 24 +++ src/test/subscription/t/029_on_error.pl| 8 ++--- 17 files changed, 111 insertions(+), 111 deletions(-)
pgsql: Allow Incremental Sorts on GiST and SP-GiST indexes
Allow Incremental Sorts on GiST and SP-GiST indexes Previously an "amcanorderbyop" index would only be used when the index could provide sorted results which satisfied all query_pathkeys. Here we relax this so that we also allow these indexes to be considered by the planner when they only provide partially sorted results. This allows the planner to later consider making use of an Incremental Sort to satisfy the remaining pathkeys. This change is particularly useful for KNN-type queries which contain a LIMIT clause and an additional ORDER BY clause for a non-indexed column. Author: Miroslav Bendik Reviewed-by: Richard Guo, David Rowley Discussion: https://postgr.es/m/CAPoEpV0QYDtzjwamwWUBqyWpaCVbJV2d6qOD7Uy09bWn47PJtw%40mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/625d5b3ca0968c1d8c080d5210f7209184c0d134 Modified Files -- src/backend/optimizer/path/indxpath.c | 53 ++ src/test/regress/expected/incremental_sort.out | 33 src/test/regress/sql/incremental_sort.sql | 16 3 files changed, 77 insertions(+), 25 deletions(-)
Re: pgsql: ci: Change macOS builds from Intel to ARM.
On 29.06.23 15:42, Daniel Gustafsson wrote: On 29 Jun 2023, at 11:20, Peter Eisentraut wrote: On 31.12.22 22:46, Thomas Munro wrote: ci: Change macOS builds from Intel to ARM. I would like to backpatch the changes to src/test/kerberos/t/001_auth.pl and src/test/ldap/t/001_auth.pl further, to PG11..PG14, so that the tests can be run manually on macOS/ARM+Homebrew as well. +1 done
pgsql: Adjust kerberos and ldap tests for Homebrew on ARM
Adjust kerberos and ldap tests for Homebrew on ARM The Homebrew package manager changed its default installation prefix for the new architecture, so a couple of tests need tweaks to find binaries. This is a partial backpatch of dc513bc654. Branch -- REL_12_STABLE Details --- https://git.postgresql.org/pg/commitdiff/3bba0f917f8aa79cdb261b50bb29d96982911e4a Modified Files -- src/test/kerberos/t/001_auth.pl | 9 - src/test/ldap/t/001_auth.pl | 10 -- 2 files changed, 16 insertions(+), 3 deletions(-)
pgsql: Adjust kerberos and ldap tests for Homebrew on ARM
Adjust kerberos and ldap tests for Homebrew on ARM The Homebrew package manager changed its default installation prefix for the new architecture, so a couple of tests need tweaks to find binaries. This is a partial backpatch of dc513bc654. Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/070bf5cda56a4750e09818ae203959ed4985137f Modified Files -- src/test/kerberos/t/001_auth.pl | 9 - src/test/ldap/t/001_auth.pl | 10 -- 2 files changed, 16 insertions(+), 3 deletions(-)
pgsql: Adjust kerberos and ldap tests for Homebrew on ARM
Adjust kerberos and ldap tests for Homebrew on ARM The Homebrew package manager changed its default installation prefix for the new architecture, so a couple of tests need tweaks to find binaries. This is a partial backpatch of dc513bc654. Branch -- REL_11_STABLE Details --- https://git.postgresql.org/pg/commitdiff/c2cff443b0b182a07ef4aa17559302ace48c3e62 Modified Files -- src/test/kerberos/t/001_auth.pl | 9 - src/test/ldap/t/001_auth.pl | 10 -- 2 files changed, 16 insertions(+), 3 deletions(-)
pgsql: Adjust kerberos and ldap tests for Homebrew on ARM
Adjust kerberos and ldap tests for Homebrew on ARM The Homebrew package manager changed its default installation prefix for the new architecture, so a couple of tests need tweaks to find binaries. This is a partial backpatch of dc513bc654. Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/75373ff6c0ef2c517eff57c8b6ea996a4858a0ba Modified Files -- src/test/kerberos/t/001_auth.pl | 9 - src/test/ldap/t/001_auth.pl | 10 -- 2 files changed, 16 insertions(+), 3 deletions(-)
