Hi hackers, The current per-backend statistics design excludes several process types and duplicates accounting and storage: WAL, lock, and IO activity is reported both to the global statistics and to PGSTAT_KIND_BACKEND.
This duplicated work was raised as a concern in [1], and its cost would grow as more statistics acquire per-backend reporting. Andres proposed a new design in [2]: " I think what we'd want is rather something where for each applicable stats kind we have a shared counter for all exited backends and then per-backend counters for live backends, with helpers to aggregate the exited + live stats to a total " and this is what the attached patch series is implementing. Its main benefits are: 1/ each counter is reported and stored once, rather than in both the fixed global statistics and PGSTAT_KIND_BACKEND. 2/ WAL, lock, and IO flushes update a per backend entry without acquiring the corresponding global statistics lock or performing a hash lookup. This removes global lock contention from the flush path. 3/ WAL, lock, and IO statistics have independent storage and locking instead of sharing one PGSTAT_KIND_BACKEND entry. 4/ per-backend statistics become available for auxiliary processes previously excluded by the backend type filtering. There are 2 tradeoff though: 1/ fetching global statistics now requires combining all live entries. 2/ a shared statistics reset now clears both the global stats and all live entries. Otherwise, values from live entries would immediately reappear in the global view. The first tradeoff moves work from the frequent flush path to the comparatively infrequent query path. The second is a documented behavior change. The patch series is organized that way: 0001: add tests for per-backend statistics It adds new tests that will serve as compatibility coverage for the redesign. It could be applied while we are discussing the other patches. 0002: add new infrastructure for per-backend statistics It introduces the common per-backend entry header, the kind metadata needed to describe per-backend storage, and backend-local state holding the attached dshash and cached entry pointer. It provides the common operations for creating and attaching the hashes, creating and caching the current process's entries, fetching an individual entry, constructing consistent snapshots, transferring entries into global statistics, removing entries, and the accumulation at clean server shutdown. No statistics kind registers per-backend metadata in this patch. 0003 moves WAL statistics to the infrastructure introduced by 0002. 0004 performs the corresponding conversion for lock statistics. 0005 performs the corresponding conversion for IO statistics. Once WAL, lock, and IO have moved, PGSTAT_KIND_BACKEND contains no data, so this patch also removes that kind and pgstat_backend.c, together with their obsolete infrastructure. Design explanation for the new hashes: - a fixed array indexed by ProcNumber would avoid hash operations, but it would reserve shared memory for every possible process slot and every participating kind. Queries could also have to inspect unused slots. The dshash allocates entries for processes that actually exist and lets queries iterate those entries directly. - reusing the current variable numbered statistics hash would require aggregate queries to scan unrelated statistics entries or require another structure for enumerating only the live entries of that kind. A dedicated per kind hash provides that enumeration directly. Remark: The patch series limits this new infrastructure to built in fixed numbered statistics kinds as this is the only use case we have had so far. We could extend to variable ones later on if needed. [1]: https://postgr.es/m/7fhpds4xqk6bnudzmzkqi33pinsxammpljwde5gfkjdygvejrj@ojkzfr7dxkmm [2]: https://postgr.es/m/et272fdhdx6yphlgzvrgsf7bgwnf3vqciwp4gxqubro42uaflp%40ohslaocvwgvi Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
>From abc8330baf690a036114ea688fcaa4643ab641c9 Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot <[email protected]> Date: Fri, 17 Jul 2026 13:23:40 +0000 Subject: [PATCH v1 1/5] pgstat: add tests for per-backend statistics Add an isolation test for per-backend WAL statistics. Verify that SNAPSHOT, CACHE and NONE modes behave as expected. Add a TAP test that starts a backend after another session has built a statistics snapshot. Per-backend WAL, IO, and Lock queries for the new backend must return no rows, rather than fabricated rows containing zeroes. Extend test_shm_mq with a shared-memory-only worker that reports WAL usage through a routine nonblocking flush and remains alive while the controlling backend verifies that the global WAL counters include it. Finally, extend the core statistics regression test for per-backend WAL reset. Check that resetting one backend reduces its counters and sets its reset timestamp without removing the same activity from the global WAL statistics. These tests will serve as compatibility coverage for the per-backend redesign changes coming in following commits. Author: Bertrand Drouvot <[email protected]> Reviewed-by: Discussion: --- .../isolation/expected/stats-per-backend.out | 143 ++++++++++++++++++ src/test/isolation/isolation_schedule | 1 + .../isolation/specs/stats-per-backend.spec | 123 +++++++++++++++ src/test/modules/test_misc/meson.build | 1 + .../modules/test_misc/t/015_stats_snapshot.pl | 48 ++++++ .../test_shm_mq/expected/test_shm_mq.out | 7 + .../modules/test_shm_mq/sql/test_shm_mq.sql | 3 + src/test/modules/test_shm_mq/test.c | 51 +++++++ .../modules/test_shm_mq/test_shm_mq--1.0.sql | 4 + src/test/modules/test_shm_mq/test_shm_mq.h | 4 + src/test/modules/test_shm_mq/worker.c | 10 ++ src/test/regress/expected/stats.out | 29 ++++ src/test/regress/sql/stats.sql | 10 ++ 13 files changed, 434 insertions(+) 30.6% src/test/isolation/expected/ 31.4% src/test/isolation/specs/ 9.7% src/test/modules/test_misc/t/ 17.6% src/test/modules/test_shm_mq/ 5.6% src/test/regress/expected/ 4.3% src/test/regress/sql/ diff --git a/src/test/isolation/expected/stats-per-backend.out b/src/test/isolation/expected/stats-per-backend.out new file mode 100644 index 00000000000..0d85ae99033 --- /dev/null +++ b/src/test/isolation/expected/stats-per-backend.out @@ -0,0 +1,143 @@ +Parsed test spec with 2 sessions + +starting permutation: s1_fetch_consistency_snapshot s1_begin s1_build_snapshot s2_generate s1_check_snapshot_backend s1_commit s1_check_live_backend +step s1_fetch_consistency_snapshot: SET stats_fetch_consistency = 'snapshot'; +step s1_begin: BEGIN; +step s1_build_snapshot: + SELECT wal_records >= 0 AS snapshot_created FROM pg_stat_wal; + +snapshot_created +---------------- +t +(1 row) + +step s2_generate: + INSERT INTO stats_per_backend_data VALUES (1); + SELECT pg_stat_force_next_flush(); + +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_check_snapshot_backend: + SELECT wal_records = 0 AS snapshot_excludes_later_update + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); + +snapshot_excludes_later_update +------------------------------ +t +(1 row) + +step s1_commit: COMMIT; +step s1_check_live_backend: + SELECT wal_records > 0 AS live_entry_exists + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); + +live_entry_exists +----------------- +t +(1 row) + + +starting permutation: s1_fetch_consistency_cache s2_generate s1_begin s1_save_backend_stats s2_generate_more s1_check_cached_backend s1_commit s1_check_refreshed_backend +step s1_fetch_consistency_cache: SET stats_fetch_consistency = 'cache'; +step s2_generate: + INSERT INTO stats_per_backend_data VALUES (1); + SELECT pg_stat_force_next_flush(); + +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_begin: BEGIN; +step s1_save_backend_stats: + INSERT INTO stats_per_backend_saved + SELECT wal_records + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); + +step s2_generate_more: + INSERT INTO stats_per_backend_data VALUES (2); + SELECT pg_stat_force_next_flush(); + +pg_stat_force_next_flush +------------------------ + +(1 row) + +step s1_check_cached_backend: + SELECT current.wal_records = saved.wal_records AS cache_is_stable + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')) AS current + CROSS JOIN stats_per_backend_saved AS saved; + +cache_is_stable +--------------- +t +(1 row) + +step s1_commit: COMMIT; +step s1_check_refreshed_backend: + SELECT current.wal_records > saved.wal_records AS cache_is_refreshed + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')) AS current + CROSS JOIN stats_per_backend_saved AS saved; + +cache_is_refreshed +------------------ +t +(1 row) + + +starting permutation: s1_reset_backend s1_check_backend_reset +step s1_reset_backend: + SELECT pg_stat_reset_backend_stats( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); + +pg_stat_reset_backend_stats +--------------------------- + +(1 row) + +step s1_check_backend_reset: + SELECT stats_reset IS NOT NULL AS reset_timestamp_set + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); + +reset_timestamp_set +------------------- +t +(1 row) + + +starting permutation: s1_reset_shared s1_check_shared_reset +step s1_reset_shared: + SELECT pg_stat_reset_shared('wal'); + +pg_stat_reset_shared +-------------------- + +(1 row) + +step s1_check_shared_reset: + SELECT stats_reset IS NOT NULL AS reset_timestamp_set + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); + +reset_timestamp_set +------------------- +t +(1 row) + diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index 26abed9f9f0..5d97c56b5d6 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -104,6 +104,7 @@ test: vacuum-concurrent-drop test: vacuum-conflict test: vacuum-skip-locked test: stats +test: stats-per-backend test: horizons test: predicate-hash test: predicate-gist diff --git a/src/test/isolation/specs/stats-per-backend.spec b/src/test/isolation/specs/stats-per-backend.spec new file mode 100644 index 00000000000..5dfc8dbe90e --- /dev/null +++ b/src/test/isolation/specs/stats-per-backend.spec @@ -0,0 +1,123 @@ +# Test snapshot, cache, and reset behavior for per-backend statistics. +# +# Session s2 generates and flushes WAL records, which provide a deterministic +# per-backend counter. Session s1 verifies that SNAPSHOT mode fixes all +# per-backend entries at the initial statistics snapshot, while CACHE mode +# fixes each entry on first access and refreshes it after the transaction +# ends. The final permutations verify reset timestamps after per-backend and +# shared resets. + +setup +{ + CREATE TABLE stats_per_backend_data(id int); + CREATE TABLE stats_per_backend_saved(wal_records bigint); +} + +teardown +{ + DROP TABLE stats_per_backend_data; + DROP TABLE stats_per_backend_saved; +} + +session s1 +setup { SET stats_fetch_consistency = 'none'; } + +step s1_fetch_consistency_cache { SET stats_fetch_consistency = 'cache'; } +step s1_fetch_consistency_snapshot { SET stats_fetch_consistency = 'snapshot'; } +step s1_begin { BEGIN; } +step s1_commit { COMMIT; } +step s1_build_snapshot { + SELECT wal_records >= 0 AS snapshot_created FROM pg_stat_wal; +} +step s1_check_snapshot_backend { + SELECT wal_records = 0 AS snapshot_excludes_later_update + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); +} +step s1_check_live_backend { + SELECT wal_records > 0 AS live_entry_exists + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); +} +step s1_save_backend_stats { + INSERT INTO stats_per_backend_saved + SELECT wal_records + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); +} +step s1_check_cached_backend { + SELECT current.wal_records = saved.wal_records AS cache_is_stable + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')) AS current + CROSS JOIN stats_per_backend_saved AS saved; +} +step s1_check_refreshed_backend { + SELECT current.wal_records > saved.wal_records AS cache_is_refreshed + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')) AS current + CROSS JOIN stats_per_backend_saved AS saved; +} +step s1_reset_backend { + SELECT pg_stat_reset_backend_stats( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); +} +step s1_check_backend_reset { + SELECT stats_reset IS NOT NULL AS reset_timestamp_set + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); +} +step s1_reset_shared { + SELECT pg_stat_reset_shared('wal'); +} +step s1_check_shared_reset { + SELECT stats_reset IS NOT NULL AS reset_timestamp_set + FROM pg_stat_get_backend_wal( + (SELECT pid FROM pg_stat_activity + WHERE application_name = 'isolation/stats-per-backend/s2')); +} + +session s2 +step s2_generate { + INSERT INTO stats_per_backend_data VALUES (1); + SELECT pg_stat_force_next_flush(); +} +step s2_generate_more { + INSERT INTO stats_per_backend_data VALUES (2); + SELECT pg_stat_force_next_flush(); +} + +# with stats_fetch_consistency=snapshot s1 should not see flushed changes from +# s2 after building the statistics snapshot, but should see them after commit +permutation + s1_fetch_consistency_snapshot + s1_begin + s1_build_snapshot + s2_generate + s1_check_snapshot_backend + s1_commit + s1_check_live_backend + +# with stats_fetch_consistency=cache s1 should not see flushed changes from s2 +# after the first access, but should see them after commit +permutation + s1_fetch_consistency_cache + s2_generate + s1_begin + s1_save_backend_stats + s2_generate_more + s1_check_cached_backend + s1_commit + s1_check_refreshed_backend + +# a per-backend reset should set that backend's reset timestamp +permutation s1_reset_backend s1_check_backend_reset + +# a shared WAL reset should set the reset timestamp for live backends +permutation s1_reset_shared s1_check_shared_reset diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build index ee290698b31..cf4c8558f3d 100644 --- a/src/test/modules/test_misc/meson.build +++ b/src/test/modules/test_misc/meson.build @@ -23,6 +23,7 @@ tests += { 't/012_ddlutils.pl', 't/013_temp_obj_multisession.pl', 't/014_log_statement_max_length.pl', + 't/015_stats_snapshot.pl', ], # The injection points are cluster-wide, so disable installcheck 'runningcheck': false, diff --git a/src/test/modules/test_misc/t/015_stats_snapshot.pl b/src/test/modules/test_misc/t/015_stats_snapshot.pl new file mode 100644 index 00000000000..43c9157bbb4 --- /dev/null +++ b/src/test/modules/test_misc/t/015_stats_snapshot.pl @@ -0,0 +1,48 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Verify that a statistics snapshot excludes backends that started after the +# snapshot was created. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $node = PostgreSQL::Test::Cluster->new('stats_snapshot'); +$node->init; +$node->start; + +my $snapshot = $node->background_psql('postgres'); + +$snapshot->query_safe( + q[ + BEGIN; + SET LOCAL stats_fetch_consistency = 'snapshot'; + SELECT count(*) FROM pg_stat_wal; +]); + +# Establish this connection only after the first session built its snapshot. +my $late = $node->background_psql('postgres'); +my $late_pid = $late->query_safe('SELECT pg_backend_pid()'); + +# WAL returns one scalar composite, so count a field to distinguish a NULL +# result from the fabricated all-zero row. +is( $snapshot->query_safe( + qq[ + SELECT + (SELECT count(wal_records) + FROM pg_stat_get_backend_wal($late_pid)), + (SELECT count(*) FROM pg_stat_get_backend_io($late_pid)), + (SELECT count(*) FROM pg_stat_get_backend_lock($late_pid)); + ]), + '0|0|0', + 'statistics snapshot excludes a backend created later'); + +$snapshot->query_safe('ROLLBACK'); +$late->quit; +$snapshot->quit; +$node->stop; + +done_testing(); diff --git a/src/test/modules/test_shm_mq/expected/test_shm_mq.out b/src/test/modules/test_shm_mq/expected/test_shm_mq.out index c4858b0c205..ec3a7d3199e 100644 --- a/src/test/modules/test_shm_mq/expected/test_shm_mq.out +++ b/src/test/modules/test_shm_mq/expected/test_shm_mq.out @@ -34,3 +34,10 @@ SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int (1 row) +-- A shmem-only worker can publish statistics before its clean exit. +SELECT test_shm_mq_worker_stats(); + test_shm_mq_worker_stats +-------------------------- + t +(1 row) + diff --git a/src/test/modules/test_shm_mq/sql/test_shm_mq.sql b/src/test/modules/test_shm_mq/sql/test_shm_mq.sql index 9de19d304a2..e80e4399881 100644 --- a/src/test/modules/test_shm_mq/sql/test_shm_mq.sql +++ b/src/test/modules/test_shm_mq/sql/test_shm_mq.sql @@ -10,3 +10,6 @@ SELECT test_shm_mq(1024, 'a', 2001, 1); SELECT test_shm_mq(32768, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,(100+900*random())::int)), 10000, 1); SELECT test_shm_mq(100, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,(100+200*random())::int)), 10000, 1); SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,270000)), 200, 3); + +-- A shmem-only worker can publish statistics before its clean exit. +SELECT test_shm_mq_worker_stats(); diff --git a/src/test/modules/test_shm_mq/test.c b/src/test/modules/test_shm_mq/test.c index 0e55287e510..4464d796ef9 100644 --- a/src/test/modules/test_shm_mq/test.c +++ b/src/test/modules/test_shm_mq/test.c @@ -26,6 +26,7 @@ PG_MODULE_MAGIC; PG_FUNCTION_INFO_V1(test_shm_mq); PG_FUNCTION_INFO_V1(test_shm_mq_pipelined); +PG_FUNCTION_INFO_V1(test_shm_mq_worker_stats); static void verify_message(Size origlen, char *origdata, Size newlen, char *newdata); @@ -253,6 +254,56 @@ test_shm_mq_pipelined(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } +/* + * Verify that a shmem-only worker can publish statistics through a routine + * nonblocking flush while it remains alive. + */ +Datum +test_shm_mq_worker_stats(PG_FUNCTION_ARGS) +{ + const char *message = PG_TEST_SHM_MQ_STATS_MESSAGE; + PgStat_Counter before; + PgStat_Counter after; + dsm_segment *seg; + shm_mq_handle *outqh; + shm_mq_handle *inqh; + shm_mq_result res; + Size len; + void *data; + + pgstat_clear_snapshot(); + before = pgstat_fetch_stat_wal()->wal_counters.wal_records; + + test_shm_mq_setup(1024, 1, &seg, &outqh, &inqh); + + res = shm_mq_send(outqh, strlen(message), message, false, true); + + if (res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not send statistics test message"))); + + res = shm_mq_receive(inqh, &len, &data, false); + + if (res != SHM_MQ_SUCCESS) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not receive statistics test message"))); + + verify_message(strlen(message), (char *) message, len, data); + + /* + * The worker is waiting for another message here. Check its live entry + * before detaching the queues and triggering its shutdown flush. + */ + pgstat_clear_snapshot(); + after = pgstat_fetch_stat_wal()->wal_counters.wal_records; + + dsm_detach(seg); + + PG_RETURN_BOOL(after >= before + PG_TEST_SHM_MQ_STATS_RECORDS); +} + /* * Verify that two messages are the same. */ diff --git a/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql b/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql index 56db05d93df..dce6f7378a5 100644 --- a/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql +++ b/src/test/modules/test_shm_mq/test_shm_mq--1.0.sql @@ -17,3 +17,7 @@ CREATE FUNCTION test_shm_mq_pipelined(queue_size pg_catalog.int8, verify pg_catalog.bool default true) RETURNS pg_catalog.void STRICT AS 'MODULE_PATHNAME' LANGUAGE C; + +CREATE FUNCTION test_shm_mq_worker_stats() + RETURNS pg_catalog.bool + AS 'MODULE_PATHNAME' LANGUAGE C; diff --git a/src/test/modules/test_shm_mq/test_shm_mq.h b/src/test/modules/test_shm_mq/test_shm_mq.h index b6a0290289c..501be180642 100644 --- a/src/test/modules/test_shm_mq/test_shm_mq.h +++ b/src/test/modules/test_shm_mq/test_shm_mq.h @@ -21,6 +21,10 @@ /* Identifier for shared memory segments used by this extension. */ #define PG_TEST_SHM_MQ_MAGIC 0x79fb2447 +/* Special message used to test statistics from a shmem-only worker. */ +#define PG_TEST_SHM_MQ_STATS_MESSAGE "test_shm_mq worker stats" +#define PG_TEST_SHM_MQ_STATS_RECORDS 1000000 + /* * This structure is stored in the dynamic shared memory segment. We use * it to determine whether all workers started up OK and successfully diff --git a/src/test/modules/test_shm_mq/worker.c b/src/test/modules/test_shm_mq/worker.c index 0ba1cfcac47..afcea7fc0cb 100644 --- a/src/test/modules/test_shm_mq/worker.c +++ b/src/test/modules/test_shm_mq/worker.c @@ -19,6 +19,7 @@ #include "postgres.h" +#include "executor/instrument.h" #include "miscadmin.h" #include "storage/ipc.h" #include "storage/latch.h" @@ -27,6 +28,7 @@ #include "storage/shm_mq.h" #include "storage/shm_toc.h" #include "tcop/tcopprot.h" +#include "utils/pgstat_internal.h" #include "test_shm_mq.h" @@ -185,6 +187,14 @@ copy_messages(shm_mq_handle *inqh, shm_mq_handle *outqh) if (res != SHM_MQ_SUCCESS) break; + if (len == strlen(PG_TEST_SHM_MQ_STATS_MESSAGE) && + memcmp(data, PG_TEST_SHM_MQ_STATS_MESSAGE, len) == 0) + { + pgWalUsage.wal_records += PG_TEST_SHM_MQ_STATS_RECORDS; + pgstat_report_fixed = true; + (void) pgstat_report_stat(false); + } + /* Send it back out. */ res = shm_mq_send(outqh, len, data, false, true); if (res != SHM_MQ_SUCCESS) diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out index e230356de13..fae5f939523 100644 --- a/src/test/regress/expected/stats.out +++ b/src/test/regress/expected/stats.out @@ -1044,6 +1044,35 @@ SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_bac t (1 row) +-- Test pg_stat_reset_backend_stats() for WAL +SELECT wal_bytes AS wal_bytes_before_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset +SELECT pg_stat_reset_backend_stats(pg_backend_pid()); + pg_stat_reset_backend_stats +----------------------------- + +(1 row) + +SELECT wal_bytes AS wal_bytes_after_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset +SELECT :wal_bytes_after_reset < :wal_bytes_before_reset; + ?column? +---------- + t +(1 row) + +-- global view should still have the data +SELECT wal_bytes >= :wal_bytes_before_reset FROM pg_stat_wal; + ?column? +---------- + t +(1 row) + +-- reset timestamp should be set +SELECT stats_reset IS NOT NULL FROM pg_stat_get_backend_wal(pg_backend_pid()); + ?column? +---------- + t +(1 row) + -- Test pg_stat_get_backend_idset() and some allied functions. -- In particular, verify that their notion of backend ID matches -- our temp schema index. diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql index 4c265d1245c..c17753dcd20 100644 --- a/src/test/regress/sql/stats.sql +++ b/src/test/regress/sql/stats.sql @@ -482,6 +482,16 @@ SELECT wal_bytes > :wal_bytes_before FROM pg_stat_wal; SELECT pg_stat_force_next_flush(); SELECT wal_bytes > :backend_wal_bytes_before FROM pg_stat_get_backend_wal(pg_backend_pid()); +-- Test pg_stat_reset_backend_stats() for WAL +SELECT wal_bytes AS wal_bytes_before_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset +SELECT pg_stat_reset_backend_stats(pg_backend_pid()); +SELECT wal_bytes AS wal_bytes_after_reset FROM pg_stat_get_backend_wal(pg_backend_pid()) \gset +SELECT :wal_bytes_after_reset < :wal_bytes_before_reset; +-- global view should still have the data +SELECT wal_bytes >= :wal_bytes_before_reset FROM pg_stat_wal; +-- reset timestamp should be set +SELECT stats_reset IS NOT NULL FROM pg_stat_get_backend_wal(pg_backend_pid()); + -- Test pg_stat_get_backend_idset() and some allied functions. -- In particular, verify that their notion of backend ID matches -- our temp schema index. -- 2.34.1
>From b1d28f72cfc6fe8925a334a0d8f7da0626d1cf9d Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot <[email protected]> Date: Wed, 29 Jul 2026 14:11:27 +0000 Subject: [PATCH v1 2/5] pgstat: add new infrastructure for per-backend statistics Add a new infrastructure for per-backend statistics that keep one shared entry per live backend while keeping their existing shared data as global stats for exited backends. Define a common dshash entry header containing the ProcNumber key, BackendType, and content LWLock. Extend PgStat_KindInfo with per-backend statistics related informations. At shared-memory initialization, create a dshash for every participating kind. Create entries during pgstat_initialize(), so that auxiliary and shared memory only processes are covered. Nonblocking flushes can then acquire the cached entry's content lock without performing a dshash lookup, DSA address resolution, allocation, or hash resize. Add helpers to fetch per-backend entries, include live entries in global snapshots, and transfer entries to the global statistics when backends exit. Add a local cache keyed by statistics kind and ProcNumber for per-backend fetches. No statistics kind registers per backend metadata in this commit, so the new shared memory creation and backend initialization loops are no ops. Subsequent commits will add WAL, Lock, and IO statistics into the infrastructure individually. Limit this infrastructure to built in fixed numbered statistics kinds as this is the only use case we have had so far. We could extend to variable ones later on if needed. Author: Bertrand Drouvot <[email protected]> Reviewed-by: Discussion: --- src/backend/utils/activity/pgstat.c | 506 ++++++++++++++++++++++ src/backend/utils/activity/pgstat_shmem.c | 46 ++ src/include/utils/pgstat_internal.h | 43 ++ src/tools/pgindent/typedefs.list | 4 + 4 files changed, 599 insertions(+) 89.6% src/backend/utils/activity/ 9.6% src/include/utils/ diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 50cd07822b4..efc7d427f22 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -175,6 +175,41 @@ typedef struct PgStat_SnapshotEntry #define SH_DECLARE #include "lib/simplehash.h" +/* hash table for per-backend stats snapshot entries */ +typedef struct PgStat_PerBackendSnapshotKey +{ + PgStat_Kind kind; + ProcNumber procnum; +} PgStat_PerBackendSnapshotKey; + +typedef struct PgStat_PerBackendSnapshotEntry +{ + PgStat_PerBackendSnapshotKey key; + char status; /* for simplehash use */ + void *data; /* the stats data itself */ +} PgStat_PerBackendSnapshotEntry; + +#define SH_PREFIX pgstat_per_backend_snapshot +#define SH_ELEMENT_TYPE PgStat_PerBackendSnapshotEntry +#define SH_KEY_TYPE PgStat_PerBackendSnapshotKey +#define SH_KEY key +#define SH_HASH_KEY(tb, key) \ + fasthash32((const char *) &key, sizeof(PgStat_PerBackendSnapshotKey), 0) +#define SH_EQUAL(tb, a, b) \ + (memcmp(&a, &b, sizeof(PgStat_PerBackendSnapshotKey)) == 0) +#define SH_SCOPE static inline +#define SH_DEFINE +#define SH_DECLARE +#include "lib/simplehash.h" + +/* Per-kind, backend-local state for the per-backend dshashes. */ +typedef struct PgStat_PerBackendLocalState +{ + dshash_table *hash; + PgStatShared_PerBackendEntry *my_entry; +} PgStat_PerBackendLocalState; + +static PgStat_PerBackendLocalState per_backend_states[PGSTAT_KIND_BUILTIN_SIZE]; /* ---------- * Local function forward declarations @@ -195,6 +230,7 @@ static void pgstat_build_snapshot(void); static void pgstat_build_snapshot_fixed(PgStat_Kind kind); static inline bool pgstat_is_kind_valid(PgStat_Kind kind); +static void pgstat_create_my_per_backend_entries(void); /* ---------- @@ -673,6 +709,13 @@ pgstat_initialize(void) pgstat_attach_shmem(); + /* + * Create and cache per-backend statistics entries here. This also covers + * processes that never call InitPostgres(), such as shared-memory-only + * background workers. + */ + pgstat_create_my_per_backend_entries(); + pgstat_init_snapshot_fixed(); /* Backend initialization callbacks */ @@ -946,6 +989,7 @@ pgstat_clear_snapshot(void) /* Reset variables */ pgStatLocal.snapshot.context = NULL; + pgStatLocal.snapshot.per_backend_stats = NULL; } /* @@ -1161,6 +1205,463 @@ pgstat_prep_snapshot(void) NULL); } +/* + * Look up a per-backend stats entry in the backend snapshot hash. + * + * The returned entry may be empty when no matching statistics were found on + * first access. + */ +static PgStat_PerBackendSnapshotEntry * +pgstat_lookup_per_backend_entry(PgStat_Kind kind, ProcNumber procnum) +{ + PgStat_PerBackendSnapshotKey key; + + if (pgStatLocal.snapshot.per_backend_stats == NULL) + return NULL; + + key.kind = kind; + key.procnum = procnum; + + return pgstat_per_backend_snapshot_lookup(pgStatLocal.snapshot.per_backend_stats, + key); +} + +/* + * Cache a per-backend stats entry in the backend snapshot hash. + * + * If data is NULL, cache an empty entry to record that no matching statistics + * were found on first access. + */ +static void * +pgstat_cache_per_backend_entry(PgStat_Kind kind, ProcNumber procnum, + const void *data) +{ + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + PgStat_PerBackendSnapshotKey key; + PgStat_PerBackendSnapshotEntry *entry; + bool found; + + Assert(pgstat_fetch_consistency > PGSTAT_FETCH_CONSISTENCY_NONE); + Assert(kind_info != NULL); + Assert(kind_info->per_backend_data_len > 0); + + /* Ensure snapshot context exists */ + if (!pgStatLocal.snapshot.context) + pgStatLocal.snapshot.context = AllocSetContextCreate(TopMemoryContext, + "PgStat Snapshot", + ALLOCSET_SMALL_SIZES); + + /* Create per-backend hash on first use */ + if (pgStatLocal.snapshot.per_backend_stats == NULL) + pgStatLocal.snapshot.per_backend_stats = + pgstat_per_backend_snapshot_create(pgStatLocal.snapshot.context, 64, NULL); + + key.kind = kind; + key.procnum = procnum; + + /* If already cached, return cached data */ + entry = pgstat_per_backend_snapshot_lookup(pgStatLocal.snapshot.per_backend_stats, key); + + if (entry) + return entry->data; + + /* Insert new entry into the hash */ + entry = pgstat_per_backend_snapshot_insert(pgStatLocal.snapshot.per_backend_stats, + key, &found); + Assert(!found); + + if (data != NULL) + { + entry->data = MemoryContextAlloc(pgStatLocal.snapshot.context, + kind_info->per_backend_data_len); + memcpy(entry->data, data, kind_info->per_backend_data_len); + } + else + entry->data = NULL; + + return entry->data; +} + +static inline PgStat_PerBackendLocalState * +pgstat_get_per_backend_local_state(PgStat_Kind kind) +{ + const PgStat_KindInfo *kind_info PG_USED_FOR_ASSERTS_ONLY = pgstat_get_kind_info(kind); + + Assert(kind_info != NULL); + Assert(kind_info->fixed_amount); + + if (!pgstat_is_kind_builtin(kind)) + elog(ERROR, "invalid statistics kind: %u", kind); + + return &per_backend_states[kind]; +} + +/* + * Create and cache this process's entry for one per-backend statistics kind. + */ +static PgStatShared_PerBackendEntry * +pgstat_create_my_per_backend_entry(PgStat_Kind kind) +{ + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind); + dshash_table *hash = pgstat_per_backend_attach(kind); + PgStatShared_PerBackendEntry *entry; + bool found; + + Assert(state->my_entry == NULL); + + if (hash == NULL) + return NULL; + + Assert(kind_info != NULL); + Assert(kind_info->per_backend_data_len > 0); + + entry = dshash_find_or_insert(hash, &MyProcNumber, &found); + + /* + * A forced flush during early backend startup may already have created + * the entry. Preserve any statistics it contains. + */ + if (!found) + { + entry->backend_type = MyBackendType; + LWLockInitialize(&entry->lock, LWTRANCHE_PGSTATS_DATA); + memset((char *) entry + kind_info->per_backend_data_off, 0, + kind_info->per_backend_data_len); + } + + state->my_entry = entry; + dshash_release_lock(hash, entry); + + return entry; +} + +/* + * Create entries for all the kinds that use per-backend dshashes. + * + * Allocations and dshash resizes are deliberately done during backend + * initialization so routine stats flushes only need to conditionally acquire + * the cached entry's content lock. + */ +static void +pgstat_create_my_per_backend_entries(void) +{ + for (PgStat_Kind kind = PGSTAT_KIND_BUILTIN_MIN; + kind <= PGSTAT_KIND_BUILTIN_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (kind_info == NULL || kind_info->per_backend_data_len == 0) + continue; + + (void) pgstat_create_my_per_backend_entry(kind); + } +} + +/* + * Attach to the per-backend dshash for the given kind. + * Returns NULL if the hash is not available (e.g. during bootstrap or + * if this kind doesn't have per-backend tracking). + */ +dshash_table * +pgstat_per_backend_attach(PgStat_Kind kind) +{ + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind); + char *shared_struct; + dshash_table_handle *handle_ptr; + MemoryContext oldcontext; + dshash_parameters params; + + if (state->hash != NULL) + return state->hash; + + if (!kind_info || !kind_info->per_backend_data_len) + return NULL; + + /* Get the shared struct for this kind */ + shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off; + handle_ptr = (dshash_table_handle *) (shared_struct + kind_info->per_backend_hash_handle_off); + + if (*handle_ptr == DSHASH_HANDLE_INVALID) + return NULL; + + /* + * Build dshash parameters from kind info. All per-backend hashes use + * ProcNumber keys and dshash_memcmp/dshash_memhash. + */ + params.key_size = sizeof(ProcNumber); + params.entry_size = kind_info->per_backend_data_off + kind_info->per_backend_data_len; + params.compare_function = dshash_memcmp; + params.hash_function = dshash_memhash; + params.copy_function = dshash_memcpy; + params.tranche_id = LWTRANCHE_PGSTATS_HASH; + + /* Attach in TopMemoryContext */ + oldcontext = MemoryContextSwitchTo(TopMemoryContext); + state->hash = dshash_attach(pgStatLocal.dsa, ¶ms, *handle_ptr, NULL); + MemoryContextSwitchTo(oldcontext); + + return state->hash; +} + +/* + * Lock this process's cached entry for a per-backend statistics kind. + * + * A missing entry is only created in the blocking path. The routine nowait + * path must not perform dshash lookups, allocations, or DSA address + * resolution. + */ +void * +pgstat_lock_my_per_backend_entry(PgStat_Kind kind, bool nowait) +{ + const PgStat_KindInfo *kind_info PG_USED_FOR_ASSERTS_ONLY = pgstat_get_kind_info(kind); + PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind); + PgStatShared_PerBackendEntry *entry = state->my_entry; + + Assert(kind_info != NULL); + Assert(kind_info->per_backend_data_len > 0); + + if (entry == NULL) + { + if (nowait) + return NULL; + + entry = pgstat_create_my_per_backend_entry(kind); + if (entry == NULL) + return NULL; + } + + if (nowait) + { + if (!LWLockConditionalAcquire(&entry->lock, LW_EXCLUSIVE)) + return NULL; + } + else + LWLockAcquire(&entry->lock, LW_EXCLUSIVE); + + return entry; +} + +/* + * Accumulate all live per-backend entries into the kind's data in + * pgStatLocal.snapshot, optionally caching each entry for SNAPSHOT mode. + */ +void +pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash, void *snap) +{ + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + dshash_seq_status hstat; + PgStatShared_PerBackendEntry *entry; + + dshash_seq_init(&hstat, hash, false); + + while ((entry = dshash_seq_next(&hstat)) != NULL) + { + LWLockAcquire(&entry->lock, LW_SHARED); + + /* Kind-specific accumulation into global snapshot */ + kind_info->per_backend_acc_cb(snap, entry); + + /* + * In SNAPSHOT mode, cache each per-backend entry so that + * pgstat_fetch_per_backend() can return a consistent point-in-time + * view without re-reading from shared memory. + */ + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT) + { + pgstat_cache_per_backend_entry(kind, entry->key, + (char *) entry + kind_info->per_backend_data_off); + } + + LWLockRelease(&entry->lock); + } + + dshash_seq_term(&hstat); +} + +/* + * Fetch per-backend stats for the given kind and ProcNumber. + * Returns NULL if no entry exists. In NONE mode, returns a copy allocated in + * the current memory context. In CACHE and SNAPSHOT modes, returns a pointer + * owned by the statistics snapshot cache. + */ +void * +pgstat_fetch_per_backend(PgStat_Kind kind, ProcNumber procnum) +{ + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + dshash_table *hash; + PgStatShared_PerBackendEntry *entry; + void *stats_data; + PgStat_PerBackendSnapshotEntry *snapshot_entry; + + if (force_stats_snapshot_clear) + pgstat_clear_snapshot(); + + hash = pgstat_per_backend_attach(kind); + + if (hash == NULL) + return NULL; + + /* In NONE mode, read directly and don't cache */ + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_NONE) + { + entry = dshash_find(hash, &procnum, false); + if (entry == NULL) + return NULL; + + LWLockAcquire(&entry->lock, LW_SHARED); + stats_data = palloc(kind_info->per_backend_data_len); + memcpy(stats_data, (char *) entry + kind_info->per_backend_data_off, + kind_info->per_backend_data_len); + LWLockRelease(&entry->lock); + dshash_release_lock(hash, entry); + + return stats_data; + } + + Assert(pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_CACHE || + pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT); + + /* + * Building a full snapshot pre-caches all existing per-backend entries. + * CACHE mode only needs the requested entry, so it must not build the + * aggregate fixed-kind snapshot and scan every live backend. + */ + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT) + pgstat_snapshot_fixed(kind); + + snapshot_entry = pgstat_lookup_per_backend_entry(kind, procnum); + + if (snapshot_entry != NULL) + return snapshot_entry->data; + + /* + * Once a full snapshot has been built, a cache miss means the entry did + * not exist at the snapshot point. Do not admit an entry created later. + */ + if (pgstat_fetch_consistency == PGSTAT_FETCH_CONSISTENCY_SNAPSHOT) + return NULL; + + /* CACHE miss: copy the live entry directly into the snapshot context. */ + entry = dshash_find(hash, &procnum, false); + + if (entry == NULL) + return pgstat_cache_per_backend_entry(kind, procnum, NULL); + + LWLockAcquire(&entry->lock, LW_SHARED); + + stats_data = pgstat_cache_per_backend_entry(kind, procnum, + (char *) entry + kind_info->per_backend_data_off); + + LWLockRelease(&entry->lock); + dshash_release_lock(hash, entry); + + return stats_data; +} + +/* + * Accumulate this process's per-backend stats into the global stats, then + * remove the entry from the dshash. + * Acquire the kind lock before the dshash partition lock. Snapshots and resets + * hold the kind lock while accessing both the global stats and live entries, + * so an entry cannot move between them during either operation. + * + * NB: The entry may belong to an earlier process that used the same ProcNumber. + * It must still be accumulated before removal. + */ +void +pgstat_acc_my_per_backend(PgStat_Kind kind, LWLock *lock) +{ + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind); + char *shared_struct; + dshash_table *hash; + void *dst; + PgStatShared_PerBackendEntry *entry; + + hash = pgstat_per_backend_attach(kind); + + if (hash == NULL) + return; + + shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off; + dst = shared_struct + kind_info->shared_data_off; + + LWLockAcquire(lock, LW_EXCLUSIVE); + + entry = dshash_find(hash, &MyProcNumber, true); + + if (entry == NULL) + { + state->my_entry = NULL; + LWLockRelease(lock); + return; + } + + LWLockAcquire(&entry->lock, LW_EXCLUSIVE); + + if (state->my_entry == entry) + state->my_entry = NULL; + + /* Kind-specific accumulation */ + kind_info->per_backend_acc_cb(dst, entry); + + LWLockRelease(&entry->lock); + + /* Remove the entry */ + dshash_delete_entry(hash, entry); + + LWLockRelease(lock); +} + +/* + * Accumulate all per-backend entries into global stats and delete them. + * Called at clean server shutdown before writing the stats file. Acquire the + * kind's global lock before starting the dshash scan. + */ +void +pgstat_acc_all_per_backend(PgStat_Kind kind, LWLock *lock) +{ + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + PgStat_PerBackendLocalState *state = pgstat_get_per_backend_local_state(kind); + char *shared_struct; + dshash_table *hash; + void *dst; + dshash_seq_status hstat; + PgStatShared_PerBackendEntry *entry; + + hash = pgstat_per_backend_attach(kind); + + if (hash == NULL) + return; + + shared_struct = (char *) pgStatLocal.shmem + kind_info->shared_ctl_off; + dst = shared_struct + kind_info->shared_data_off; + + LWLockAcquire(lock, LW_EXCLUSIVE); + + dshash_seq_init(&hstat, hash, true); + + while ((entry = dshash_seq_next(&hstat)) != NULL) + { + LWLockAcquire(&entry->lock, LW_EXCLUSIVE); + + if (state->my_entry == entry) + state->my_entry = NULL; + + /* Kind-specific accumulation */ + kind_info->per_backend_acc_cb(dst, entry); + + LWLockRelease(&entry->lock); + dshash_delete_current(&hstat); + } + + dshash_seq_term(&hstat); + + LWLockRelease(lock); +} + static void pgstat_build_snapshot(void) { @@ -1540,6 +2041,11 @@ pgstat_register_kind(PgStat_Kind kind, const PgStat_KindInfo *kind_info) errdetail("Custom cumulative statistics must be registered while initializing modules in \"%s\".", "shared_preload_libraries"))); + if (kind_info->per_backend_data_len != 0) + ereport(ERROR, + (errmsg("failed to register custom cumulative statistics \"%s\" with ID %u", kind_info->name, kind), + errdetail("Per-backend statistics are not supported for custom cumulative statistics."))); + /* * Check some data for fixed-numbered stats. */ diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c index 8511c25fd29..3b003f00245 100644 --- a/src/backend/utils/activity/pgstat_shmem.c +++ b/src/backend/utils/activity/pgstat_shmem.c @@ -168,6 +168,47 @@ StatsShmemRequest(void *arg) ); } +/* + * Create a dshash for each built-in kind that stores per-backend statistics. + * Derive the entry size and handle location from the kind metadata, just as + * attachment does. + */ +static void +pgstat_create_per_backend_hashes(dsa_area *dsa, PgStat_ShmemControl *ctl) +{ + dshash_parameters params = { + sizeof(ProcNumber), + 0, + dshash_memcmp, + dshash_memhash, + dshash_memcpy, + LWTRANCHE_PGSTATS_HASH + }; + + for (PgStat_Kind kind = PGSTAT_KIND_BUILTIN_MIN; + kind <= PGSTAT_KIND_BUILTIN_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + char *shared_struct; + dshash_table_handle *handle_ptr; + dshash_table *dsh; + + if (kind_info == NULL || kind_info->per_backend_data_len == 0) + continue; + + shared_struct = (char *) ctl + kind_info->shared_ctl_off; + handle_ptr = (dshash_table_handle *) + (shared_struct + kind_info->per_backend_hash_handle_off); + + params.entry_size = kind_info->per_backend_data_off + + kind_info->per_backend_data_len; + + dsh = dshash_create(dsa, ¶ms, NULL); + *handle_ptr = dshash_get_hash_table_handle(dsh); + dshash_detach(dsh); + } +} + /* * Initialize cumulative statistics system during startup */ @@ -210,6 +251,11 @@ StatsShmemInit(void *arg) /* lift limit set above */ dsa_set_size_limit(dsa, -1); + /* + * Create per-backend hashes while the local DSA reference is available. + */ + pgstat_create_per_backend_hashes(dsa, ctl); + /* * Postmaster will never access these again, thus free the local * dsa/dshash references. diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index b0a17691966..340244252c9 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -380,6 +380,26 @@ typedef struct PgStat_KindInfo */ void (*snapshot_cb) (void); + /* + * Per-backend dshash support for built-in fixed-numbered statistics kinds + * that also maintain per-backend entries in a dedicated dshash. If + * per_backend_data_len is non-zero, the generic infrastructure handles + * attach, fetch, accumulate, and snapshot pre-caching automatically. + * + * Each entry starts with PgStatShared_PerBackendEntry, followed by the + * kind-specific statistics payload. + */ + uint32 per_backend_data_off; /* offset of stats data in entry */ + uint32 per_backend_data_len; /* size of stats data in entry */ + /* offset of dshash_table_handle in shared struct */ + uint32 per_backend_hash_handle_off; + + /* + * Callback to accumulate one per-backend entry into a destination of the + * kind's statistics type. Called with the entry's content lock held. + */ + void (*per_backend_acc_cb) (void *dst, void *entry); + /* name of the kind of stats */ const char *const name; } PgStat_KindInfo; @@ -479,6 +499,17 @@ typedef struct PgStatShared_SLRU PgStat_SLRUStats stats[SLRU_NUM_ELEMENTS]; } PgStatShared_SLRU; +/* + * Common header for entries in per-backend statistics dshashes. The + * ProcNumber key must be the first field for dshash. + */ +typedef struct PgStatShared_PerBackendEntry +{ + ProcNumber key; + BackendType backend_type; + LWLock lock; +} PgStatShared_PerBackendEntry; + typedef struct PgStatShared_Wal { /* lock protects ->stats */ @@ -617,6 +648,9 @@ typedef struct PgStat_Snapshot PgStat_WalStats wal; + /* Per-backend snapshot hash */ + struct pgstat_per_backend_snapshot_hash *per_backend_stats; + /* * Data in snapshot for custom fixed-numbered statistics, indexed by * (PgStat_Kind - PGSTAT_KIND_CUSTOM_MIN). Each entry is allocated in @@ -691,6 +725,15 @@ extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *may_free); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +/* Generic per-backend helpers */ +extern dshash_table *pgstat_per_backend_attach(PgStat_Kind kind); +extern void *pgstat_lock_my_per_backend_entry(PgStat_Kind kind, bool nowait); +extern void pgstat_per_backend_snapshot(PgStat_Kind kind, dshash_table *hash, + void *snap); +extern void *pgstat_fetch_per_backend(PgStat_Kind kind, ProcNumber procnum); +extern void pgstat_acc_my_per_backend(PgStat_Kind kind, LWLock *lock); +extern void pgstat_acc_all_per_backend(PgStat_Kind kind, LWLock *lock); + /* * Functions in pgstat_archiver.c diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 85d989f395d..0b7dbc42e80 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2327,6 +2327,7 @@ PgStatShared_Function PgStatShared_HashEntry PgStatShared_IO PgStatShared_Lock +PgStatShared_PerBackendEntry PgStatShared_Relation PgStatShared_ReplSlot PgStatShared_SLRU @@ -2351,6 +2352,9 @@ PgStat_KindInfo PgStat_LocalState PgStat_Lock PgStat_LockEntry +PgStat_PerBackendLocalState +PgStat_PerBackendSnapshotEntry +PgStat_PerBackendSnapshotKey PgStat_PendingDroppedStatsItem PgStat_PendingIO PgStat_PendingLock -- 2.34.1
>From 753426d6e851c9942985e0bd90583c6de3deaaa8 Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot <[email protected]> Date: Wed, 29 Jul 2026 14:13:28 +0000 Subject: [PATCH v1 3/5] pgstat: move WAL statistics to new per-backend infrastructure PGSTAT_KIND_BACKEND stores each backend's WAL, Lock, and IO statistics together in one variable-numbered entry keyed by ProcNumber. Move WAL statistics into a dedicated ProcNumber keyed dshash associated with the fixed WAL statistics kind. The global WAL stats hold data for backends that have exited, while the dshash holds statistics for live backends. Build pg_stat_wal snapshots by copying the global stats and adding every live per-backend entry. Transfer the current process's entry into the global stats before deleting it at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown writes the statistics file. Remove WAL counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure in pg_stat_get_backend_wal(). This also makes WAL statistics available for auxiliary and shared memory-only workers. Update the documentation to describe the new behavior. Author: Bertrand Drouvot <[email protected]> Reviewed-by: Discussion: --- doc/src/sgml/monitoring.sgml | 7 +- src/backend/utils/activity/pgstat.c | 18 ++ src/backend/utils/activity/pgstat_backend.c | 75 -------- src/backend/utils/activity/pgstat_wal.c | 179 +++++++++++++++++--- src/backend/utils/adt/pgstatfuncs.c | 37 +++- src/include/pgstat.h | 3 +- src/include/utils/pgstat_internal.h | 22 ++- src/tools/pgindent/typedefs.list | 1 + 8 files changed, 232 insertions(+), 110 deletions(-) 3.6% doc/src/sgml/ 78.0% src/backend/utils/activity/ 10.2% src/backend/utils/adt/ 6.0% src/include/utils/ diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 099e9b6f4e9..bd5881beb61 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -5853,10 +5853,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage Returns WAL statistics about the backend with the specified process ID. The output fields are exactly the same as the ones in the <structname>pg_stat_wal</structname> view. - </para> - <para> - The function does not return WAL statistics for the checkpointer, - the background writer, the startup process and the autovacuum launcher. </para></entry> </row> @@ -5996,7 +5992,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage <listitem> <para> <literal>wal</literal>: Reset all the counters shown in the - <structname>pg_stat_wal</structname> view. + <structname>pg_stat_wal</structname> view, as well as per-backend + WAL statistics returned by <function>pg_stat_get_backend_wal</function>. </para> </listitem> <listitem> diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index efc7d427f22..19a98cdd8b7 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -535,6 +535,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .init_shmem_cb = pgstat_wal_init_shmem_cb, .reset_all_cb = pgstat_wal_reset_all_cb, .snapshot_cb = pgstat_wal_snapshot_cb, + + .per_backend_data_off = offsetof(PgStatShared_WalBackendEntry, stats), + .per_backend_data_len = sizeof(PgStat_WalStats), + .per_backend_hash_handle_off = offsetof(PgStatShared_Wal, backend_hash_handle), + .per_backend_acc_cb = pgstat_wal_per_backend_acc_cb, }, }; @@ -646,6 +651,9 @@ pgstat_before_server_shutdown(int code, Datum arg) */ if (code == 0) { + /* Transfer all live per-backend stats before writing the stats file. */ + pgstat_wal_acc_all_backends(); + pgStatLocal.shmem->is_shutdown = true; pgstat_write_statsfile(); } @@ -689,6 +697,9 @@ pgstat_shutdown_hook(int code, Datum arg) if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false)) pgstat_request_entry_refs_gc(); + /* Accumulate per-backend WAL stats into the global stats */ + pgstat_wal_acc_backend_cb(); + pgstat_detach_shmem(); #ifdef USE_ASSERT_CHECKING @@ -709,6 +720,13 @@ pgstat_initialize(void) pgstat_attach_shmem(); + /* + * NB: need to accept that there might be stats from an older backend that + * used the same proc number. Accumulate them into the global stats before + * we start using the entry. + */ + pgstat_wal_acc_backend_cb(); + /* * Create and cache per-backend statistics entries here. This also covers * processes that never call InitPostgres(), such as shared-memory-only diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index b736b2ccc6f..76b970a8c41 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -41,14 +41,6 @@ static PgStat_BackendPending PendingBackendStats; static bool backend_has_iostats = false; static bool backend_has_lockstats = false; -/* - * WAL usage counters saved from pgWalUsage at the previous call to - * pgstat_flush_backend(). This is used to calculate how much WAL usage - * happens between pgstat_flush_backend() calls, by subtracting the - * previous counters from the current ones. - */ -static WalUsage prevBackendWalUsage; - /* * Utility routines to report I/O stats for backends, kept here to avoid * exposing PendingBackendStats to the outside world. @@ -244,58 +236,6 @@ pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref) backend_has_iostats = false; } -/* - * To determine whether WAL usage happened. - */ -static inline bool -pgstat_backend_wal_have_pending(void) -{ - return (pgWalUsage.wal_records != prevBackendWalUsage.wal_records); -} - -/* - * Flush out locally pending backend WAL statistics. Locking is managed - * by the caller. - */ -static void -pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) -{ - PgStatShared_Backend *shbackendent; - PgStat_WalCounters *bktype_shstats; - WalUsage wal_usage_diff = {0}; - - /* - * This function can be called even if nothing at all has happened for WAL - * statistics. In this case, avoid unnecessarily modifying the stats - * entry. - */ - if (!pgstat_backend_wal_have_pending()) - return; - - shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats; - bktype_shstats = &shbackendent->stats.wal_counters; - - /* - * Calculate how much WAL usage counters were increased by subtracting the - * previous counters from the current ones. - */ - WalUsageAccumDiff(&wal_usage_diff, &pgWalUsage, &prevBackendWalUsage); - -#define WALSTAT_ACC(fld, var_to_add) \ - (bktype_shstats->fld += var_to_add.fld) - WALSTAT_ACC(wal_buffers_full, wal_usage_diff); - WALSTAT_ACC(wal_records, wal_usage_diff); - WALSTAT_ACC(wal_fpi, wal_usage_diff); - WALSTAT_ACC(wal_bytes, wal_usage_diff); - WALSTAT_ACC(wal_fpi_bytes, wal_usage_diff); -#undef WALSTAT_ACC - - /* - * Save the current counters for the subsequent calculation of WAL usage. - */ - prevBackendWalUsage = pgWalUsage; -} - /* * Flush out locally pending backend lock statistics. Locking is managed * by the caller. @@ -345,11 +285,6 @@ pgstat_flush_backend(bool nowait, uint32 flags) if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats) has_pending_data = true; - /* Some WAL data pending? */ - if ((flags & PGSTAT_BACKEND_FLUSH_WAL) && - pgstat_backend_wal_have_pending()) - has_pending_data = true; - /* Some lock data pending? */ if ((flags & PGSTAT_BACKEND_FLUSH_LOCK) && backend_has_lockstats) has_pending_data = true; @@ -366,9 +301,6 @@ pgstat_flush_backend(bool nowait, uint32 flags) if (flags & PGSTAT_BACKEND_FLUSH_IO) pgstat_flush_backend_entry_io(entry_ref); - if (flags & PGSTAT_BACKEND_FLUSH_WAL) - pgstat_flush_backend_entry_wal(entry_ref); - if (flags & PGSTAT_BACKEND_FLUSH_LOCK) pgstat_flush_backend_entry_lock(entry_ref); @@ -411,13 +343,6 @@ pgstat_create_backend(ProcNumber procnum) MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending)); backend_has_iostats = false; backend_has_lockstats = false; - - /* - * Initialize prevBackendWalUsage with pgWalUsage so that - * pgstat_backend_flush_cb() can calculate how much pgWalUsage counters - * are increased by subtracting prevBackendWalUsage from pgWalUsage. - */ - prevBackendWalUsage = pgWalUsage; } /* diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..ece4d91ed70 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -8,6 +8,11 @@ * storage implementation and the details about individual types of * statistics. * + * WAL statistics use a per-backend dshash to avoid double-counting. Each + * backend flushes WAL usage to its own entry in the dshash (keyed by + * ProcNumber). The global pg_stat_wal view aggregates the global stats + * (which holds stats from exited backends) plus all live per-backend entries. + * * Copyright (c) 2001-2026, PostgreSQL Global Development Group * * IDENTIFICATION @@ -52,7 +57,6 @@ pgstat_report_wal(bool force) /* flush wal stats */ (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); /* flush IO stats */ pgstat_flush_io(nowait); @@ -84,13 +88,15 @@ pgstat_wal_have_pending(void) * Calculate how much WAL usage counters have increased by subtracting the * previous counters from the current ones. * + * Flush WAL usage counters to the per-backend dshash entry. + * * If nowait is true, this function returns true if the lock could not be * acquired. Otherwise return false. */ bool pgstat_wal_flush_cb(bool nowait) { - PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; + PgStatShared_WalBackendEntry *entry; WalUsage wal_usage_diff = {0}; Assert(IsUnderPostmaster || !IsPostmasterEnvironment); @@ -105,19 +111,18 @@ pgstat_wal_flush_cb(bool nowait) return false; /* - * We don't update the WAL usage portion of the local WalStats elsewhere. * Calculate how much WAL usage counters were increased by subtracting the * previous counters from the current ones. */ WalUsageAccumDiff(&wal_usage_diff, &pgWalUsage, &prevWalUsage); - if (!nowait) - LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE); - else if (!LWLockConditionalAcquire(&stats_shmem->lock, LW_EXCLUSIVE)) - return true; + entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_WAL, nowait); + + if (entry == NULL) + return nowait; #define WALSTAT_ACC(fld, var_to_add) \ - (stats_shmem->stats.wal_counters.fld += var_to_add.fld) + (entry->stats.wal_counters.fld += var_to_add.fld) WALSTAT_ACC(wal_records, wal_usage_diff); WALSTAT_ACC(wal_fpi, wal_usage_diff); WALSTAT_ACC(wal_bytes, wal_usage_diff); @@ -125,7 +130,7 @@ pgstat_wal_flush_cb(bool nowait) WALSTAT_ACC(wal_buffers_full, wal_usage_diff); #undef WALSTAT_ACC - LWLockRelease(&stats_shmem->lock); + LWLockRelease(&entry->header.lock); /* * Save the current counters for the subsequent calculation of WAL usage. @@ -157,21 +162,157 @@ pgstat_wal_init_shmem_cb(void *stats) void pgstat_wal_reset_all_cb(TimestampTz ts) { - PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; + PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal; + dshash_seq_status hstat; + PgStatShared_WalBackendEntry *entry; + dshash_table *hash; + + hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL); + + /* + * Hold the kind lock while resetting both the global stats and live + * entries. Transfers hold the same lock, so pre-reset counters cannot be + * moved into the global stats after it is reset. + */ + LWLockAcquire(&shmem->lock, LW_EXCLUSIVE); + memset(&shmem->stats, 0, sizeof(shmem->stats)); + shmem->stats.stat_reset_timestamp = ts; + + /* Reset all per-backend entries */ + if (hash != NULL) + { + dshash_seq_init(&hstat, hash, true); + while ((entry = dshash_seq_next(&hstat)) != NULL) + { + LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE); + memset(&entry->stats.wal_counters, 0, sizeof(PgStat_WalCounters)); + entry->stats.stat_reset_timestamp = ts; + LWLockRelease(&entry->header.lock); + } + dshash_seq_term(&hstat); + } - LWLockAcquire(&stats_shmem->lock, LW_EXCLUSIVE); - memset(&stats_shmem->stats, 0, sizeof(stats_shmem->stats)); - stats_shmem->stats.stat_reset_timestamp = ts; - LWLockRelease(&stats_shmem->lock); + LWLockRelease(&shmem->lock); } +/* + * Build WAL stats snapshot by aggregating global stats and all live + * per-backend entries. + */ void pgstat_wal_snapshot_cb(void) { - PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; + PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal; + PgStat_WalStats *snap = &pgStatLocal.snapshot.wal; + dshash_table *hash; + + hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL); + + /* + * Prevent entries from moving to the global stats between copying it and + * scanning the per-backend hash. + */ + LWLockAcquire(&shmem->lock, LW_SHARED); + memcpy(snap, &shmem->stats, sizeof(PgStat_WalStats)); + + /* Add in all live per-backend entries */ + if (hash != NULL) + pgstat_per_backend_snapshot(PGSTAT_KIND_WAL, hash, snap); - LWLockAcquire(&stats_shmem->lock, LW_SHARED); - memcpy(&pgStatLocal.snapshot.wal, &stats_shmem->stats, - sizeof(pgStatLocal.snapshot.wal)); - LWLockRelease(&stats_shmem->lock); + LWLockRelease(&shmem->lock); +} + +/* Macro to accumulate WAL counters from src into dst */ +#define WAL_ACCUMULATE_COUNTERS(dst, src) \ +do { \ + (dst).wal_records += (src).wal_records; \ + (dst).wal_fpi += (src).wal_fpi; \ + (dst).wal_bytes += (src).wal_bytes; \ + (dst).wal_fpi_bytes += (src).wal_fpi_bytes; \ + (dst).wal_buffers_full += (src).wal_buffers_full; \ +} while (0) + +/* + * Accumulate one per-backend WAL entry into a snapshot or the global stats. + */ +void +pgstat_wal_per_backend_acc_cb(void *dst, void *entry) +{ + PgStat_WalStats *stats = dst; + PgStatShared_WalBackendEntry *e = (PgStatShared_WalBackendEntry *) entry; + + WAL_ACCUMULATE_COUNTERS(stats->wal_counters, e->stats.wal_counters); +} + +/* + * Accumulate a backend's WAL stats into the global stats, then + * remove the entry from the dshash. + * + * Called at backend exit after the final flush, or when a ProcNumber is + * being reused. + */ +void +pgstat_wal_acc_backend_cb(void) +{ + pgstat_acc_my_per_backend(PGSTAT_KIND_WAL, &pgStatLocal.shmem->wal.lock); +} + +/* + * Returns per-backend WAL statistics for the given ProcNumber. + */ +PgStat_WalStats * +pgstat_fetch_stat_backend_wal(ProcNumber procnum) +{ + return (PgStat_WalStats *) pgstat_fetch_per_backend(PGSTAT_KIND_WAL, procnum); +} + +/* + * Reset a backend's WAL stats. Accumulate the entry's counters into the + * global stats, then zero the stats and set the reset timestamp. + */ +void +pgstat_wal_reset_backend_cb(ProcNumber procnum, TimestampTz ts) +{ + PgStatShared_Wal *shmem = &pgStatLocal.shmem->wal; + dshash_table *hash; + PgStatShared_WalBackendEntry *entry; + + hash = pgstat_per_backend_attach(PGSTAT_KIND_WAL); + + if (hash == NULL) + return; + + LWLockAcquire(&shmem->lock, LW_EXCLUSIVE); + + entry = dshash_find(hash, &procnum, true); + + if (entry == NULL) + { + LWLockRelease(&shmem->lock); + return; + } + + LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE); + + /* Accumulate current stats into global before zeroing */ + WAL_ACCUMULATE_COUNTERS(shmem->stats.wal_counters, entry->stats.wal_counters); + + /* Zero stats and set reset timestamp */ + memset(&entry->stats.wal_counters, 0, sizeof(PgStat_WalCounters)); + entry->stats.stat_reset_timestamp = ts; + + LWLockRelease(&entry->header.lock); + dshash_release_lock(hash, entry); + LWLockRelease(&shmem->lock); +} + +/* + * Accumulate all per-backend WAL stats entries into the global stats and remove + * them. Called at clean server shutdown to ensure all flushed data is preserved + * in the stats file. + */ +void +pgstat_wal_acc_all_backends(void) +{ + pgstat_acc_all_per_backend(PGSTAT_KIND_WAL, &pgStatLocal.shmem->wal.lock); } diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 565d0e70768..5b0c819492a 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -1707,19 +1707,33 @@ Datum pg_stat_get_backend_wal(PG_FUNCTION_ARGS) { int pid; - PgStat_Backend *backend_stats; - PgStat_WalCounters bktype_stats; + PGPROC *proc; + ProcNumber procnum; + PgBackendStatus *beentry; + PgStat_WalStats *wal_stats; pid = PG_GETARG_INT32(0); - backend_stats = pgstat_fetch_stat_backend_by_pid(pid, NULL); - if (!backend_stats) + proc = BackendPidGetProc(pid); + + if (!proc) + proc = AuxiliaryPidGetProc(pid); + if (!proc) + PG_RETURN_NULL(); + + procnum = GetNumberFromPGProc(proc); + beentry = pgstat_get_beentry_by_proc_number(procnum); + + if (!beentry || beentry->st_procpid != pid) PG_RETURN_NULL(); - bktype_stats = backend_stats->wal_counters; + wal_stats = pgstat_fetch_stat_backend_wal(procnum); - /* save tuples with data from this PgStat_WalCounters */ - return (pg_stat_wal_build_tuple(bktype_stats, backend_stats->stat_reset_timestamp)); + if (!wal_stats) + PG_RETURN_NULL(); + + return (pg_stat_wal_build_tuple(wal_stats->wal_counters, + wal_stats->stat_reset_timestamp)); } /* @@ -2066,6 +2080,7 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS) PGPROC *proc; PgBackendStatus *beentry; ProcNumber procNumber; + TimestampTz ts; int backend_pid = PG_GETARG_INT32(0); proc = BackendPidGetProc(backend_pid); @@ -2087,6 +2102,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS) if (!pgstat_tracks_backend_bktype(beentry->st_backendType)) PG_RETURN_VOID(); + /* + * Accumulate the backend's WAL stats into the global stats, then zero the + * entry. + */ + ts = GetCurrentTimestamp(); + pgstat_wal_reset_backend_cb(procNumber, ts); + + /* Reset IO and Lock stats still in PGSTAT_KIND_BACKEND */ pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber); PG_RETURN_VOID(); diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 58a44857f13..0ade7f2f053 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -522,7 +522,6 @@ typedef struct PgStat_Backend { TimestampTz stat_reset_timestamp; PgStat_BktypeIO io_stats; - PgStat_WalCounters wal_counters; PgStat_PendingLock lock_stats; } PgStat_Backend; @@ -842,6 +841,8 @@ extern void pgstat_execute_transactional_drops(int ndrops, struct xl_xact_stats_ extern void pgstat_report_wal(bool force); extern PgStat_WalStats *pgstat_fetch_stat_wal(void); +extern PgStat_WalStats *pgstat_fetch_stat_backend_wal(ProcNumber procnum); +extern void pgstat_wal_reset_backend_cb(ProcNumber procnum, TimestampTz ts); /* diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 340244252c9..23ed4bdb4cc 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -510,11 +510,25 @@ typedef struct PgStatShared_PerBackendEntry LWLock lock; } PgStatShared_PerBackendEntry; +/* + * Per-backend entry for WAL statistics, stored in a dshash keyed by + * ProcNumber. + */ +typedef struct PgStatShared_WalBackendEntry +{ + PgStatShared_PerBackendEntry header; + PgStat_WalStats stats; +} PgStatShared_WalBackendEntry; + typedef struct PgStatShared_Wal { - /* lock protects ->stats */ LWLock lock; PgStat_WalStats stats; + + /* + * Per-backend dshash, keyed by ProcNumber. + */ + dshash_table_handle backend_hash_handle; } PgStatShared_Wal; @@ -749,9 +763,8 @@ extern void pgstat_archiver_snapshot_cb(void); /* flags for pgstat_flush_backend() */ #define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */ -#define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_LOCK (1 << 2) /* Flush lock statistics */ -#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL | PGSTAT_BACKEND_FLUSH_LOCK) +#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_LOCK) extern bool pgstat_flush_backend(bool nowait, uint32 flags); extern bool pgstat_backend_flush_cb(bool nowait); @@ -890,6 +903,9 @@ extern bool pgstat_wal_flush_cb(bool nowait); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); +extern void pgstat_wal_acc_backend_cb(void); +extern void pgstat_wal_acc_all_backends(void); +extern void pgstat_wal_per_backend_acc_cb(void *dst, void *entry); /* diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 0b7dbc42e80..5a4be198ada 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2333,6 +2333,7 @@ PgStatShared_ReplSlot PgStatShared_SLRU PgStatShared_Subscription PgStatShared_Wal +PgStatShared_WalBackendEntry PgStat_ArchiverStats PgStat_Backend PgStat_BackendPending -- 2.34.1
>From a105e6c596907d2def426ac05d1b0c828d3ac50a Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot <[email protected]> Date: Fri, 17 Jul 2026 13:25:10 +0000 Subject: [PATCH v1 4/5] pgstat: move Lock statistics to new per-backend infrastructure PGSTAT_KIND_BACKEND stores each backend's Lock and IO statistics together in one variable-numbered entry keyed by ProcNumber. Move Lock statistics into a dedicated ProcNumber keyed dshash associated with the fixed Lock statistics kind. The global Lock stats hold data for backends that have exited, while the dshash holds statistics for live backends. Build pg_stat_lock snapshots by copying the global stats and adding every live per-backend entry. Transfer the current process's entry into the global stats before deleting it at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown writes the statistics file. Remove Lock counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure in pg_stat_get_backend_lock(). This also makes Lock statistics available for auxiliary and shared memory-only workers. Update the documentation to describe the new behavior. Author: Bertrand Drouvot <[email protected]> Reviewed-by: Discussion: --- doc/src/sgml/monitoring.sgml | 8 +- src/backend/utils/activity/pgstat.c | 10 +- src/backend/utils/activity/pgstat_backend.c | 72 -------- src/backend/utils/activity/pgstat_lock.c | 183 +++++++++++++++++--- src/backend/utils/adt/pgstatfuncs.c | 34 +++- src/include/pgstat.h | 12 +- src/include/utils/pgstat_internal.h | 21 ++- src/tools/pgindent/typedefs.list | 1 + 8 files changed, 218 insertions(+), 123 deletions(-) 3.6% doc/src/sgml/ 74.4% src/backend/utils/activity/ 9.8% src/backend/utils/adt/ 6.9% src/include/utils/ 4.7% src/include/ diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index bd5881beb61..d3618e0b920 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -5834,10 +5834,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage Returns lock statistics about the backend with the specified process ID. The output fields are exactly the same as the ones in the <structname>pg_stat_lock</structname> view. - </para> - <para> - The function does not return lock statistics for the checkpointer, - the background writer, the startup process and the autovacuum launcher. </para></entry> </row> @@ -5974,7 +5970,9 @@ description | Waiting for a newly initialized WAL file to reach durable storage <listitem> <para> <literal>lock</literal>: Reset all the counters shown in the - <structname>pg_stat_lock</structname> view. + <structname>pg_stat_lock</structname> view, as well as per-backend + lock statistics returned by + <function>pg_stat_get_backend_lock</function>. </para> </listitem> <listitem> diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 19a98cdd8b7..2234bdb6857 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -500,6 +500,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .init_shmem_cb = pgstat_lock_init_shmem_cb, .reset_all_cb = pgstat_lock_reset_all_cb, .snapshot_cb = pgstat_lock_snapshot_cb, + + .per_backend_data_off = offsetof(PgStatShared_LockBackendEntry, stats), + .per_backend_data_len = sizeof(PgStat_Lock), + .per_backend_hash_handle_off = offsetof(PgStatShared_Lock, backend_hash_handle), + .per_backend_acc_cb = pgstat_lock_per_backend_acc_cb, }, [PGSTAT_KIND_SLRU] = { @@ -653,6 +658,7 @@ pgstat_before_server_shutdown(int code, Datum arg) { /* Transfer all live per-backend stats before writing the stats file. */ pgstat_wal_acc_all_backends(); + pgstat_lock_acc_all_backends(); pgStatLocal.shmem->is_shutdown = true; pgstat_write_statsfile(); @@ -697,8 +703,9 @@ pgstat_shutdown_hook(int code, Datum arg) if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false)) pgstat_request_entry_refs_gc(); - /* Accumulate per-backend WAL stats into the global stats */ + /* Accumulate per-backend stats into the global stats */ pgstat_wal_acc_backend_cb(); + pgstat_lock_acc_backend_cb(); pgstat_detach_shmem(); @@ -726,6 +733,7 @@ pgstat_initialize(void) * we start using the entry. */ pgstat_wal_acc_backend_cb(); + pgstat_lock_acc_backend_cb(); /* * Create and cache per-backend statistics entries here. This also covers diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index 76b970a8c41..8dda2cd88c1 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -39,7 +39,6 @@ */ static PgStat_BackendPending PendingBackendStats; static bool backend_has_iostats = false; -static bool backend_has_lockstats = false; /* * Utility routines to report I/O stats for backends, kept here to avoid @@ -79,39 +78,6 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, pgstat_report_fixed = true; } -/* - * Utility routines to report lock stats for backends, kept here to avoid - * exposing PendingBackendStats to the outside world. - */ -void -pgstat_count_backend_lock_waits(uint8 locktag_type, PgStat_Counter usecs) -{ - if (!pgstat_tracks_backend_bktype(MyBackendType)) - return; - - Assert(locktag_type <= LOCKTAG_LAST_TYPE); - - PendingBackendStats.pending_lock.stats[locktag_type].waits++; - PendingBackendStats.pending_lock.stats[locktag_type].wait_time += usecs; - - backend_has_lockstats = true; - pgstat_report_fixed = true; -} - -void -pgstat_count_backend_lock_fastpath_exceeded(uint8 locktag_type) -{ - if (!pgstat_tracks_backend_bktype(MyBackendType)) - return; - - Assert(locktag_type <= LOCKTAG_LAST_TYPE); - - PendingBackendStats.pending_lock.stats[locktag_type].fastpath_exceeded++; - - backend_has_lockstats = true; - pgstat_report_fixed = true; -} - /* * Returns statistics of a backend by proc number. */ @@ -236,36 +202,6 @@ pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref) backend_has_iostats = false; } -/* - * Flush out locally pending backend lock statistics. Locking is managed - * by the caller. - */ -static void -pgstat_flush_backend_entry_lock(PgStat_EntryRef *entry_ref) -{ - PgStatShared_Backend *shbackendent; - PgStat_PendingLock *bktype_shstats; - - if (!backend_has_lockstats) - return; - - shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats; - bktype_shstats = &shbackendent->stats.lock_stats; - - for (int i = 0; i <= LOCKTAG_LAST_TYPE; i++) - { -#define LOCKSTAT_ACC(fld) \ - (bktype_shstats->stats[i].fld += PendingBackendStats.pending_lock.stats[i].fld) - LOCKSTAT_ACC(waits); - LOCKSTAT_ACC(wait_time); - LOCKSTAT_ACC(fastpath_exceeded); -#undef LOCKSTAT_ACC - } - - MemSet(&PendingBackendStats.pending_lock, 0, sizeof(PgStat_PendingLock)); - backend_has_lockstats = false; -} - /* * Flush out locally pending backend statistics * @@ -285,10 +221,6 @@ pgstat_flush_backend(bool nowait, uint32 flags) if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats) has_pending_data = true; - /* Some lock data pending? */ - if ((flags & PGSTAT_BACKEND_FLUSH_LOCK) && backend_has_lockstats) - has_pending_data = true; - if (!has_pending_data) return false; @@ -301,9 +233,6 @@ pgstat_flush_backend(bool nowait, uint32 flags) if (flags & PGSTAT_BACKEND_FLUSH_IO) pgstat_flush_backend_entry_io(entry_ref); - if (flags & PGSTAT_BACKEND_FLUSH_LOCK) - pgstat_flush_backend_entry_lock(entry_ref); - pgstat_unlock_entry(entry_ref); return false; @@ -342,7 +271,6 @@ pgstat_create_backend(ProcNumber procnum) MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending)); backend_has_iostats = false; - backend_has_lockstats = false; } /* diff --git a/src/backend/utils/activity/pgstat_lock.c b/src/backend/utils/activity/pgstat_lock.c index c20c7599683..5aa380a6243 100644 --- a/src/backend/utils/activity/pgstat_lock.c +++ b/src/backend/utils/activity/pgstat_lock.c @@ -8,6 +8,12 @@ * access / storage implementation and the details about individual types * of statistics. * + * Lock statistics use a per-backend dshash to avoid double-counting. Each + * backend flushes lock stats (waits, wait_time, fastpath_exceeded per lock + * tag type) to its own entry in the dshash. The global pg_stat_lock view + * aggregates the global stats (which holds stats from exited backends) plus + * all live per-backend entries. + * * Copyright (c) 2021-2026, PostgreSQL Global Development Group * * IDENTIFICATION @@ -40,7 +46,7 @@ pgstat_lock_flush(bool nowait) } /* - * Flush out locally pending lock statistics + * Flush out locally pending lock statistics to the per-backend dshash entry. * * If no stats have been recorded, this function returns false. * @@ -50,31 +56,27 @@ pgstat_lock_flush(bool nowait) bool pgstat_lock_flush_cb(bool nowait) { - LWLock *lckstat_lock; - PgStatShared_Lock *shstats; + PgStatShared_LockBackendEntry *entry; if (!have_lockstats) return false; - shstats = &pgStatLocal.shmem->lock; - lckstat_lock = &shstats->lock; + entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_LOCK, nowait); - if (!nowait) - LWLockAcquire(lckstat_lock, LW_EXCLUSIVE); - else if (!LWLockConditionalAcquire(lckstat_lock, LW_EXCLUSIVE)) - return true; + if (entry == NULL) + return nowait; for (int i = 0; i <= LOCKTAG_LAST_TYPE; i++) { #define LOCKSTAT_ACC(fld) \ - (shstats->stats.stats[i].fld += PendingLockStats.stats[i].fld) + (entry->stats.stats[i].fld += PendingLockStats.stats[i].fld) LOCKSTAT_ACC(waits); LOCKSTAT_ACC(wait_time); LOCKSTAT_ACC(fastpath_exceeded); #undef LOCKSTAT_ACC } - LWLockRelease(lckstat_lock); + LWLockRelease(&entry->header.lock); memset(&PendingLockStats, 0, sizeof(PendingLockStats)); have_lockstats = false; @@ -93,28 +95,125 @@ pgstat_lock_init_shmem_cb(void *stats) void pgstat_lock_reset_all_cb(TimestampTz ts) { - LWLock *lckstat_lock = &pgStatLocal.shmem->lock.lock; + PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock; + dshash_seq_status hstat; + PgStatShared_LockBackendEntry *entry; + dshash_table *hash; - LWLockAcquire(lckstat_lock, LW_EXCLUSIVE); + hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK); - pgStatLocal.shmem->lock.stats.stat_reset_timestamp = ts; + /* + * Hold the kind lock while resetting both the global stats and live + * entries. Transfers hold the same lock, so pre-reset counters cannot be + * moved into the global stats after it is reset. + */ + LWLockAcquire(&shmem->lock, LW_EXCLUSIVE); + memset(&shmem->stats, 0, sizeof(shmem->stats)); + shmem->stats.stat_reset_timestamp = ts; - memset(pgStatLocal.shmem->lock.stats.stats, 0, - sizeof(pgStatLocal.shmem->lock.stats.stats)); + /* Reset all per-backend entries, since they contribute to the global view */ + if (hash != NULL) + { + dshash_seq_init(&hstat, hash, true); + while ((entry = dshash_seq_next(&hstat)) != NULL) + { + LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE); + memset(&entry->stats.stats, 0, sizeof(entry->stats.stats)); + entry->stats.stat_reset_timestamp = ts; + LWLockRelease(&entry->header.lock); + } + dshash_seq_term(&hstat); + } - LWLockRelease(lckstat_lock); + LWLockRelease(&shmem->lock); } +/* + * Build lock stats snapshot by aggregating global stats and all live + * per-backend entries. + */ void pgstat_lock_snapshot_cb(void) { - LWLock *lckstat_lock = &pgStatLocal.shmem->lock.lock; + PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock; + PgStat_Lock *snap = &pgStatLocal.snapshot.lock; + dshash_table *hash; + + hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK); + + /* + * Prevent entries from moving to the global stats between copying it and + * scanning the per-backend hash. + */ + LWLockAcquire(&shmem->lock, LW_SHARED); + memcpy(snap, &shmem->stats, sizeof(PgStat_Lock)); + + /* Add in all live per-backend entries */ + if (hash != NULL) + pgstat_per_backend_snapshot(PGSTAT_KIND_LOCK, hash, snap); + + LWLockRelease(&shmem->lock); +} + +/* + * Accumulate one per-backend lock entry into a snapshot or the global stats. + */ +void +pgstat_lock_per_backend_acc_cb(void *dst, void *entry) +{ + PgStat_Lock *stats = dst; + PgStatShared_LockBackendEntry *e = (PgStatShared_LockBackendEntry *) entry; + + for (int j = 0; j <= LOCKTAG_LAST_TYPE; j++) + { + stats->stats[j].waits += e->stats.stats[j].waits; + stats->stats[j].wait_time += e->stats.stats[j].wait_time; + stats->stats[j].fastpath_exceeded += e->stats.stats[j].fastpath_exceeded; + } +} + +/* Macro to accumulate lock counters from src into dst */ +#define LOCK_ACCUMULATE_COUNTERS(dst, src) \ +do { \ + for (int _i = 0; _i <= LOCKTAG_LAST_TYPE; _i++) \ + { \ + (dst)[_i].waits += (src)[_i].waits; \ + (dst)[_i].wait_time += (src)[_i].wait_time; \ + (dst)[_i].fastpath_exceeded += (src)[_i].fastpath_exceeded; \ + } \ +} while (0) - LWLockAcquire(lckstat_lock, LW_SHARED); +/* + * Accumulate a backend's lock stats into the global stats, then + * remove the entry from the dshash. + * + * Called at backend exit after the final flush, or when a ProcNumber is + * being reused. + */ +void +pgstat_lock_acc_backend_cb(void) +{ + pgstat_acc_my_per_backend(PGSTAT_KIND_LOCK, &pgStatLocal.shmem->lock.lock); +} - pgStatLocal.snapshot.lock = pgStatLocal.shmem->lock.stats; +/* + * Accumulate all per-backend lock stats entries into the global stats and remove + * them. Called at clean server shutdown to ensure all flushed data is preserved + * in the stats file. + */ +void +pgstat_lock_acc_all_backends(void) +{ + pgstat_acc_all_per_backend(PGSTAT_KIND_LOCK, &pgStatLocal.shmem->lock.lock); +} - LWLockRelease(lckstat_lock); +/* + * Returns per-backend lock statistics for the given ProcNumber. + */ +PgStat_Lock * +pgstat_fetch_stat_backend_lock(ProcNumber procnum) +{ + return (PgStat_Lock *) pgstat_fetch_per_backend(PGSTAT_KIND_LOCK, procnum); } /* @@ -131,8 +230,6 @@ pgstat_count_lock_fastpath_exceeded(uint8 locktag_type) PendingLockStats.stats[locktag_type].fastpath_exceeded++; have_lockstats = true; pgstat_report_fixed = true; - - pgstat_count_backend_lock_fastpath_exceeded(locktag_type); } /* @@ -149,6 +246,44 @@ pgstat_count_lock_waits(uint8 locktag_type, PgStat_Counter usecs) PendingLockStats.stats[locktag_type].wait_time += usecs; have_lockstats = true; pgstat_report_fixed = true; +} + +/* + * Reset a backend's lock stats. Accumulate the entry's counters into the + * global stats, then zero the stats and set the reset timestamp. + */ +void +pgstat_lock_reset_backend_cb(ProcNumber procnum, TimestampTz ts) +{ + PgStatShared_Lock *shmem = &pgStatLocal.shmem->lock; + dshash_table *hash; + PgStatShared_LockBackendEntry *entry; + + hash = pgstat_per_backend_attach(PGSTAT_KIND_LOCK); + + if (hash == NULL) + return; + + LWLockAcquire(&shmem->lock, LW_EXCLUSIVE); + + entry = dshash_find(hash, &procnum, true); + + if (entry == NULL) + { + LWLockRelease(&shmem->lock); + return; + } + + LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE); + + /* Accumulate current stats into global before zeroing */ + LOCK_ACCUMULATE_COUNTERS(shmem->stats.stats, entry->stats.stats); + + /* Zero stats and set reset timestamp */ + memset(&entry->stats.stats, 0, sizeof(entry->stats.stats)); + entry->stats.stat_reset_timestamp = ts; - pgstat_count_backend_lock_waits(locktag_type, usecs); + LWLockRelease(&entry->header.lock); + dshash_release_lock(hash, entry); + LWLockRelease(&shmem->lock); } diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 5b0c819492a..c26b50e0d28 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -1810,19 +1810,36 @@ pg_stat_get_backend_lock(PG_FUNCTION_ARGS) { int pid; ReturnSetInfo *rsinfo; - PgStat_Backend *backend_stats; + PGPROC *proc; + ProcNumber procnum; + PgBackendStatus *beentry; + PgStat_Lock *lock_stats; InitMaterializedSRF(fcinfo, 0); rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; pid = PG_GETARG_INT32(0); - backend_stats = pgstat_fetch_stat_backend_by_pid(pid, NULL); - if (!backend_stats) + proc = BackendPidGetProc(pid); + + if (!proc) + proc = AuxiliaryPidGetProc(pid); + if (!proc) return (Datum) 0; - pg_stat_lock_build_tuples(rsinfo, backend_stats->lock_stats.stats, - backend_stats->stat_reset_timestamp); + procnum = GetNumberFromPGProc(proc); + beentry = pgstat_get_beentry_by_proc_number(procnum); + + if (!beentry || beentry->st_procpid != pid) + return (Datum) 0; + + lock_stats = pgstat_fetch_stat_backend_lock(procnum); + + if (!lock_stats) + return (Datum) 0; + + pg_stat_lock_build_tuples(rsinfo, lock_stats->stats, + lock_stats->stat_reset_timestamp); return (Datum) 0; } @@ -2103,13 +2120,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS) PG_RETURN_VOID(); /* - * Accumulate the backend's WAL stats into the global stats, then zero the - * entry. + * Accumulate the backend's WAL and lock stats into the global stats, then + * zero the entries. */ ts = GetCurrentTimestamp(); pgstat_wal_reset_backend_cb(procNumber, ts); + pgstat_lock_reset_backend_cb(procNumber, ts); - /* Reset IO and Lock stats still in PGSTAT_KIND_BACKEND */ + /* Reset IO stats still in PGSTAT_KIND_BACKEND */ pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber); PG_RETURN_VOID(); diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 0ade7f2f053..3911ee56943 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -522,7 +522,6 @@ typedef struct PgStat_Backend { TimestampTz stat_reset_timestamp; PgStat_BktypeIO io_stats; - PgStat_PendingLock lock_stats; } PgStat_Backend; /* --------- @@ -535,12 +534,6 @@ typedef struct PgStat_BackendPending * Backend statistics store the same amount of IO data as PGSTAT_KIND_IO. */ PgStat_PendingIO pending_io; - - /* - * Backend statistics store the same amount of lock data as - * PGSTAT_KIND_LOCK. - */ - PgStat_PendingLock pending_lock; } PgStat_BackendPending; /* @@ -593,9 +586,6 @@ extern void pgstat_count_backend_io_op(IOObject io_object, IOOp io_op, uint32 cnt, uint64 bytes); -/* used by pgstat_lock.c for lock stats tracked in backends */ -extern void pgstat_count_backend_lock_waits(uint8 locktag_type, PgStat_Counter usecs); -extern void pgstat_count_backend_lock_fastpath_exceeded(uint8 locktag_type); extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber); extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid, @@ -652,6 +642,8 @@ extern void pgstat_count_lock_fastpath_exceeded(uint8 locktag_type); extern void pgstat_count_lock_waits(uint8 locktag_type, PgStat_Counter usecs); extern PgStat_Lock *pgstat_fetch_stat_lock(void); +extern PgStat_Lock *pgstat_fetch_stat_backend_lock(ProcNumber procnum); +extern void pgstat_lock_reset_backend_cb(ProcNumber procnum, TimestampTz ts); /* * Functions in pgstat_database.c diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 23ed4bdb4cc..9a765eca359 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -487,9 +487,12 @@ typedef struct PgStatShared_IO typedef struct PgStatShared_Lock { - /* lock protects ->stats */ + /* lock protects ->stats (global stats) */ LWLock lock; PgStat_Lock stats; + + /* Per-backend dshash, keyed by ProcNumber */ + dshash_table_handle backend_hash_handle; } PgStatShared_Lock; typedef struct PgStatShared_SLRU @@ -510,6 +513,16 @@ typedef struct PgStatShared_PerBackendEntry LWLock lock; } PgStatShared_PerBackendEntry; +/* + * Per-backend entry for lock statistics, stored in a dshash keyed by + * ProcNumber. + */ +typedef struct PgStatShared_LockBackendEntry +{ + PgStatShared_PerBackendEntry header; + PgStat_Lock stats; +} PgStatShared_LockBackendEntry; + /* * Per-backend entry for WAL statistics, stored in a dshash keyed by * ProcNumber. @@ -763,8 +776,7 @@ extern void pgstat_archiver_snapshot_cb(void); /* flags for pgstat_flush_backend() */ #define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */ -#define PGSTAT_BACKEND_FLUSH_LOCK (1 << 2) /* Flush lock statistics */ -#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_LOCK) +#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO) extern bool pgstat_flush_backend(bool nowait, uint32 flags); extern bool pgstat_backend_flush_cb(bool nowait); @@ -830,6 +842,9 @@ extern bool pgstat_lock_flush_cb(bool nowait); extern void pgstat_lock_init_shmem_cb(void *stats); extern void pgstat_lock_reset_all_cb(TimestampTz ts); extern void pgstat_lock_snapshot_cb(void); +extern void pgstat_lock_acc_backend_cb(void); +extern void pgstat_lock_acc_all_backends(void); +extern void pgstat_lock_per_backend_acc_cb(void *dst, void *entry); /* * Functions in pgstat_relation.c diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 5a4be198ada..d89fccf482d 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2327,6 +2327,7 @@ PgStatShared_Function PgStatShared_HashEntry PgStatShared_IO PgStatShared_Lock +PgStatShared_LockBackendEntry PgStatShared_PerBackendEntry PgStatShared_Relation PgStatShared_ReplSlot -- 2.34.1
>From c5e8cffb9c42c0c8ebfe1c49ad58a099b97a4566 Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot <[email protected]> Date: Fri, 17 Jul 2026 13:26:07 +0000 Subject: [PATCH v1 5/5] pgstat: move IO statistics to new per-backend infrastructure PGSTAT_KIND_BACKEND stores each backend's IO statistics in one variable-numbered entry keyed by ProcNumber. Move IO statistics into a dedicated ProcNumber keyed dshash associated with the fixed IO statistics kind. The global IO stats hold data for backends that have exited, while the dshash holds statistics for live backends. Build pg_stat_io snapshots by copying the global stats and adding every live per-backend entry. Transfer the current process's entry into the global stats before deleting it at process exit or ProcNumber reuse. Transfer all entries before a clean shutdown writes the statistics file. Remove IO counters from PGSTAT_KIND_BACKEND and make use of the new infrastructure in pg_stat_get_backend_io(). This also makes IO statistics available for auxiliary and shared memory-only workers. Update the documentation to describe the resulting new behavior. After WAL, Lock, and IO have moved, PGSTAT_KIND_BACKEND contains no data. Remove that kind and pgstat_backend.c together with their build entries, shared and pending structures, callbacks, flags, and call sites. Author: Bertrand Drouvot <[email protected]> Reviewed-by: Discussion: --- doc/src/sgml/monitoring.sgml | 9 +- src/backend/replication/walsender.c | 2 - src/backend/utils/activity/Makefile | 1 - src/backend/utils/activity/backend_status.c | 4 - src/backend/utils/activity/meson.build | 1 - src/backend/utils/activity/pgstat.c | 29 +- src/backend/utils/activity/pgstat_backend.c | 332 ------------------- src/backend/utils/activity/pgstat_io.c | 225 ++++++++++--- src/backend/utils/activity/pgstat_relation.c | 2 - src/backend/utils/activity/pgstat_shmem.c | 9 +- src/backend/utils/activity/pgstat_wal.c | 1 - src/backend/utils/adt/pgstatfuncs.c | 47 +-- src/include/pgstat.h | 53 +-- src/include/utils/pgstat_internal.h | 41 +-- src/include/utils/pgstat_kind.h | 15 +- src/test/regress/expected/stats.out | 40 ++- src/test/regress/sql/stats.sql | 11 +- src/tools/pgindent/typedefs.list | 5 +- 18 files changed, 282 insertions(+), 545 deletions(-) 70.6% src/backend/utils/activity/ 6.0% src/backend/utils/adt/ 6.3% src/include/utils/ 5.5% src/include/ 6.1% src/test/regress/expected/ 3.2% src/ diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index d3618e0b920..e83a68b686e 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -5797,12 +5797,6 @@ description | Waiting for a newly initialized WAL file to reach durable storage Returns I/O statistics about the backend with the specified process ID. The output fields are exactly the same as the ones in the <structname>pg_stat_io</structname> view. - </para> - <para> - The function does not return I/O statistics for the checkpointer, - the background writer, the startup process and the autovacuum launcher - as they are already visible in the <structname>pg_stat_io</structname> - view and there is only one of each. </para></entry> </row> @@ -5964,7 +5958,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage <listitem> <para> <literal>io</literal>: Reset all the counters shown in the - <structname>pg_stat_io</structname> view. + <structname>pg_stat_io</structname> view, as well as per-backend + I/O statistics returned by <function>pg_stat_get_backend_io</function>. </para> </listitem> <listitem> diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index c65dd324325..6d4c15cad4c 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -2083,7 +2083,6 @@ WalSndWaitForWal(XLogRecPtr loc) WALSENDER_STATS_FLUSH_INTERVAL)) { pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); last_flush = now; } @@ -3177,7 +3176,6 @@ WalSndLoop(WalSndSendDataCallback send_data) WALSENDER_STATS_FLUSH_INTERVAL)) { pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); last_flush = now; } diff --git a/src/backend/utils/activity/Makefile b/src/backend/utils/activity/Makefile index 5fed953c28a..015e6e05978 100644 --- a/src/backend/utils/activity/Makefile +++ b/src/backend/utils/activity/Makefile @@ -20,7 +20,6 @@ OBJS = \ backend_status.o \ pgstat.o \ pgstat_archiver.o \ - pgstat_backend.o \ pgstat_bgwriter.o \ pgstat_checkpointer.o \ pgstat_database.o \ diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c index d685fc5cd87..7088fc09651 100644 --- a/src/backend/utils/activity/backend_status.c +++ b/src/backend/utils/activity/backend_status.c @@ -458,10 +458,6 @@ pgstat_bestart_final(void) PGSTAT_END_WRITE_ACTIVITY(beentry); - /* Create the backend statistics entry */ - if (pgstat_tracks_backend_bktype(MyBackendType)) - pgstat_create_backend(MyProcNumber); - /* Update app name to current GUC setting */ if (application_name) pgstat_report_appname(application_name); diff --git a/src/backend/utils/activity/meson.build b/src/backend/utils/activity/meson.build index 470b5dac402..afbc5a625ea 100644 --- a/src/backend/utils/activity/meson.build +++ b/src/backend/utils/activity/meson.build @@ -5,7 +5,6 @@ backend_sources += files( 'backend_status.c', 'pgstat.c', 'pgstat_archiver.c', - 'pgstat_backend.c', 'pgstat_bgwriter.c', 'pgstat_checkpointer.c', 'pgstat_database.c', diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 2234bdb6857..6af5f2ce6ab 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -77,7 +77,6 @@ * * Each statistics kind is handled in a dedicated file: * - pgstat_archiver.c - * - pgstat_backend.c * - pgstat_bgwriter.c * - pgstat_checkpointer.c * - pgstat_database.c @@ -402,22 +401,6 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .reset_timestamp_cb = pgstat_subscription_reset_timestamp_cb, }, - [PGSTAT_KIND_BACKEND] = { - .name = "backend", - - .fixed_amount = false, - .write_to_file = false, - - .accessed_across_databases = true, - - .shared_size = sizeof(PgStatShared_Backend), - .shared_data_off = offsetof(PgStatShared_Backend, stats), - .shared_data_len = sizeof(((PgStatShared_Backend *) 0)->stats), - - .flush_static_cb = pgstat_backend_flush_cb, - .reset_timestamp_cb = pgstat_backend_reset_timestamp_cb, - }, - /* stats for fixed-numbered (mostly 1) objects */ [PGSTAT_KIND_ARCHIVER] = { @@ -483,6 +466,11 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .init_shmem_cb = pgstat_io_init_shmem_cb, .reset_all_cb = pgstat_io_reset_all_cb, .snapshot_cb = pgstat_io_snapshot_cb, + + .per_backend_data_off = offsetof(PgStatShared_IOBackendEntry, stats), + .per_backend_data_len = sizeof(PgStat_BackendIO), + .per_backend_hash_handle_off = offsetof(PgStatShared_IO, backend_hash_handle), + .per_backend_acc_cb = pgstat_io_per_backend_acc_cb, }, [PGSTAT_KIND_LOCK] = { @@ -659,6 +647,7 @@ pgstat_before_server_shutdown(int code, Datum arg) /* Transfer all live per-backend stats before writing the stats file. */ pgstat_wal_acc_all_backends(); pgstat_lock_acc_all_backends(); + pgstat_io_acc_all_backends(); pgStatLocal.shmem->is_shutdown = true; pgstat_write_statsfile(); @@ -699,13 +688,10 @@ pgstat_shutdown_hook(int code, Datum arg) Assert(dlist_is_empty(&pgStatPending)); dlist_init(&pgStatPending); - /* drop the backend stats entry */ - if (!pgstat_drop_entry(PGSTAT_KIND_BACKEND, InvalidOid, MyProcNumber, false)) - pgstat_request_entry_refs_gc(); - /* Accumulate per-backend stats into the global stats */ pgstat_wal_acc_backend_cb(); pgstat_lock_acc_backend_cb(); + pgstat_io_acc_backend_cb(); pgstat_detach_shmem(); @@ -734,6 +720,7 @@ pgstat_initialize(void) */ pgstat_wal_acc_backend_cb(); pgstat_lock_acc_backend_cb(); + pgstat_io_acc_backend_cb(); /* * Create and cache per-backend statistics entries here. This also covers diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c deleted file mode 100644 index 8dda2cd88c1..00000000000 --- a/src/backend/utils/activity/pgstat_backend.c +++ /dev/null @@ -1,332 +0,0 @@ -/* ------------------------------------------------------------------------- - * - * pgstat_backend.c - * Implementation of backend statistics. - * - * This file contains the implementation of backend statistics. It is kept - * separate from pgstat.c to enforce the line between the statistics access / - * storage implementation and the details about individual types of - * statistics. - * - * This statistics kind uses a proc number as object ID for the hash table - * of pgstats. Entries are created each time a process is spawned, and are - * dropped when the process exits. These are not written to the pgstats file - * on disk. Pending statistics are managed without direct interactions with - * PgStat_EntryRef->pending, relying on PendingBackendStats instead so as it - * is possible to report data within critical sections. - * - * Copyright (c) 2001-2026, PostgreSQL Global Development Group - * - * IDENTIFICATION - * src/backend/utils/activity/pgstat_backend.c - * ------------------------------------------------------------------------- - */ - -#include "postgres.h" - -#include "access/xlog.h" -#include "executor/instrument.h" -#include "storage/bufmgr.h" -#include "storage/proc.h" -#include "storage/procarray.h" -#include "utils/memutils.h" -#include "utils/pgstat_internal.h" - -/* - * Backend statistics counts waiting to be flushed out. These counters may be - * reported within critical sections so we use static memory in order to avoid - * memory allocation. - */ -static PgStat_BackendPending PendingBackendStats; -static bool backend_has_iostats = false; - -/* - * Utility routines to report I/O stats for backends, kept here to avoid - * exposing PendingBackendStats to the outside world. - */ -void -pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, - IOOp io_op, instr_time io_time) -{ - Assert(track_io_timing || track_wal_io_timing); - - if (!pgstat_tracks_backend_bktype(MyBackendType)) - return; - - Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op)); - - INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], - io_time); - - backend_has_iostats = true; - pgstat_report_fixed = true; -} - -void -pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, - IOOp io_op, uint32 cnt, uint64 bytes) -{ - if (!pgstat_tracks_backend_bktype(MyBackendType)) - return; - - Assert(pgstat_tracks_io_op(MyBackendType, io_object, io_context, io_op)); - - PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; - PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; - - backend_has_iostats = true; - pgstat_report_fixed = true; -} - -/* - * Returns statistics of a backend by proc number. - */ -PgStat_Backend * -pgstat_fetch_stat_backend(ProcNumber procNumber) -{ - PgStat_Backend *backend_entry; - - backend_entry = (PgStat_Backend *) pgstat_fetch_entry(PGSTAT_KIND_BACKEND, - InvalidOid, procNumber, - NULL); - - return backend_entry; -} - -/* - * Returns statistics of a backend by pid. - * - * This routine includes sanity checks to ensure that the backend exists and - * is running. "bktype" can be optionally defined to return the BackendType - * of the backend whose statistics are returned. - */ -PgStat_Backend * -pgstat_fetch_stat_backend_by_pid(int pid, BackendType *bktype) -{ - PGPROC *proc; - PgBackendStatus *beentry; - ProcNumber procNumber; - PgStat_Backend *backend_stats; - - proc = BackendPidGetProc(pid); - if (bktype) - *bktype = B_INVALID; - - /* this could be an auxiliary process */ - if (!proc) - proc = AuxiliaryPidGetProc(pid); - - if (!proc) - return NULL; - - procNumber = GetNumberFromPGProc(proc); - - beentry = pgstat_get_beentry_by_proc_number(procNumber); - if (!beentry) - return NULL; - - /* check if the backend type tracks statistics */ - if (!pgstat_tracks_backend_bktype(beentry->st_backendType)) - return NULL; - - /* if PID does not match, leave */ - if (beentry->st_procpid != pid) - return NULL; - - if (bktype) - *bktype = beentry->st_backendType; - - /* - * Retrieve the entry. Note that "beentry" may be freed depending on the - * value of stats_fetch_consistency, so do not access it from this point. - */ - backend_stats = pgstat_fetch_stat_backend(procNumber); - if (!backend_stats) - { - if (bktype) - *bktype = B_INVALID; - return NULL; - } - - return backend_stats; -} - -/* - * Flush out locally pending backend IO statistics. Locking is managed - * by the caller. - */ -static void -pgstat_flush_backend_entry_io(PgStat_EntryRef *entry_ref) -{ - PgStatShared_Backend *shbackendent; - PgStat_BktypeIO *bktype_shstats; - PgStat_PendingIO pending_io; - - /* - * This function can be called even if nothing at all has happened for IO - * statistics. In this case, avoid unnecessarily modifying the stats - * entry. - */ - if (!backend_has_iostats) - return; - - shbackendent = (PgStatShared_Backend *) entry_ref->shared_stats; - bktype_shstats = &shbackendent->stats.io_stats; - pending_io = PendingBackendStats.pending_io; - - for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++) - { - for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++) - { - for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++) - { - instr_time time; - - bktype_shstats->counts[io_object][io_context][io_op] += - pending_io.counts[io_object][io_context][io_op]; - bktype_shstats->bytes[io_object][io_context][io_op] += - pending_io.bytes[io_object][io_context][io_op]; - time = pending_io.pending_times[io_object][io_context][io_op]; - - bktype_shstats->times[io_object][io_context][io_op] += - INSTR_TIME_GET_MICROSEC(time); - } - } - } - - /* - * Clear out the statistics buffer, so it can be re-used. - */ - MemSet(&PendingBackendStats.pending_io, 0, sizeof(PgStat_PendingIO)); - - backend_has_iostats = false; -} - -/* - * Flush out locally pending backend statistics - * - * "flags" parameter controls which statistics to flush. Returns true - * if some statistics could not be flushed due to lock contention. - */ -bool -pgstat_flush_backend(bool nowait, uint32 flags) -{ - PgStat_EntryRef *entry_ref; - bool has_pending_data = false; - - if (!pgstat_tracks_backend_bktype(MyBackendType)) - return false; - - /* Some IO data pending? */ - if ((flags & PGSTAT_BACKEND_FLUSH_IO) && backend_has_iostats) - has_pending_data = true; - - if (!has_pending_data) - return false; - - entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid, - MyProcNumber, nowait); - if (!entry_ref) - return true; - - /* Flush requested statistics */ - if (flags & PGSTAT_BACKEND_FLUSH_IO) - pgstat_flush_backend_entry_io(entry_ref); - - pgstat_unlock_entry(entry_ref); - - return false; -} - -/* - * Callback to flush out locally pending backend statistics. - * - * If some stats could not be flushed due to lock contention, return true. - */ -bool -pgstat_backend_flush_cb(bool nowait) -{ - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); -} - -/* - * Create backend statistics entry for proc number. - */ -void -pgstat_create_backend(ProcNumber procnum) -{ - PgStat_EntryRef *entry_ref; - PgStatShared_Backend *shstatent; - - entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_BACKEND, InvalidOid, - procnum, false); - shstatent = (PgStatShared_Backend *) entry_ref->shared_stats; - - /* - * NB: need to accept that there might be stats from an older backend, - * e.g. if we previously used this proc number. - */ - memset(&shstatent->stats, 0, sizeof(shstatent->stats)); - pgstat_unlock_entry(entry_ref); - - MemSet(&PendingBackendStats, 0, sizeof(PgStat_BackendPending)); - backend_has_iostats = false; -} - -/* - * Backend statistics are not collected for all BackendTypes. - * - * The following BackendTypes do not participate in the backend stats - * subsystem: - * - The same and for the same reasons as in pgstat_tracks_io_bktype(). - * - B_BG_WRITER, B_CHECKPOINTER, B_STARTUP and B_AUTOVAC_LAUNCHER because their - * I/O stats are already visible in pg_stat_io and there is only one of those. - * - * Function returns true if BackendType participates in the backend stats - * subsystem and false if it does not. - * - * When adding a new BackendType, also consider adding relevant restrictions to - * pgstat_tracks_io_object() and pgstat_tracks_io_op(). - */ -bool -pgstat_tracks_backend_bktype(BackendType bktype) -{ - /* - * List every type so that new backend types trigger a warning about - * needing to adjust this switch. - */ - switch (bktype) - { - case B_INVALID: - case B_AUTOVAC_LAUNCHER: - case B_DEAD_END_BACKEND: - case B_ARCHIVER: - case B_LOGGER: - case B_BG_WRITER: - case B_CHECKPOINTER: - case B_IO_WORKER: - case B_STARTUP: - case B_DATACHECKSUMSWORKER_LAUNCHER: - case B_DATACHECKSUMSWORKER_WORKER: - return false; - - case B_AUTOVAC_WORKER: - case B_BACKEND: - case B_BG_WORKER: - case B_STANDALONE_BACKEND: - case B_SLOTSYNC_WORKER: - case B_WAL_RECEIVER: - case B_WAL_SENDER: - case B_WAL_SUMMARIZER: - case B_WAL_WRITER: - return true; - } - - return false; -} - -void -pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts) -{ - ((PgStatShared_Backend *) header)->stats.stat_reset_timestamp = ts; -} diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 8ec1aad5078..5e65bc5cb4b 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -7,6 +7,11 @@ * from pgstat.c to enforce the line between the statistics access / storage * implementation and the details about individual types of statistics. * + * IO statistics use a per-backend dshash to avoid double-counting. Each + * process flushes IO stats to its own entry in the dshash (keyed by + * ProcNumber). The global pg_stat_io view aggregates the global stats + * (which holds stats from exited processes) plus all live per-backend entries. + * * Copyright (c) 2021-2026, PostgreSQL Global Development Group * * IDENTIFICATION @@ -68,9 +73,6 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, PendingIOStats.counts[io_object][io_context][io_op] += cnt; PendingIOStats.bytes[io_object][io_context][io_op] += bytes; - /* Add the per-backend counts */ - pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); - have_iostats = true; pgstat_report_fixed = true; } @@ -143,10 +145,6 @@ pgstat_count_io_op_time(IOObject io_object, IOContext io_context, IOOp io_op, INSTR_TIME_ADD(PendingIOStats.pending_times[io_object][io_context][io_op], io_time); - - /* Add the per-backend count */ - pgstat_count_backend_io_op_time(io_object, io_context, io_op, - io_time); } pgstat_count_io_op(io_object, io_context, io_op, cnt, bytes); @@ -170,7 +168,7 @@ pgstat_flush_io(bool nowait) } /* - * Flush out locally pending IO statistics + * Flush out locally pending IO statistics to the per-backend dshash entry. * * If no stats have been recorded, this function returns false. * @@ -180,20 +178,18 @@ pgstat_flush_io(bool nowait) bool pgstat_io_flush_cb(bool nowait) { - LWLock *bktype_lock; + PgStatShared_IOBackendEntry *entry; PgStat_BktypeIO *bktype_shstats; if (!have_iostats) return false; - bktype_lock = &pgStatLocal.shmem->io.locks[MyBackendType]; - bktype_shstats = - &pgStatLocal.shmem->io.stats.stats[MyBackendType]; + entry = pgstat_lock_my_per_backend_entry(PGSTAT_KIND_IO, nowait); - if (!nowait) - LWLockAcquire(bktype_lock, LW_EXCLUSIVE); - else if (!LWLockConditionalAcquire(bktype_lock, LW_EXCLUSIVE)) - return true; + if (entry == NULL) + return nowait; + + bktype_shstats = &entry->stats.stats; for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++) { @@ -217,12 +213,9 @@ pgstat_io_flush_cb(bool nowait) } } - Assert(pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType)); - - LWLockRelease(bktype_lock); + LWLockRelease(&entry->header.lock); memset(&PendingIOStats, 0, sizeof(PendingIOStats)); - have_iostats = false; return false; @@ -271,55 +264,70 @@ pgstat_io_init_shmem_cb(void *stats) { PgStatShared_IO *stat_shmem = (PgStatShared_IO *) stats; - for (int i = 0; i < BACKEND_NUM_TYPES; i++) - LWLockInitialize(&stat_shmem->locks[i], LWTRANCHE_PGSTATS_DATA); + LWLockInitialize(&stat_shmem->lock, LWTRANCHE_PGSTATS_DATA); } void pgstat_io_reset_all_cb(TimestampTz ts) { - for (int i = 0; i < BACKEND_NUM_TYPES; i++) - { - LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i]; - PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i]; + PgStatShared_IO *shmem = &pgStatLocal.shmem->io; + dshash_seq_status hstat; + PgStatShared_IOBackendEntry *entry; + dshash_table *hash; - LWLockAcquire(bktype_lock, LW_EXCLUSIVE); + hash = pgstat_per_backend_attach(PGSTAT_KIND_IO); - /* - * Use the lock in the first BackendType's PgStat_BktypeIO to protect - * the reset timestamp as well. - */ - if (i == 0) - pgStatLocal.shmem->io.stats.stat_reset_timestamp = ts; + /* + * Hold the kind lock while resetting both the global stats and live + * entries. Transfers hold the same lock, so pre-reset counters cannot be + * moved into the global stats after it is reset. + */ + LWLockAcquire(&shmem->lock, LW_EXCLUSIVE); + memset(&shmem->stats, 0, sizeof(shmem->stats)); + shmem->stats.stat_reset_timestamp = ts; - memset(bktype_shstats, 0, sizeof(*bktype_shstats)); - LWLockRelease(bktype_lock); + /* Reset all per-backend entries */ + if (hash != NULL) + { + dshash_seq_init(&hstat, hash, true); + while ((entry = dshash_seq_next(&hstat)) != NULL) + { + LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE); + memset(&entry->stats.stats, 0, sizeof(PgStat_BktypeIO)); + entry->stats.stat_reset_timestamp = ts; + LWLockRelease(&entry->header.lock); + } + dshash_seq_term(&hstat); } + + LWLockRelease(&shmem->lock); } +/* + * Build IO stats snapshot by aggregating global stats and all live + * per-backend entries. + */ void pgstat_io_snapshot_cb(void) { - for (int i = 0; i < BACKEND_NUM_TYPES; i++) - { - LWLock *bktype_lock = &pgStatLocal.shmem->io.locks[i]; - PgStat_BktypeIO *bktype_shstats = &pgStatLocal.shmem->io.stats.stats[i]; - PgStat_BktypeIO *bktype_snap = &pgStatLocal.snapshot.io.stats[i]; + PgStatShared_IO *shmem = &pgStatLocal.shmem->io; + PgStat_IO *snap = &pgStatLocal.snapshot.io; + dshash_table *hash; - LWLockAcquire(bktype_lock, LW_SHARED); + hash = pgstat_per_backend_attach(PGSTAT_KIND_IO); - /* - * Use the lock in the first BackendType's PgStat_BktypeIO to protect - * the reset timestamp as well. - */ - if (i == 0) - pgStatLocal.snapshot.io.stat_reset_timestamp = - pgStatLocal.shmem->io.stats.stat_reset_timestamp; + /* + * Prevent entries from moving to the global stats between copying it and + * scanning the per-backend hash. + */ + LWLockAcquire(&shmem->lock, LW_SHARED); + memcpy(snap, &shmem->stats, sizeof(PgStat_IO)); - /* using struct assignment due to better type safety */ - *bktype_snap = *bktype_shstats; - LWLockRelease(bktype_lock); - } + /* Add in all live per-backend entries */ + if (hash != NULL) + pgstat_per_backend_snapshot(PGSTAT_KIND_IO, hash, snap); + + LWLockRelease(&shmem->lock); } /* @@ -581,3 +589,116 @@ pgstat_tracks_io_op(BackendType bktype, IOObject io_object, return true; } + +/* + * Accumulate IO counters from src into dst. + */ +static inline void +pgstat_io_accumulate_counters(PgStat_BktypeIO *dst, const PgStat_BktypeIO *src) +{ + for (int io_object = 0; io_object < IOOBJECT_NUM_TYPES; io_object++) + { + for (int io_context = 0; io_context < IOCONTEXT_NUM_TYPES; io_context++) + { + for (int io_op = 0; io_op < IOOP_NUM_TYPES; io_op++) + { + dst->counts[io_object][io_context][io_op] += + src->counts[io_object][io_context][io_op]; + dst->bytes[io_object][io_context][io_op] += + src->bytes[io_object][io_context][io_op]; + dst->times[io_object][io_context][io_op] += + src->times[io_object][io_context][io_op]; + } + } + } +} + +/* + * Accumulate one per-backend IO entry into a snapshot or the global stats. + */ +void +pgstat_io_per_backend_acc_cb(void *dst, void *entry) +{ + PgStat_IO *stats = dst; + PgStatShared_IOBackendEntry *e = (PgStatShared_IOBackendEntry *) entry; + BackendType bktype = e->header.backend_type; + + if (bktype == B_INVALID) + return; + + pgstat_io_accumulate_counters(&stats->stats[bktype], &e->stats.stats); +} + +/* + * Accumulate a backend's IO stats into the global stats, then remove the + * entry from the dshash. + * + * Called at backend exit after the final flush, or when a ProcNumber is + * being reused. + */ +void +pgstat_io_acc_backend_cb(void) +{ + pgstat_acc_my_per_backend(PGSTAT_KIND_IO, &pgStatLocal.shmem->io.lock); +} + +/* + * Accumulate all remaining per-backend IO stats entries into the global stats + * and remove them. Called at clean server shutdown to ensure all flushed data + * is preserved in the stats file. + */ +void +pgstat_io_acc_all_backends(void) +{ + pgstat_acc_all_per_backend(PGSTAT_KIND_IO, &pgStatLocal.shmem->io.lock); +} + +/* + * Returns per-backend IO statistics for the given ProcNumber. + */ +PgStat_BackendIO * +pgstat_fetch_stat_backend_io(ProcNumber procnum) +{ + return (PgStat_BackendIO *) pgstat_fetch_per_backend(PGSTAT_KIND_IO, procnum); +} + +/* + * Reset a backend's IO stats. Accumulate the entry's counters into the + * global stats, then zero the stats and set the reset timestamp. + */ +void +pgstat_io_reset_backend_cb(ProcNumber procnum, TimestampTz ts) +{ + PgStatShared_IO *shmem = &pgStatLocal.shmem->io; + dshash_table *hash; + PgStatShared_IOBackendEntry *entry; + + hash = pgstat_per_backend_attach(PGSTAT_KIND_IO); + + if (hash == NULL) + return; + + LWLockAcquire(&shmem->lock, LW_EXCLUSIVE); + + entry = dshash_find(hash, &procnum, true); + + if (entry == NULL) + { + LWLockRelease(&shmem->lock); + return; + } + + LWLockAcquire(&entry->header.lock, LW_EXCLUSIVE); + + /* Accumulate current stats into global before zeroing */ + pgstat_io_accumulate_counters(&shmem->stats.stats[entry->header.backend_type], + &entry->stats.stats); + + /* Zero stats and set reset timestamp */ + memset(&entry->stats, 0, sizeof(entry->stats)); + entry->stats.stat_reset_timestamp = ts; + + LWLockRelease(&entry->header.lock); + dshash_release_lock(hash, entry); + LWLockRelease(&shmem->lock); +} diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index 04f2eb21d0b..8080bc1cbba 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -269,7 +269,6 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * VACUUM command has processed all tables and committed. */ pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); } /* @@ -364,7 +363,6 @@ pgstat_report_analyze(Relation rel, /* see pgstat_report_vacuum() */ pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); } /* diff --git a/src/backend/utils/activity/pgstat_shmem.c b/src/backend/utils/activity/pgstat_shmem.c index 3b003f00245..7730dd95766 100644 --- a/src/backend/utils/activity/pgstat_shmem.c +++ b/src/backend/utils/activity/pgstat_shmem.c @@ -118,12 +118,11 @@ pgstat_dsa_init_size(void) /* * The dshash header / initial buckets array needs to fit into "plain" * shared memory, but it's beneficial to not need dsm segments - * immediately. A size of 256kB seems works well and is not - * disproportional compared to other constant sized shared memory - * allocations. NB: To avoid DSMs further, the user can configure - * min_dynamic_shared_memory. + * immediately. A size of 1MB works well and is not disproportional + * compared to other constant sized shared memory allocations. NB: To + * avoid DSMs further, the user can configure min_dynamic_shared_memory. */ - sz = 256 * 1024; + sz = 1024 * 1024; Assert(dsa_minimum_size() <= sz); return MAXALIGN(sz); } diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index ece4d91ed70..99fd20a04c2 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -60,7 +60,6 @@ pgstat_report_wal(bool force) /* flush IO stats */ pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); } /* diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index c26b50e0d28..d8517abed2e 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -1609,32 +1609,47 @@ Datum pg_stat_get_backend_io(PG_FUNCTION_ARGS) { ReturnSetInfo *rsinfo; - BackendType bktype; int pid; - PgStat_Backend *backend_stats; - PgStat_BktypeIO *bktype_stats; + PGPROC *proc; + ProcNumber procnum; + PgBackendStatus *beentry; + BackendType bktype; + PgStat_BackendIO *backend_io; InitMaterializedSRF(fcinfo, 0); rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; pid = PG_GETARG_INT32(0); - backend_stats = pgstat_fetch_stat_backend_by_pid(pid, &bktype); - if (!backend_stats) + proc = BackendPidGetProc(pid); + + if (!proc) + proc = AuxiliaryPidGetProc(pid); + if (!proc) + return (Datum) 0; + + procnum = GetNumberFromPGProc(proc); + beentry = pgstat_get_beentry_by_proc_number(procnum); + + if (!beentry || beentry->st_procpid != pid) return (Datum) 0; - bktype_stats = &backend_stats->io_stats; + bktype = beentry->st_backendType; + backend_io = pgstat_fetch_stat_backend_io(procnum); + + if (!backend_io) + return (Datum) 0; /* * In Assert builds, we can afford an extra loop through all of the * counters (in pg_stat_io_build_tuples()), checking that only expected * stats are non-zero, since it keeps the non-Assert code cleaner. */ - Assert(pgstat_bktype_io_stats_valid(bktype_stats, bktype)); + Assert(pgstat_bktype_io_stats_valid(&backend_io->stats, bktype)); + + pg_stat_io_build_tuples(rsinfo, &backend_io->stats, bktype, + backend_io->stat_reset_timestamp); - /* save tuples with data from this PgStat_BktypeIO */ - pg_stat_io_build_tuples(rsinfo, bktype_stats, bktype, - backend_stats->stat_reset_timestamp); return (Datum) 0; } @@ -2115,20 +2130,14 @@ pg_stat_reset_backend_stats(PG_FUNCTION_ARGS) if (!beentry) PG_RETURN_VOID(); - /* Check if the backend type tracks statistics */ - if (!pgstat_tracks_backend_bktype(beentry->st_backendType)) - PG_RETURN_VOID(); - /* - * Accumulate the backend's WAL and lock stats into the global stats, then - * zero the entries. + * Accumulate the backend's WAL, lock and IO stats into the global stats, + * then zero the entries. */ ts = GetCurrentTimestamp(); pgstat_wal_reset_backend_cb(procNumber, ts); pgstat_lock_reset_backend_cb(procNumber, ts); - - /* Reset IO stats still in PGSTAT_KIND_BACKEND */ - pgstat_reset(PGSTAT_KIND_BACKEND, InvalidOid, procNumber); + pgstat_io_reset_backend_cb(procNumber, ts); PG_RETURN_VOID(); } diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 3911ee56943..5d63d19673f 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -218,7 +218,7 @@ typedef struct PgStat_TableXactStatus * ------------------------------------------------------------ */ -#define PGSTAT_FILE_FORMAT_ID 0x01A5BCBC +#define PGSTAT_FILE_FORMAT_ID 0x01A5BCBD typedef struct PgStat_ArchiverStats { @@ -333,6 +333,12 @@ typedef struct PgStat_BktypeIO PgStat_Counter times[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES]; } PgStat_BktypeIO; +typedef struct PgStat_BackendIO +{ + TimestampTz stat_reset_timestamp; + PgStat_BktypeIO stats; +} PgStat_BackendIO; + typedef struct PgStat_PendingIO { uint64 bytes[IOOBJECT_NUM_TYPES][IOCONTEXT_NUM_TYPES][IOOP_NUM_TYPES]; @@ -514,28 +520,6 @@ typedef struct PgStat_WalStats TimestampTz stat_reset_timestamp; } PgStat_WalStats; -/* ------- - * PgStat_Backend Backend statistics - * ------- - */ -typedef struct PgStat_Backend -{ - TimestampTz stat_reset_timestamp; - PgStat_BktypeIO io_stats; -} PgStat_Backend; - -/* --------- - * PgStat_BackendPending Non-flushed backend stats. - * --------- - */ -typedef struct PgStat_BackendPending -{ - /* - * Backend statistics store the same amount of IO data as PGSTAT_KIND_IO. - */ - PgStat_PendingIO pending_io; -} PgStat_BackendPending; - /* * Functions in pgstat.c */ @@ -572,27 +556,6 @@ extern bool pgstat_have_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_report_archiver(const char *xlog, bool failed); extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void); -/* - * Functions in pgstat_backend.c - */ - -/* used by pgstat_io.c for I/O stats tracked in backends */ -extern void pgstat_count_backend_io_op_time(IOObject io_object, - IOContext io_context, - IOOp io_op, - instr_time io_time); -extern void pgstat_count_backend_io_op(IOObject io_object, - IOContext io_context, - IOOp io_op, uint32 cnt, - uint64 bytes); - - -extern PgStat_Backend *pgstat_fetch_stat_backend(ProcNumber procNumber); -extern PgStat_Backend *pgstat_fetch_stat_backend_by_pid(int pid, - BackendType *bktype); -extern bool pgstat_tracks_backend_bktype(BackendType bktype); -extern void pgstat_create_backend(ProcNumber procnum); - /* * Functions in pgstat_bgwriter.c */ @@ -623,6 +586,8 @@ extern void pgstat_count_io_op_time(IOObject io_object, IOContext io_context, uint32 cnt, uint64 bytes); extern PgStat_IO *pgstat_fetch_stat_io(void); +extern PgStat_BackendIO *pgstat_fetch_stat_backend_io(ProcNumber procnum); +extern void pgstat_io_reset_backend_cb(ProcNumber procnum, TimestampTz ts); extern const char *pgstat_get_io_context_name(IOContext io_context); extern const char *pgstat_get_io_object_name(IOObject io_object); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9a765eca359..4e4603fe5b1 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -477,12 +477,13 @@ typedef struct PgStatShared_Checkpointer /* Shared-memory ready PgStat_IO */ typedef struct PgStatShared_IO { + LWLock lock; + PgStat_IO stats; + /* - * locks[i] protects stats.stats[i]. locks[0] also protects - * stats.stat_reset_timestamp. + * Per-backend dshash, keyed by ProcNumber. */ - LWLock locks[BACKEND_NUM_TYPES]; - PgStat_IO stats; + dshash_table_handle backend_hash_handle; } PgStatShared_IO; typedef struct PgStatShared_Lock @@ -513,6 +514,16 @@ typedef struct PgStatShared_PerBackendEntry LWLock lock; } PgStatShared_PerBackendEntry; +/* + * Per-backend entry for IO statistics, stored in a dshash keyed by + * ProcNumber. + */ +typedef struct PgStatShared_IOBackendEntry +{ + PgStatShared_PerBackendEntry header; + PgStat_BackendIO stats; +} PgStatShared_IOBackendEntry; + /* * Per-backend entry for lock statistics, stored in a dshash keyed by * ProcNumber. @@ -584,12 +595,6 @@ typedef struct PgStatShared_ReplSlot PgStat_StatReplSlotEntry stats; } PgStatShared_ReplSlot; -typedef struct PgStatShared_Backend -{ - PgStatShared_Common header; - PgStat_Backend stats; -} PgStatShared_Backend; - /* * Central shared memory entry for the cumulative stats system. * @@ -770,19 +775,6 @@ extern void pgstat_archiver_init_shmem_cb(void *stats); extern void pgstat_archiver_reset_all_cb(TimestampTz ts); extern void pgstat_archiver_snapshot_cb(void); -/* - * Functions in pgstat_backend.c - */ - -/* flags for pgstat_flush_backend() */ -#define PGSTAT_BACKEND_FLUSH_IO (1 << 0) /* Flush I/O statistics */ -#define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO) - -extern bool pgstat_flush_backend(bool nowait, uint32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); -extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, - TimestampTz ts); - /* * Functions in pgstat_bgwriter.c */ @@ -833,6 +825,9 @@ extern bool pgstat_io_flush_cb(bool nowait); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); +extern void pgstat_io_acc_backend_cb(void); +extern void pgstat_io_acc_all_backends(void); +extern void pgstat_io_per_backend_acc_cb(void *dst, void *entry); /* * Functions in pgstat_lock.c diff --git a/src/include/utils/pgstat_kind.h b/src/include/utils/pgstat_kind.h index 2d78a029683..6d53d8e53c6 100644 --- a/src/include/utils/pgstat_kind.h +++ b/src/include/utils/pgstat_kind.h @@ -29,16 +29,15 @@ #define PGSTAT_KIND_FUNCTION 3 /* per-function statistics */ #define PGSTAT_KIND_REPLSLOT 4 /* per-slot statistics */ #define PGSTAT_KIND_SUBSCRIPTION 5 /* per-subscription statistics */ -#define PGSTAT_KIND_BACKEND 6 /* per-backend statistics */ /* stats for fixed-numbered objects */ -#define PGSTAT_KIND_ARCHIVER 7 -#define PGSTAT_KIND_BGWRITER 8 -#define PGSTAT_KIND_CHECKPOINTER 9 -#define PGSTAT_KIND_IO 10 -#define PGSTAT_KIND_LOCK 11 -#define PGSTAT_KIND_SLRU 12 -#define PGSTAT_KIND_WAL 13 +#define PGSTAT_KIND_ARCHIVER 6 +#define PGSTAT_KIND_BGWRITER 7 +#define PGSTAT_KIND_CHECKPOINTER 8 +#define PGSTAT_KIND_IO 9 +#define PGSTAT_KIND_LOCK 10 +#define PGSTAT_KIND_SLRU 11 +#define PGSTAT_KIND_WAL 12 #define PGSTAT_KIND_BUILTIN_MIN PGSTAT_KIND_DATABASE #define PGSTAT_KIND_BUILTIN_MAX PGSTAT_KIND_WAL diff --git a/src/test/regress/expected/stats.out b/src/test/regress/expected/stats.out index fae5f939523..e4369221ef3 100644 --- a/src/test/regress/expected/stats.out +++ b/src/test/regress/expected/stats.out @@ -119,15 +119,14 @@ SELECT id, name, fixed_amount, 3 | function | f | f | t 4 | replslot | f | t | t 5 | subscription | f | t | t - 6 | backend | f | t | f - 7 | archiver | t | f | t - 8 | bgwriter | t | f | t - 9 | checkpointer | t | f | t - 10 | io | t | f | t - 11 | lock | t | f | t - 12 | slru | t | f | t - 13 | wal | t | f | t -(13 rows) + 6 | archiver | t | f | t + 7 | bgwriter | t | f | t + 8 | checkpointer | t | f | t + 9 | io | t | f | t + 10 | lock | t | f | t + 11 | slru | t | f | t + 12 | wal | t | f | t +(12 rows) -- ensure that both seqscan and indexscan plans are allowed SET enable_seqscan TO on; @@ -1870,13 +1869,20 @@ SELECT :io_stats_post_reset < :io_stats_pre_reset; SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_reset FROM pg_stat_get_backend_io(pg_backend_pid()) \gset --- pg_stat_reset_shared() did not reset backend IO stats -SELECT :my_io_stats_pre_reset <= :my_io_stats_post_reset; +-- pg_stat_reset_shared() also resets per-backend IO stats +SELECT :my_io_stats_pre_reset > :my_io_stats_post_reset; ?column? ---------- t (1 row) +SELECT bool_and(stats_reset IS NOT NULL) AS backend_io_reset_timestamp_set + FROM pg_stat_get_backend_io(pg_backend_pid()); + backend_io_reset_timestamp_set +-------------------------------- + t +(1 row) + -- but pg_stat_reset_backend_stats() does SELECT pg_stat_reset_backend_stats(pg_backend_pid()); pg_stat_reset_backend_stats @@ -1903,11 +1909,13 @@ SELECT pg_stat_get_backend_io(0); ------------------------ (0 rows) --- Auxiliary processes return no data. -SELECT pg_stat_get_backend_io(:checkpointer_pid); - pg_stat_get_backend_io ------------------------- -(0 rows) +-- Auxiliary processes now return data +SELECT count(*) > 0 AS checkpointer_has_io_stats + FROM pg_stat_get_backend_io(:checkpointer_pid); + checkpointer_has_io_stats +--------------------------- + t +(1 row) -- test BRIN index doesn't block HOT update CREATE TABLE brin_hot ( diff --git a/src/test/regress/sql/stats.sql b/src/test/regress/sql/stats.sql index c17753dcd20..d5877cd8404 100644 --- a/src/test/regress/sql/stats.sql +++ b/src/test/regress/sql/stats.sql @@ -878,8 +878,10 @@ SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + SELECT :io_stats_post_reset < :io_stats_pre_reset; SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_reset FROM pg_stat_get_backend_io(pg_backend_pid()) \gset --- pg_stat_reset_shared() did not reset backend IO stats -SELECT :my_io_stats_pre_reset <= :my_io_stats_post_reset; +-- pg_stat_reset_shared() also resets per-backend IO stats +SELECT :my_io_stats_pre_reset > :my_io_stats_post_reset; +SELECT bool_and(stats_reset IS NOT NULL) AS backend_io_reset_timestamp_set + FROM pg_stat_get_backend_io(pg_backend_pid()); -- but pg_stat_reset_backend_stats() does SELECT pg_stat_reset_backend_stats(pg_backend_pid()); SELECT sum(evictions) + sum(reuses) + sum(extends) + sum(fsyncs) + sum(reads) + sum(writes) + sum(writebacks) + sum(hits) AS my_io_stats_post_backend_reset @@ -889,8 +891,9 @@ SELECT :my_io_stats_pre_reset > :my_io_stats_post_backend_reset; -- Check invalid input for pg_stat_get_backend_io() SELECT pg_stat_get_backend_io(NULL); SELECT pg_stat_get_backend_io(0); --- Auxiliary processes return no data. -SELECT pg_stat_get_backend_io(:checkpointer_pid); +-- Auxiliary processes now return data +SELECT count(*) > 0 AS checkpointer_has_io_stats + FROM pg_stat_get_backend_io(:checkpointer_pid); -- test BRIN index doesn't block HOT update CREATE TABLE brin_hot ( diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index d89fccf482d..52de6e93c31 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2316,7 +2316,6 @@ PgFdwSamplingMethod PgFdwScanState PgIfAddrCallback PgStatShared_Archiver -PgStatShared_Backend PgStatShared_BgWriter PgStatShared_Checkpointer PgStatShared_Common @@ -2326,6 +2325,7 @@ PgStatShared_Database PgStatShared_Function PgStatShared_HashEntry PgStatShared_IO +PgStatShared_IOBackendEntry PgStatShared_Lock PgStatShared_LockBackendEntry PgStatShared_PerBackendEntry @@ -2336,8 +2336,7 @@ PgStatShared_Subscription PgStatShared_Wal PgStatShared_WalBackendEntry PgStat_ArchiverStats -PgStat_Backend -PgStat_BackendPending +PgStat_BackendIO PgStat_BackendSubEntry PgStat_BgWriterStats PgStat_BktypeIO -- 2.34.1
