v14 attached.
On Tue, Oct 19, 2021 at 3:29 PM Andres Freund <and...@anarazel.de> wrote:
>
>
> > > Is pgstattuple the best place for this helper? It's not really pgstatfuncs
> > > specific...
> > >
> > > It also looks vaguely familiar - I wonder if we have a helper roughly like
> > > this somewhere else already...
> > >
> >
> > I don't see a function which is specifically a utility to make a
> > tuplestore. Looking at the callers of tuplestore_begin_heap(), I notice
> > very similar code to the function I added in pg_tablespace_databases()
> > in utils/adt/misc.c, pg_stop_backup_v2() in xlogfuncs.c,
> > pg_event_trigger_dropped_objects() and pg_event_trigger_ddl_commands in
> > event_tigger.c, pg_available_extensions in extension.c, etc.
> >
> > Do you think it makes sense to refactor this code out of all of these
> > places?
>
> Yes, I think it'd make sense. We have about 40 copies of this stuff, which is
> fairly ridiculous.
>
>
> > If so, where would such a utility function belong?
>
> Not quite sure. src/backend/utils/fmgr/funcapi.c maybe? I suggest starting a
> separate thread about that...
>
done [1]. also, I dropped that commit from this patchset.
>
> > > > @@ -2999,6 +3036,14 @@ pgstat_shutdown_hook(int code, Datum arg)
> > > > {
> > > > Assert(!pgstat_is_shutdown);
> > > >
> > > > + /*
> > > > + * Only need to send stats on IO Ops for IO Paths when a process
> > > > exits, as
> > > > + * pg_stat_get_buffers() will read from live backends'
> > > > PgBackendStatus and
> > > > + * then sum this with totals from exited backends persisted by
> > > > the stats
> > > > + * collector.
> > > > + */
> > > > + pgstat_send_buffers();
> > > > +
> > > > /*
> > > > * If we got as far as discovering our own database ID, we can
> > > > report what
> > > > * we did to the collector. Otherwise, we'd be sending an invalid
> > > > @@ -3092,6 +3137,30 @@ pgstat_send(void *msg, int len)
> > > > #endif
> > > > }
> > >
> > > I think it might be nicer to move pgstat_beshutdown_hook() to be a
> > > before_shmem_exit(), and do this in there.
> > >
> >
> > Not really sure the correct way to do this. A cursory attempt to do so
> > failed because ShutdownXLOG() is also registered as a
> > before_shmem_exit() and ends up being called after
> > pgstat_beshutdown_hook(). pgstat_beshutdown_hook() zeroes out
> > PgBackendStatus, ShutdownXLOG() initiates a checkpoint, and during a
> > checkpoint, the checkpointer increments IO op counter for writes in its
> > PgBackendStatus.
>
> I think we'll really need to do a proper redesign of the shutdown callback
> mechanism :(.
>
I've left what I originally had, then.
>
>
> > > > +PgBackendStatus *
> > > > +pgstat_fetch_backend_statuses(void)
> > > > +{
> > > > + return BackendStatusArray;
> > > > +}
> > >
> > > Hm, not sure this adds much?
> >
> > Is there a better way to access the whole BackendStatusArray from within
> > pgstatfuncs.c?
>
> Export the variable itself?
>
done but wasn't sure about PGDLLIMPORT
>
> > > IIRC Thomas Munro had a patch adding a nonatomic_add or such
> > > somewhere. Perhaps in the recovery readahead thread? Might be worth using
> > > instead?
> > >
> >
> > I've added Thomas' function in a separate commit. I looked for a better
> > place to add it (I was thinking somewhere in src/backend/utils/misc) but
> > couldn't find anywhere that made sense.
>
> I think it should just live in atomics.h?
>
done
-- melanie
[1]
https://www.postgresql.org/message-id/flat/CAAKRu_azyd1Z3W_r7Ou4sorTjRCs%2BPxeHw1CWJeXKofkE6TuZg%40mail.gmail.com
From 41242606a03aace906e307a38fc67c5cefcaec20 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplage...@gmail.com>
Date: Mon, 11 Oct 2021 16:15:06 -0400
Subject: [PATCH v14 2/4] Read-only atomic backend write function
For counters in shared memory which can be read by any backend but only
written to by one backend, an atomic is still needed to protect against
torn values, however, pg_atomic_fetch_add_u64() is overkill for
incrementing the counter. inc_counter() is a helper function which can
be used to increment these values safely but without unnecessary
overhead.
Author: Thomas Munro
---
src/include/port/atomics.h | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/src/include/port/atomics.h b/src/include/port/atomics.h
index 856338f161..09a2575d6a 100644
--- a/src/include/port/atomics.h
+++ b/src/include/port/atomics.h
@@ -519,6 +519,16 @@ pg_atomic_sub_fetch_u64(volatile pg_atomic_uint64 *ptr, int64 sub_)
return pg_atomic_sub_fetch_u64_impl(ptr, sub_);
}
+/*
+ * On modern systems this is really just *counter++. On some older systems
+ * there might be more to it, due to inability to read and write 64 bit values
+ * atomically.
+ */
+static inline void inc_counter(pg_atomic_uint64 *counter)
+{
+ pg_atomic_write_u64(counter, pg_atomic_read_u64(counter) + 1);
+}
+
#undef INSIDE_ATOMICS_H
#endif /* ATOMICS_H */
--
2.30.2
From cfd8941958877a9bf3f5946e18aa098b1351d36f Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplage...@gmail.com>
Date: Wed, 29 Sep 2021 15:44:51 -0400
Subject: [PATCH v14 4/4] Remove superfluous bgwriter stats
Remove stats from pg_stat_bgwriter which are now more clearly expressed
in pg_stat_buffers.
TODO:
- make pg_stat_checkpointer view and move relevant stats into it
- add additional stats to pg_stat_bgwriter
---
doc/src/sgml/monitoring.sgml | 47 ---------------------------
src/backend/catalog/system_views.sql | 6 +---
src/backend/postmaster/checkpointer.c | 26 ---------------
src/backend/postmaster/pgstat.c | 5 ---
src/backend/storage/buffer/bufmgr.c | 6 ----
src/backend/utils/adt/pgstatfuncs.c | 30 -----------------
src/include/catalog/pg_proc.dat | 22 -------------
src/include/pgstat.h | 10 ------
src/test/regress/expected/rules.out | 5 ---
9 files changed, 1 insertion(+), 156 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 77b044343d..1aa94c5b19 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -3400,24 +3400,6 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
</para></entry>
</row>
- <row>
- <entry role="catalog_table_entry"><para role="column_definition">
- <structfield>buffers_checkpoint</structfield> <type>bigint</type>
- </para>
- <para>
- Number of buffers written during checkpoints
- </para></entry>
- </row>
-
- <row>
- <entry role="catalog_table_entry"><para role="column_definition">
- <structfield>buffers_clean</structfield> <type>bigint</type>
- </para>
- <para>
- Number of buffers written by the background writer
- </para></entry>
- </row>
-
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>maxwritten_clean</structfield> <type>bigint</type>
@@ -3428,35 +3410,6 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
</para></entry>
</row>
- <row>
- <entry role="catalog_table_entry"><para role="column_definition">
- <structfield>buffers_backend</structfield> <type>bigint</type>
- </para>
- <para>
- Number of buffers written directly by a backend
- </para></entry>
- </row>
-
- <row>
- <entry role="catalog_table_entry"><para role="column_definition">
- <structfield>buffers_backend_fsync</structfield> <type>bigint</type>
- </para>
- <para>
- Number of times a backend had to execute its own
- <function>fsync</function> call (normally the background writer handles those
- even when the backend does its own write)
- </para></entry>
- </row>
-
- <row>
- <entry role="catalog_table_entry"><para role="column_definition">
- <structfield>buffers_alloc</structfield> <type>bigint</type>
- </para>
- <para>
- Number of buffers allocated
- </para></entry>
- </row>
-
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>stats_reset</structfield> <type>timestamp with time zone</type>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 86ca35121b..1e93937198 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1062,18 +1062,14 @@ CREATE VIEW pg_stat_archiver AS
s.stats_reset
FROM pg_stat_get_archiver() s;
+-- TODO: make separate pg_stat_checkpointer view
CREATE VIEW pg_stat_bgwriter AS
SELECT
pg_stat_get_bgwriter_timed_checkpoints() AS checkpoints_timed,
pg_stat_get_bgwriter_requested_checkpoints() AS checkpoints_req,
pg_stat_get_checkpoint_write_time() AS checkpoint_write_time,
pg_stat_get_checkpoint_sync_time() AS checkpoint_sync_time,
- pg_stat_get_bgwriter_buf_written_checkpoints() AS buffers_checkpoint,
- pg_stat_get_bgwriter_buf_written_clean() AS buffers_clean,
pg_stat_get_bgwriter_maxwritten_clean() AS maxwritten_clean,
- pg_stat_get_buf_written_backend() AS buffers_backend,
- pg_stat_get_buf_fsync_backend() AS buffers_backend_fsync,
- pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
CREATE VIEW pg_stat_buffers AS
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 1306b5238d..dec325e40e 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -90,17 +90,9 @@
* requesting backends since the last checkpoint start. The flags are
* chosen so that OR'ing is the correct way to combine multiple requests.
*
- * num_backend_writes is used to count the number of buffer writes performed
- * by user backend processes. This counter should be wide enough that it
- * can't overflow during a single processing cycle. num_backend_fsync
- * counts the subset of those writes that also had to do their own fsync,
- * because the checkpointer failed to absorb their request.
- *
* The requests array holds fsync requests sent by backends and not yet
* absorbed by the checkpointer.
*
- * Unlike the checkpoint fields, num_backend_writes, num_backend_fsync, and
- * the requests fields are protected by CheckpointerCommLock.
*----------
*/
typedef struct
@@ -124,9 +116,6 @@ typedef struct
ConditionVariable start_cv; /* signaled when ckpt_started advances */
ConditionVariable done_cv; /* signaled when ckpt_done advances */
- uint32 num_backend_writes; /* counts user backend buffer writes */
- uint32 num_backend_fsync; /* counts user backend fsync calls */
-
int num_requests; /* current # of requests */
int max_requests; /* allocated array size */
CheckpointerRequest requests[FLEXIBLE_ARRAY_MEMBER];
@@ -1085,10 +1074,6 @@ ForwardSyncRequest(const FileTag *ftag, SyncRequestType type)
LWLockAcquire(CheckpointerCommLock, LW_EXCLUSIVE);
- /* Count all backend writes regardless of if they fit in the queue */
- if (!AmBackgroundWriterProcess())
- CheckpointerShmem->num_backend_writes++;
-
/*
* If the checkpointer isn't running or the request queue is full, the
* backend will have to perform its own fsync request. But before forcing
@@ -1103,8 +1088,6 @@ ForwardSyncRequest(const FileTag *ftag, SyncRequestType type)
* Count the subset of writes where backends have to do their own
* fsync
*/
- if (!AmBackgroundWriterProcess())
- CheckpointerShmem->num_backend_fsync++;
pgstat_inc_ioop(IOOP_FSYNC, IOPATH_SHARED);
return false;
}
@@ -1261,15 +1244,6 @@ AbsorbSyncRequests(void)
LWLockAcquire(CheckpointerCommLock, LW_EXCLUSIVE);
- /* Transfer stats counts into pending pgstats message */
- PendingCheckpointerStats.m_buf_written_backend
- += CheckpointerShmem->num_backend_writes;
- PendingCheckpointerStats.m_buf_fsync_backend
- += CheckpointerShmem->num_backend_fsync;
-
- CheckpointerShmem->num_backend_writes = 0;
- CheckpointerShmem->num_backend_fsync = 0;
-
/*
* We try to avoid holding the lock for a long time by copying the request
* array, and processing the requests after releasing the lock.
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 355690d944..e0762444af 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -5616,9 +5616,7 @@ pgstat_recv_archiver(PgStat_MsgArchiver *msg, int len)
static void
pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len)
{
- globalStats.bgwriter.buf_written_clean += msg->m_buf_written_clean;
globalStats.bgwriter.maxwritten_clean += msg->m_maxwritten_clean;
- globalStats.bgwriter.buf_alloc += msg->m_buf_alloc;
}
/* ----------
@@ -5634,9 +5632,6 @@ pgstat_recv_checkpointer(PgStat_MsgCheckpointer *msg, int len)
globalStats.checkpointer.requested_checkpoints += msg->m_requested_checkpoints;
globalStats.checkpointer.checkpoint_write_time += msg->m_checkpoint_write_time;
globalStats.checkpointer.checkpoint_sync_time += msg->m_checkpoint_sync_time;
- globalStats.checkpointer.buf_written_checkpoints += msg->m_buf_written_checkpoints;
- globalStats.checkpointer.buf_written_backend += msg->m_buf_written_backend;
- globalStats.checkpointer.buf_fsync_backend += msg->m_buf_fsync_backend;
}
static void
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 6926fc5742..67447f997a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -2164,7 +2164,6 @@ BufferSync(int flags)
if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN)
{
TRACE_POSTGRESQL_BUFFER_SYNC_WRITTEN(buf_id);
- PendingCheckpointerStats.m_buf_written_checkpoints++;
num_written++;
}
}
@@ -2273,9 +2272,6 @@ BgBufferSync(WritebackContext *wb_context)
*/
strategy_buf_id = StrategySyncStart(&strategy_passes, &recent_alloc);
- /* Report buffer alloc counts to pgstat */
- PendingBgWriterStats.m_buf_alloc += recent_alloc;
-
/*
* If we're not running the LRU scan, just stop after doing the stats
* stuff. We mark the saved state invalid so that we can recover sanely
@@ -2472,8 +2468,6 @@ BgBufferSync(WritebackContext *wb_context)
reusable_buffers++;
}
- PendingBgWriterStats.m_buf_written_clean += num_written;
-
#ifdef BGW_DEBUG
elog(DEBUG1, "bgwriter: recent_alloc=%u smoothed=%.2f delta=%ld ahead=%d density=%.2f reusable_est=%d upcoming_est=%d scanned=%d wrote=%d reusable=%d",
recent_alloc, smoothed_alloc, strategy_delta, bufs_ahead,
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 734079e233..641a891e34 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1738,18 +1738,6 @@ pg_stat_get_bgwriter_requested_checkpoints(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_checkpointer()->requested_checkpoints);
}
-Datum
-pg_stat_get_bgwriter_buf_written_checkpoints(PG_FUNCTION_ARGS)
-{
- PG_RETURN_INT64(pgstat_fetch_stat_checkpointer()->buf_written_checkpoints);
-}
-
-Datum
-pg_stat_get_bgwriter_buf_written_clean(PG_FUNCTION_ARGS)
-{
- PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_written_clean);
-}
-
Datum
pg_stat_get_bgwriter_maxwritten_clean(PG_FUNCTION_ARGS)
{
@@ -1809,24 +1797,6 @@ get_pg_stat_buffers_row(Datum all_values[NROWS][COLUMN_LENGTH], BackendType back
return all_values[(backend_type - 1) * IOPATH_NUM_TYPES + io_path];
}
-Datum
-pg_stat_get_buf_written_backend(PG_FUNCTION_ARGS)
-{
- PG_RETURN_INT64(pgstat_fetch_stat_checkpointer()->buf_written_backend);
-}
-
-Datum
-pg_stat_get_buf_fsync_backend(PG_FUNCTION_ARGS)
-{
- PG_RETURN_INT64(pgstat_fetch_stat_checkpointer()->buf_fsync_backend);
-}
-
-Datum
-pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
-{
- PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
-}
-
Datum
pg_stat_get_buffers(PG_FUNCTION_ARGS)
{
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index bbdb07b222..a3cdbe1dbc 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5600,16 +5600,6 @@
proname => 'pg_stat_get_bgwriter_requested_checkpoints', provolatile => 's',
proparallel => 'r', prorettype => 'int8', proargtypes => '',
prosrc => 'pg_stat_get_bgwriter_requested_checkpoints' },
-{ oid => '2771',
- descr => 'statistics: number of buffers written by the bgwriter during checkpoints',
- proname => 'pg_stat_get_bgwriter_buf_written_checkpoints', provolatile => 's',
- proparallel => 'r', prorettype => 'int8', proargtypes => '',
- prosrc => 'pg_stat_get_bgwriter_buf_written_checkpoints' },
-{ oid => '2772',
- descr => 'statistics: number of buffers written by the bgwriter for cleaning dirty buffers',
- proname => 'pg_stat_get_bgwriter_buf_written_clean', provolatile => 's',
- proparallel => 'r', prorettype => 'int8', proargtypes => '',
- prosrc => 'pg_stat_get_bgwriter_buf_written_clean' },
{ oid => '2773',
descr => 'statistics: number of times the bgwriter stopped processing when it had written too many buffers while cleaning',
proname => 'pg_stat_get_bgwriter_maxwritten_clean', provolatile => 's',
@@ -5629,18 +5619,6 @@
proname => 'pg_stat_get_checkpoint_sync_time', provolatile => 's',
proparallel => 'r', prorettype => 'float8', proargtypes => '',
prosrc => 'pg_stat_get_checkpoint_sync_time' },
-{ oid => '2775', descr => 'statistics: number of buffers written by backends',
- proname => 'pg_stat_get_buf_written_backend', provolatile => 's',
- proparallel => 'r', prorettype => 'int8', proargtypes => '',
- prosrc => 'pg_stat_get_buf_written_backend' },
-{ oid => '3063',
- descr => 'statistics: number of backend buffer writes that did their own fsync',
- proname => 'pg_stat_get_buf_fsync_backend', provolatile => 's',
- proparallel => 'r', prorettype => 'int8', proargtypes => '',
- prosrc => 'pg_stat_get_buf_fsync_backend' },
-{ oid => '2859', descr => 'statistics: number of buffer allocations',
- proname => 'pg_stat_get_buf_alloc', provolatile => 's', proparallel => 'r',
- prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc' },
{ oid => '8459', descr => 'statistics: counts of all IO operations done to all IO paths by each type of backend.',
proname => 'pg_stat_get_buffers', provolatile => 's', proisstrict => 'f',
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index ac8aca2c61..2d72933e90 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -505,9 +505,7 @@ typedef struct PgStat_MsgBgWriter
{
PgStat_MsgHdr m_hdr;
- PgStat_Counter m_buf_written_clean;
PgStat_Counter m_maxwritten_clean;
- PgStat_Counter m_buf_alloc;
} PgStat_MsgBgWriter;
/* ----------
@@ -520,9 +518,6 @@ typedef struct PgStat_MsgCheckpointer
PgStat_Counter m_timed_checkpoints;
PgStat_Counter m_requested_checkpoints;
- PgStat_Counter m_buf_written_checkpoints;
- PgStat_Counter m_buf_written_backend;
- PgStat_Counter m_buf_fsync_backend;
PgStat_Counter m_checkpoint_write_time; /* times in milliseconds */
PgStat_Counter m_checkpoint_sync_time;
} PgStat_MsgCheckpointer;
@@ -898,9 +893,7 @@ typedef struct PgStat_ArchiverStats
*/
typedef struct PgStat_BgWriterStats
{
- PgStat_Counter buf_written_clean;
PgStat_Counter maxwritten_clean;
- PgStat_Counter buf_alloc;
TimestampTz stat_reset_timestamp;
} PgStat_BgWriterStats;
@@ -914,9 +907,6 @@ typedef struct PgStat_CheckpointerStats
PgStat_Counter requested_checkpoints;
PgStat_Counter checkpoint_write_time; /* times in milliseconds */
PgStat_Counter checkpoint_sync_time;
- PgStat_Counter buf_written_checkpoints;
- PgStat_Counter buf_written_backend;
- PgStat_Counter buf_fsync_backend;
} PgStat_CheckpointerStats;
/*
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 5e5a0324ee..090a65cdb0 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1821,12 +1821,7 @@ pg_stat_bgwriter| SELECT pg_stat_get_bgwriter_timed_checkpoints() AS checkpoints
pg_stat_get_bgwriter_requested_checkpoints() AS checkpoints_req,
pg_stat_get_checkpoint_write_time() AS checkpoint_write_time,
pg_stat_get_checkpoint_sync_time() AS checkpoint_sync_time,
- pg_stat_get_bgwriter_buf_written_checkpoints() AS buffers_checkpoint,
- pg_stat_get_bgwriter_buf_written_clean() AS buffers_clean,
pg_stat_get_bgwriter_maxwritten_clean() AS maxwritten_clean,
- pg_stat_get_buf_written_backend() AS buffers_backend,
- pg_stat_get_buf_fsync_backend() AS buffers_backend_fsync,
- pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
pg_stat_buffers| SELECT b.backend_type,
b.io_path,
--
2.30.2
From 45b4d9bd16c37965eb73b74073121c985b230abf Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplage...@gmail.com>
Date: Fri, 24 Sep 2021 17:39:12 -0400
Subject: [PATCH v14 1/4] Allow bootstrap process to beinit
---
src/backend/utils/init/postinit.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 78bc64671e..fba5864172 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -670,8 +670,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
EnablePortalManager();
/* Initialize status reporting */
- if (!bootstrap)
- pgstat_beinit();
+ pgstat_beinit();
/*
* Load relcache entries for the shared system catalogs. This must create
--
2.30.2
From 2895819b15189495dba26fa5b3b91b8fb07f35ad Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplage...@gmail.com>
Date: Wed, 29 Sep 2021 15:39:45 -0400
Subject: [PATCH v14 3/4] Add system view tracking IO ops per backend type
Add pg_stat_buffers, a system view which tracks the number of IO
operations (allocs, writes, fsyncs, and extends) done through each IO
path (e.g. shared buffers, local buffers, unbuffered IO) by each type of
backend.
Some of these should always be zero. For example, checkpointer does not
use a BufferAccessStrategy (currently), so the "strategy" IO path for
checkpointer will be 0 for all IO operations (alloc, write, fsync, and
extend).
All backends increment a counter in their PgBackendStatus when
performing an IO operation. On exit, backends send these stats to the
stats collector to be persisted.
When stats are reset, the backend sending the reset message will loop
through and collect all of the live backends' IO op stats, sending a
reset message for each backend type containing these stats. When
receiving this message, the stats collector will 1) save these reset
values in an array of "resets" and 2) zero out the exited backends'
saved IO op counters. This is required for accurate stats after a reset
without writing to other backends' PgBackendStatuses.
When the pg_stat_buffers view is queried, one backend will sum live
backends' stats with saved stats from exited backends and subtract saved
reset stats, returning the total.
Each row of the view is stats for a particular backend type for a
particular IO path (e.g. shared buffer accesses by checkpointer) and
each column in the view is the total number of IO operations done (e.g.
writes).
So a cell in the view would be, for example, the number of shared
buffers written by checkpointer since the last stats reset.
Note that this commit does not add code to increment IO ops for all IO
paths. It includes all possible combinations in the stats view but
doesn't populate all of them.
A separate proposed patch [1] which would add wrappers for smgrwrite()
and extend() would provide a good location to call pgstat_inc_ioop() for
unbuffered IO and avoid regressions for future users of these functions.
TODO:
- catalog bump
[1] https://www.postgresql.org/message-id/CAAKRu_aw72w70X1P%3Dba20K8iGUvSkyz7Yk03wPPh3f9WgmcJ3g%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/flat/20210415235954.qcypb4urtovzkat5%40alap3.anarazel.de#724d5cce4bcb587f9167b80a5824bc5c
---
doc/src/sgml/monitoring.sgml | 116 +++++++++++++-
src/backend/catalog/system_views.sql | 11 ++
src/backend/postmaster/checkpointer.c | 3 +-
src/backend/postmaster/pgstat.c | 161 +++++++++++++++++++-
src/backend/storage/buffer/bufmgr.c | 46 ++++--
src/backend/storage/buffer/freelist.c | 23 ++-
src/backend/storage/buffer/localbuf.c | 3 +
src/backend/storage/sync/sync.c | 1 +
src/backend/utils/activity/backend_status.c | 60 +++++++-
src/backend/utils/adt/pgstatfuncs.c | 152 ++++++++++++++++++
src/include/catalog/pg_proc.dat | 9 ++
src/include/miscadmin.h | 2 +
src/include/pgstat.h | 53 +++++++
src/include/storage/buf_internals.h | 4 +-
src/include/utils/backend_status.h | 80 ++++++++++
src/test/regress/expected/rules.out | 8 +
16 files changed, 701 insertions(+), 31 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 3173ec2566..77b044343d 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -435,6 +435,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
</entry>
</row>
+ <row>
+ <entry><structname>pg_stat_buffers</structname><indexterm><primary>pg_stat_buffers</primary></indexterm></entry>
+ <entry>A row for each IO path for each backend type showing
+ statistics about backend IO operations. See
+ <link linkend="monitoring-pg-stat-buffers-view">
+ <structname>pg_stat_buffers</structname></link> for details.
+ </entry>
+ </row>
+
<row>
<entry><structname>pg_stat_wal</structname><indexterm><primary>pg_stat_wal</primary></indexterm></entry>
<entry>One row only, showing statistics about WAL activity. See
@@ -3462,6 +3471,101 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
</sect2>
+ <sect2 id="monitoring-pg-stat-buffers-view">
+ <title><structname>pg_stat_buffers</structname></title>
+
+ <indexterm>
+ <primary>pg_stat_buffers</primary>
+ </indexterm>
+
+ <para>
+ The <structname>pg_stat_buffers</structname> view has a row for each backend
+ type for each possible IO path, containing global data for the cluster for
+ that backend and IO path.
+ </para>
+
+ <table id="pg-stat-buffers-view" xreflabel="pg_stat_buffers">
+ <title><structname>pg_stat_buffers</structname> View</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>backend_type</structfield> <type>text</type>
+ </para>
+ <para>
+ Type of backend (e.g. background worker, autovacuum worker).
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>io_path</structfield> <type>text</type>
+ </para>
+ <para>
+ IO path taken (e.g. shared buffers, direct).
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>alloc</structfield> <type>integer</type>
+ </para>
+ <para>
+ Number of buffers allocated.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>extend</structfield> <type>integer</type>
+ </para>
+ <para>
+ Number of buffers extended.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>fsync</structfield> <type>integer</type>
+ </para>
+ <para>
+ Number of buffers fsynced.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>write</structfield> <type>integer</type>
+ </para>
+ <para>
+ Number of buffers written.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>stats_reset</structfield> <type>timestamp with time zone</type>
+ </para>
+ <para>
+ Time at which these statistics were last reset.
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </sect2>
+
<sect2 id="monitoring-pg-stat-wal-view">
<title><structname>pg_stat_wal</structname></title>
@@ -5058,12 +5162,14 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
</para>
<para>
Resets some cluster-wide statistics counters to zero, depending on the
- argument. The argument can be <literal>bgwriter</literal> to reset
- all the counters shown in
- the <structname>pg_stat_bgwriter</structname>
+ argument. The argument can be <literal>bgwriter</literal> to reset all
+ the counters shown in the <structname>pg_stat_bgwriter</structname>
view, <literal>archiver</literal> to reset all the counters shown in
- the <structname>pg_stat_archiver</structname> view or <literal>wal</literal>
- to reset all the counters shown in the <structname>pg_stat_wal</structname> view.
+ the <structname>pg_stat_archiver</structname> view,
+ <literal>wal</literal> to reset all the counters shown in the
+ <structname>pg_stat_wal</structname> view, or
+ <literal>buffers</literal> to reset all the counters shown in the
+ <structname>pg_stat_buffers</structname> view.
</para>
<para>
This function is restricted to superusers by default, but other users
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index eb560955cd..86ca35121b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1076,6 +1076,17 @@ CREATE VIEW pg_stat_bgwriter AS
pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
+CREATE VIEW pg_stat_buffers AS
+SELECT
+ b.backend_type,
+ b.io_path,
+ b.alloc,
+ b.extend,
+ b.fsync,
+ b.write,
+ b.stats_reset
+FROM pg_stat_get_buffers() b;
+
CREATE VIEW pg_stat_wal AS
SELECT
w.wal_records,
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index be7366379d..1306b5238d 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -1098,13 +1098,14 @@ ForwardSyncRequest(const FileTag *ftag, SyncRequestType type)
(CheckpointerShmem->num_requests >= CheckpointerShmem->max_requests &&
!CompactCheckpointerRequestQueue()))
{
+ LWLockRelease(CheckpointerCommLock);
/*
* Count the subset of writes where backends have to do their own
* fsync
*/
if (!AmBackgroundWriterProcess())
CheckpointerShmem->num_backend_fsync++;
- LWLockRelease(CheckpointerCommLock);
+ pgstat_inc_ioop(IOOP_FSYNC, IOPATH_SHARED);
return false;
}
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index b7d0fbaefd..355690d944 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -124,9 +124,12 @@ char *pgstat_stat_filename = NULL;
char *pgstat_stat_tmpname = NULL;
/*
- * BgWriter and WAL global statistics counters.
- * Stored directly in a stats message structure so they can be sent
- * without needing to copy things around. We assume these init to zeroes.
+ * BgWriter, Checkpointer, WAL, and I/O global statistics counters. I/O global
+ * statistics on various IO ops are tracked in PgBackendStatus while a backend
+ * is alive and then sent to stats collector before a backend exits in a
+ * PgStat_MsgIOPathOps.
+ * All others are stored directly in a stats message structure so they can be
+ * sent without needing to copy things around. We assume these init to zeroes.
*/
PgStat_MsgBgWriter PendingBgWriterStats;
PgStat_MsgCheckpointer PendingCheckpointerStats;
@@ -362,6 +365,7 @@ static void pgstat_recv_analyze(PgStat_MsgAnalyze *msg, int len);
static void pgstat_recv_archiver(PgStat_MsgArchiver *msg, int len);
static void pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len);
static void pgstat_recv_checkpointer(PgStat_MsgCheckpointer *msg, int len);
+static void pgstat_recv_io_path_ops(PgStat_MsgIOPathOps *msg, int len);
static void pgstat_recv_wal(PgStat_MsgWal *msg, int len);
static void pgstat_recv_slru(PgStat_MsgSLRU *msg, int len);
static void pgstat_recv_funcstat(PgStat_MsgFuncstat *msg, int len);
@@ -1431,6 +1435,28 @@ pgstat_reset_counters(void)
pgstat_send(&msg, sizeof(msg));
}
+/*
+ * Helper function to collect and send live backends' current IO operations
+ * stats counters when a stats reset is initiated so that they may be deducted
+ * from future totals.
+ */
+static void
+pgstat_send_buffers_reset(PgStat_MsgResetsharedcounter *msg)
+{
+ int backend_type;
+ PgStatIOPathOps ops[BACKEND_NUM_TYPES];
+
+ memset(ops, 0, sizeof(ops));
+ pgstat_report_live_backend_io_path_ops(ops);
+
+ for (backend_type = 1; backend_type < BACKEND_NUM_TYPES; backend_type++)
+ {
+ msg->m_backend_resets.backend_type = backend_type;
+ memcpy(&msg->m_backend_resets.iop, &ops[backend_type], sizeof(msg->m_backend_resets.iop));
+ pgstat_send(msg, sizeof(PgStat_MsgResetsharedcounter));
+ }
+}
+
/* ----------
* pgstat_reset_shared_counters() -
*
@@ -1448,7 +1474,14 @@ pgstat_reset_shared_counters(const char *target)
if (pgStatSock == PGINVALID_SOCKET)
return;
- if (strcmp(target, "archiver") == 0)
+ pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
+ if (strcmp(target, "buffers") == 0)
+ {
+ msg.m_resettarget = RESET_BUFFERS;
+ pgstat_send_buffers_reset(&msg);
+ return;
+ }
+ else if (strcmp(target, "archiver") == 0)
msg.m_resettarget = RESET_ARCHIVER;
else if (strcmp(target, "bgwriter") == 0)
msg.m_resettarget = RESET_BGWRITER;
@@ -1460,8 +1493,9 @@ pgstat_reset_shared_counters(const char *target)
errmsg("unrecognized reset target: \"%s\"", target),
errhint("Target must be \"archiver\", \"bgwriter\", or \"wal\".")));
- pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_RESETSHAREDCOUNTER);
+
pgstat_send(&msg, sizeof(msg));
+
}
/* ----------
@@ -2760,6 +2794,19 @@ pgstat_twophase_postabort(TransactionId xid, uint16 info,
rec->tuples_inserted + rec->tuples_updated;
}
+/*
+ * Support function for SQL-callable pgstat* functions. Returns a pointer to
+ * the PgStat_BackendIOPathOps structure tracking IO op statistics for both
+ * exited backends and reset arithmetic.
+ */
+PgStat_BackendIOPathOps *
+pgstat_fetch_exited_backend_buffers(void)
+{
+ backend_read_statsfile();
+
+ return &globalStats.buffers;
+}
+
/* ----------
* pgstat_fetch_stat_dbentry() -
@@ -2999,6 +3046,14 @@ pgstat_shutdown_hook(int code, Datum arg)
{
Assert(!pgstat_is_shutdown);
+ /*
+ * Only need to send stats on IO Ops for IO Paths when a process exits, as
+ * pg_stat_get_buffers() will read from live backends' PgBackendStatus and
+ * then sum this with totals from exited backends persisted by the stats
+ * collector.
+ */
+ pgstat_send_buffers();
+
/*
* If we got as far as discovering our own database ID, we can report what
* we did to the collector. Otherwise, we'd be sending an invalid
@@ -3148,6 +3203,31 @@ pgstat_send_bgwriter(void)
MemSet(&PendingBgWriterStats, 0, sizeof(PendingBgWriterStats));
}
+/*
+ * Before exiting, a backend sends its IO op statistics to the collector so
+ * that they may be persisted.
+ */
+void
+pgstat_send_buffers(void)
+{
+ PgStat_MsgIOPathOps msg;
+
+ PgBackendStatus *beentry = MyBEEntry;
+
+ if (!beentry)
+ return;
+
+ memset(&msg, 0, sizeof(msg));
+ msg.backend_type = beentry->st_backendType;
+
+ pgstat_sum_io_path_ops(msg.iop.io_path_ops,
+ (IOOps *) &beentry->io_path_stats);
+
+ pgstat_setheader(&msg.m_hdr, PGSTAT_MTYPE_IO_PATH_OPS);
+ pgstat_send(&msg, sizeof(msg));
+}
+
+
/* ----------
* pgstat_send_checkpointer() -
*
@@ -3312,6 +3392,29 @@ pgstat_send_slru(void)
}
}
+/*
+ * Helper function to sum all live IO Op stats for all IO Paths (e.g. shared,
+ * local) to those in the equivalent stats structure for exited backends. Note
+ * that this adds and doesn't set, so the destination stats structure should be
+ * zeroed out by the caller initially. This would commonly be used to transfer
+ * all IO Op stats for all IO Paths for a particular backend type to the
+ * pgstats structure.
+ */
+void
+pgstat_sum_io_path_ops(PgStatIOOps *dest, IOOps *src)
+{
+ int io_path;
+ for (io_path = 0; io_path < IOPATH_NUM_TYPES; io_path++)
+ {
+ dest->allocs += pg_atomic_read_u64(&src->allocs);
+ dest->extends += pg_atomic_read_u64(&src->extends);
+ dest->fsyncs += pg_atomic_read_u64(&src->fsyncs);
+ dest->writes += pg_atomic_read_u64(&src->writes);
+ dest++;
+ src++;
+ }
+
+}
/* ----------
* PgstatCollectorMain() -
@@ -3522,6 +3625,10 @@ PgstatCollectorMain(int argc, char *argv[])
pgstat_recv_checkpointer(&msg.msg_checkpointer, len);
break;
+ case PGSTAT_MTYPE_IO_PATH_OPS:
+ pgstat_recv_io_path_ops(&msg.msg_io_path_ops, len);
+ break;
+
case PGSTAT_MTYPE_WAL:
pgstat_recv_wal(&msg.msg_wal, len);
break;
@@ -5221,10 +5328,30 @@ pgstat_recv_resetsharedcounter(PgStat_MsgResetsharedcounter *msg, int len)
{
if (msg->m_resettarget == RESET_BGWRITER)
{
- /* Reset the global, bgwriter and checkpointer statistics for the cluster. */
- memset(&globalStats, 0, sizeof(globalStats));
+ /*
+ * Reset the global, bgwriter and checkpointer statistics for the
+ * cluster.
+ */
+ memset(&globalStats.checkpointer, 0, sizeof(globalStats.checkpointer));
+ memset(&globalStats.bgwriter, 0, sizeof(globalStats.bgwriter));
globalStats.bgwriter.stat_reset_timestamp = GetCurrentTimestamp();
}
+ else if (msg->m_resettarget == RESET_BUFFERS)
+ {
+ BackendType backend_type = msg->m_backend_resets.backend_type;
+
+ /*
+ * Though globalStats.buffers only needs to be reset once, doing so
+ * for every message is less brittle and the extra cost is irrelevant
+ * given how often stats are reset.
+ */
+ memset(&globalStats.buffers.ops, 0, sizeof(globalStats.buffers.ops));
+ globalStats.buffers.stat_reset_timestamp = GetCurrentTimestamp();
+
+ memcpy(&globalStats.buffers.resets[backend_type],
+ &msg->m_backend_resets.iop.io_path_ops, sizeof(msg->m_backend_resets.iop.io_path_ops));
+
+ }
else if (msg->m_resettarget == RESET_ARCHIVER)
{
/* Reset the archiver statistics for the cluster. */
@@ -5512,6 +5639,26 @@ pgstat_recv_checkpointer(PgStat_MsgCheckpointer *msg, int len)
globalStats.checkpointer.buf_fsync_backend += msg->m_buf_fsync_backend;
}
+static void
+pgstat_recv_io_path_ops(PgStat_MsgIOPathOps *msg, int len)
+{
+ int io_path;
+ PgStatIOOps *src_io_path_ops = msg->iop.io_path_ops;
+ PgStatIOOps *dest_io_path_ops =
+ globalStats.buffers.ops[msg->backend_type].io_path_ops;
+
+ for (io_path = 0; io_path < IOPATH_NUM_TYPES; io_path++)
+ {
+ PgStatIOOps *src = &src_io_path_ops[io_path];
+ PgStatIOOps *dest = &dest_io_path_ops[io_path];
+
+ dest->allocs += src->allocs;
+ dest->extends += src->extends;
+ dest->fsyncs += src->fsyncs;
+ dest->writes += src->writes;
+ }
+}
+
/* ----------
* pgstat_recv_wal() -
*
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 08ebabfe96..6926fc5742 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -480,7 +480,7 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr,
BlockNumber blockNum,
BufferAccessStrategy strategy,
bool *foundPtr);
-static void FlushBuffer(BufferDesc *buf, SMgrRelation reln);
+static void FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOPath iopath);
static void FindAndDropRelFileNodeBuffers(RelFileNode rnode,
ForkNumber forkNum,
BlockNumber nForkBlock,
@@ -972,6 +972,7 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
if (isExtend)
{
+ pgstat_inc_ioop(IOOP_EXTEND, IOPATH_SHARED);
/* new buffers are zero-filled */
MemSet((char *) bufBlock, 0, BLCKSZ);
/* don't set checksum for all-zero page */
@@ -1172,6 +1173,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
/* Loop here in case we have to try another victim buffer */
for (;;)
{
+ bool from_ring;
+
/*
* Ensure, while the spinlock's not yet held, that there's a free
* refcount entry.
@@ -1182,7 +1185,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* Select a victim buffer. The buffer is returned with its header
* spinlock still held!
*/
- buf = StrategyGetBuffer(strategy, &buf_state);
+ buf = StrategyGetBuffer(strategy, &buf_state, &from_ring);
Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 0);
@@ -1219,6 +1222,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
if (LWLockConditionalAcquire(BufferDescriptorGetContentLock(buf),
LW_SHARED))
{
+ IOPath iopath;
+
/*
* If using a nondefault strategy, and writing the buffer
* would require a WAL flush, let the strategy decide whether
@@ -1236,7 +1241,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
UnlockBufHdr(buf, buf_state);
if (XLogNeedsFlush(lsn) &&
- StrategyRejectBuffer(strategy, buf))
+ StrategyRejectBuffer(strategy, buf, &from_ring))
{
/* Drop lock/pin and loop around for another buffer */
LWLockRelease(BufferDescriptorGetContentLock(buf));
@@ -1245,13 +1250,26 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
}
}
+ /*
+ * When a strategy is in use, if the dirty buffer was selected
+ * from the strategy ring and we did not bother checking the
+ * freelist or doing a clock sweep to look for a clean shared
+ * buffer to use, the write will be counted as a strategy
+ * write. However, if the dirty buffer was obtained from the
+ * freelist or a clock sweep, it is counted as a regular
+ * write. When a strategy is not in use, at this point, the
+ * write can only be a "regular" write of a dirty buffer.
+ */
+
+ iopath = from_ring ? IOPATH_STRATEGY : IOPATH_SHARED;
+
/* OK, do the I/O */
TRACE_POSTGRESQL_BUFFER_WRITE_DIRTY_START(forkNum, blockNum,
smgr->smgr_rnode.node.spcNode,
smgr->smgr_rnode.node.dbNode,
smgr->smgr_rnode.node.relNode);
- FlushBuffer(buf, NULL);
+ FlushBuffer(buf, NULL, iopath);
LWLockRelease(BufferDescriptorGetContentLock(buf));
ScheduleBufferTagForWriteback(&BackendWritebackContext,
@@ -2552,10 +2570,11 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
* Pin it, share-lock it, write it. (FlushBuffer will do nothing if the
* buffer is clean by the time we've locked it.)
*/
+
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOPATH_SHARED);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
@@ -2803,9 +2822,12 @@ BufferGetTag(Buffer buffer, RelFileNode *rnode, ForkNumber *forknum,
*
* If the caller has an smgr reference for the buffer's relation, pass it
* as the second parameter. If not, pass NULL.
+ *
+ * IOPath will always be IOPATH_SHARED except when a buffer access strategy is
+ * used and the buffer being flushed is a buffer from the strategy ring.
*/
static void
-FlushBuffer(BufferDesc *buf, SMgrRelation reln)
+FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOPath iopath)
{
XLogRecPtr recptr;
ErrorContextCallback errcallback;
@@ -2897,6 +2919,8 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
bufToWrite,
false);
+ pgstat_inc_ioop(IOOP_WRITE, iopath);
+
if (track_io_timing)
{
INSTR_TIME_SET_CURRENT(io_time);
@@ -3544,6 +3568,8 @@ FlushRelationBuffers(Relation rel)
localpage,
false);
+ pgstat_inc_ioop(IOOP_WRITE, IOPATH_LOCAL);
+
buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED);
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
@@ -3579,7 +3605,7 @@ FlushRelationBuffers(Relation rel)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, RelationGetSmgr(rel));
+ FlushBuffer(bufHdr, RelationGetSmgr(rel), IOPATH_SHARED);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr, true);
}
@@ -3675,7 +3701,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, srelent->srel);
+ FlushBuffer(bufHdr, srelent->srel, IOPATH_SHARED);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr, true);
}
@@ -3731,7 +3757,7 @@ FlushDatabaseBuffers(Oid dbid)
{
PinBuffer_Locked(bufHdr);
LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED);
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOPATH_SHARED);
LWLockRelease(BufferDescriptorGetContentLock(bufHdr));
UnpinBuffer(bufHdr, true);
}
@@ -3758,7 +3784,7 @@ FlushOneBuffer(Buffer buffer)
Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr)));
- FlushBuffer(bufHdr, NULL);
+ FlushBuffer(bufHdr, NULL, IOPATH_SHARED);
}
/*
diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c
index 6be80476db..c7ca8d75aa 100644
--- a/src/backend/storage/buffer/freelist.c
+++ b/src/backend/storage/buffer/freelist.c
@@ -19,6 +19,7 @@
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
#include "storage/proc.h"
+#include "utils/backend_status.h"
#define INT_ACCESS_ONCE(var) ((int)(*((volatile int *)&(var))))
@@ -198,7 +199,7 @@ have_free_buffer(void)
* return the buffer with the buffer header spinlock still held.
*/
BufferDesc *
-StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
+StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring)
{
BufferDesc *buf;
int bgwprocno;
@@ -212,7 +213,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
if (strategy != NULL)
{
buf = GetBufferFromRing(strategy, buf_state);
- if (buf != NULL)
+ *from_ring = buf != NULL;
+ if (*from_ring)
return buf;
}
@@ -247,6 +249,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state)
* the rate of buffer consumption. Note that buffers recycled by a
* strategy object are intentionally not counted here.
*/
+ pgstat_inc_ioop(IOOP_ALLOC, IOPATH_SHARED);
pg_atomic_fetch_add_u32(&StrategyControl->numBufferAllocs, 1);
/*
@@ -683,8 +686,14 @@ AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf)
* if this buffer should be written and re-used.
*/
bool
-StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf)
+StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool *from_ring)
{
+ /*
+ * Start by assuming that we will use the dirty buffer selected by
+ * StrategyGetBuffer().
+ */
+ *from_ring = true;
+
/* We only do this in bulkread mode */
if (strategy->btype != BAS_BULKREAD)
return false;
@@ -700,5 +709,13 @@ StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf)
*/
strategy->buffers[strategy->current] = InvalidBuffer;
+ /*
+ * Since we will not be writing out a dirty buffer from the ring, set
+ * from_ring to false so that the caller does not count this write as a
+ * "strategy write" and can do proper bookkeeping.
+ */
+ *from_ring = false;
+
+
return true;
}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea3..f396a2b68d 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -20,6 +20,7 @@
#include "executor/instrument.h"
#include "storage/buf_internals.h"
#include "storage/bufmgr.h"
+#include "utils/backend_status.h"
#include "utils/guc.h"
#include "utils/memutils.h"
#include "utils/resowner_private.h"
@@ -226,6 +227,8 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
localpage,
false);
+ pgstat_inc_ioop(IOOP_WRITE, IOPATH_LOCAL);
+
/* Mark not-dirty now in case we error out below */
buf_state &= ~BM_DIRTY;
pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state);
diff --git a/src/backend/storage/sync/sync.c b/src/backend/storage/sync/sync.c
index d4083e8a56..3be06d5d5a 100644
--- a/src/backend/storage/sync/sync.c
+++ b/src/backend/storage/sync/sync.c
@@ -420,6 +420,7 @@ ProcessSyncRequests(void)
total_elapsed += elapsed;
processed++;
+ pgstat_inc_ioop(IOOP_FSYNC, IOPATH_SHARED);
if (log_checkpoints)
elog(DEBUG1, "checkpoint sync: number=%d file=%s time=%.3f ms",
processed,
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 7229598822..4662ea2e24 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -50,7 +50,7 @@ int pgstat_track_activity_query_size = 1024;
PgBackendStatus *MyBEEntry = NULL;
-static PgBackendStatus *BackendStatusArray = NULL;
+PgBackendStatus *BackendStatusArray = NULL;
static char *BackendAppnameBuffer = NULL;
static char *BackendClientHostnameBuffer = NULL;
static char *BackendActivityBuffer = NULL;
@@ -236,6 +236,24 @@ CreateSharedBackendStatus(void)
#endif
}
+const char *
+GetIOPathDesc(IOPath io_path)
+{
+
+ switch (io_path)
+ {
+ case IOPATH_DIRECT:
+ return "direct";
+ case IOPATH_LOCAL:
+ return "local";
+ case IOPATH_SHARED:
+ return "shared";
+ case IOPATH_STRATEGY:
+ return "strategy";
+ }
+ return "unknown IO path";
+}
+
/*
* Initialize pgstats backend activity state, and set up our on-proc-exit
* hook. Called from InitPostgres and AuxiliaryProcessMain. For auxiliary
@@ -279,7 +297,7 @@ pgstat_beinit(void)
* pgstat_bestart() -
*
* Initialize this backend's entry in the PgBackendStatus array.
- * Called from InitPostgres.
+ * Called from InitPostgres and AuxiliaryProcessMain
*
* Apart from auxiliary processes, MyBackendId, MyDatabaseId,
* session userid, and application_name must be set for a
@@ -293,6 +311,7 @@ pgstat_bestart(void)
{
volatile PgBackendStatus *vbeentry = MyBEEntry;
PgBackendStatus lbeentry;
+ int io_path;
#ifdef USE_SSL
PgBackendSSLStatus lsslstatus;
#endif
@@ -399,6 +418,15 @@ pgstat_bestart(void)
lbeentry.st_progress_command = PROGRESS_COMMAND_INVALID;
lbeentry.st_progress_command_target = InvalidOid;
lbeentry.st_query_id = UINT64CONST(0);
+ for (io_path = 0; io_path < IOPATH_NUM_TYPES; io_path++)
+ {
+ IOOps *io_ops = &lbeentry.io_path_stats[io_path];
+
+ pg_atomic_init_u64(&io_ops->allocs, 0);
+ pg_atomic_init_u64(&io_ops->extends, 0);
+ pg_atomic_init_u64(&io_ops->fsyncs, 0);
+ pg_atomic_init_u64(&io_ops->writes, 0);
+ }
/*
* we don't zero st_progress_param here to save cycles; nobody should
@@ -621,6 +649,33 @@ pgstat_report_activity(BackendState state, const char *cmd_str)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Iterate through BackendStatusArray and capture live backends' stats on IO
+ * Ops for all IO Paths, adding them to that backend type's member of the
+ * backend_io_path_ops structure.
+ */
+void
+pgstat_report_live_backend_io_path_ops(PgStatIOPathOps *backend_io_path_ops)
+{
+ int i;
+ PgBackendStatus *beentry = BackendStatusArray;
+
+ /*
+ * Loop through live backends and capture reset values
+ */
+ for (i = 0; i < MaxBackends + NUM_AUXPROCTYPES; i++)
+ {
+ beentry++;
+ /* Don't count dead backends. They should already be counted */
+ if (beentry->st_procpid == 0)
+ continue;
+
+ pgstat_sum_io_path_ops(backend_io_path_ops[beentry->st_backendType].io_path_ops,
+ (IOOps *) beentry->io_path_stats);
+
+ }
+}
+
/* --------
* pgstat_report_query_id() -
*
@@ -1045,7 +1100,6 @@ pgstat_get_my_query_id(void)
return MyBEEntry->st_query_id;
}
-
/* ----------
* pgstat_fetch_stat_beentry() -
*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index ff5aedc99c..734079e233 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -1778,6 +1778,37 @@ pg_stat_get_bgwriter_stat_reset_time(PG_FUNCTION_ARGS)
PG_RETURN_TIMESTAMPTZ(pgstat_fetch_stat_bgwriter()->stat_reset_timestamp);
}
+/*
+* When adding a new column to the pg_stat_buffers view, add a new enum
+* value here above COLUMN_LENGTH.
+*/
+enum
+{
+ COLUMN_BACKEND_TYPE,
+ COLUMN_IO_PATH,
+ COLUMN_ALLOCS,
+ COLUMN_EXTENDS,
+ COLUMN_FSYNCS,
+ COLUMN_WRITES,
+ COLUMN_RESET_TIME,
+ COLUMN_LENGTH,
+};
+
+#define NROWS ((BACKEND_NUM_TYPES - 1) * IOPATH_NUM_TYPES)
+/*
+ * Helper function to get the correct row in the pg_stat_buffers view.
+ */
+static inline Datum *
+get_pg_stat_buffers_row(Datum all_values[NROWS][COLUMN_LENGTH], BackendType backend_type, IOPath io_path)
+{
+
+ /*
+ * Subtract 1 from backend_type to avoid having rows for B_INVALID
+ * BackendType
+ */
+ return all_values[(backend_type - 1) * IOPATH_NUM_TYPES + io_path];
+}
+
Datum
pg_stat_get_buf_written_backend(PG_FUNCTION_ARGS)
{
@@ -1796,6 +1827,127 @@ pg_stat_get_buf_alloc(PG_FUNCTION_ARGS)
PG_RETURN_INT64(pgstat_fetch_stat_bgwriter()->buf_alloc);
}
+Datum
+pg_stat_get_buffers(PG_FUNCTION_ARGS)
+{
+ PgStat_BackendIOPathOps *backend_io_path_ops;
+ PgBackendStatus *beentry;
+ int backend_type, io_path;
+ int i;
+ Datum reset_time;
+
+ ReturnSetInfo *rsinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+
+ Datum all_values[NROWS][COLUMN_LENGTH];
+ bool all_nulls[NROWS][COLUMN_LENGTH];
+
+ rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+
+ /* check to see if caller supports us returning a tuplestore */
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+ MemoryContextSwitchTo(oldcontext);
+
+ memset(all_values, 0, sizeof(all_values));
+ memset(all_nulls, 0, sizeof(all_nulls));
+
+ /*
+ * Loop through all live backends and count their IO Ops for each IO Path
+ */
+ beentry = BackendStatusArray;
+
+ for (i = 0; i < MaxBackends + NUM_AUXPROCTYPES; i++)
+ {
+ IOOps *io_ops;
+
+ beentry++;
+ /* Don't count dead backends. They should already be counted */
+ if (beentry->st_procpid == 0)
+ continue;
+
+ io_ops = beentry->io_path_stats;
+
+ for (io_path = 0; io_path < IOPATH_NUM_TYPES; io_path++)
+ {
+ Datum *values = get_pg_stat_buffers_row(all_values, beentry->st_backendType, io_path);
+
+ /*
+ * COLUMN_RESET_TIME, COLUMN_BACKEND_TYPE, and COLUMN_IO_PATH will
+ * all be set when looping through exited backends array
+ */
+ values[COLUMN_ALLOCS] += pg_atomic_read_u64(&io_ops->allocs);
+ values[COLUMN_EXTENDS] += pg_atomic_read_u64(&io_ops->extends);
+ values[COLUMN_FSYNCS] += pg_atomic_read_u64(&io_ops->fsyncs);
+ values[COLUMN_WRITES] += pg_atomic_read_u64(&io_ops->writes);
+ io_ops++;
+ }
+ }
+
+ /* Add stats from all exited backends */
+ backend_io_path_ops = pgstat_fetch_exited_backend_buffers();
+
+ reset_time = TimestampTzGetDatum(backend_io_path_ops->stat_reset_timestamp);
+
+ /* 0 is not a valid BackendType */
+ for (backend_type = 1; backend_type < BACKEND_NUM_TYPES; backend_type++)
+ {
+ PgStatIOOps *io_ops = backend_io_path_ops->ops[backend_type].io_path_ops;
+ PgStatIOOps *resets = backend_io_path_ops->resets[backend_type].io_path_ops;
+
+ Datum backend_type_desc = CStringGetTextDatum(GetBackendTypeDesc(backend_type));
+
+ for (io_path = 0; io_path < IOPATH_NUM_TYPES; io_path++)
+ {
+ Datum *values = get_pg_stat_buffers_row(all_values, backend_type, io_path);
+
+ values[COLUMN_BACKEND_TYPE] = backend_type_desc;
+ values[COLUMN_IO_PATH] = CStringGetTextDatum(GetIOPathDesc(io_path));
+ values[COLUMN_ALLOCS] = values[COLUMN_ALLOCS] + io_ops->allocs - resets->allocs;
+ values[COLUMN_EXTENDS] = values[COLUMN_EXTENDS] + io_ops->extends - resets->extends;
+ values[COLUMN_FSYNCS] = values[COLUMN_FSYNCS] + io_ops->fsyncs - resets->fsyncs;
+ values[COLUMN_WRITES] = values[COLUMN_WRITES] + io_ops->writes - resets->writes;
+ values[COLUMN_RESET_TIME] = reset_time;
+ io_ops++;
+ resets++;
+ }
+ }
+
+ for (i = 0; i < NROWS; i++)
+ {
+ Datum *values = all_values[i];
+ bool *nulls = all_nulls[i];
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* clean up and return the tuplestore */
+ tuplestore_donestoring(tupstore);
+
+ return (Datum) 0;
+}
+
/*
* Returns statistics of WAL activity
*/
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d068d6532e..bbdb07b222 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5642,6 +5642,15 @@
proname => 'pg_stat_get_buf_alloc', provolatile => 's', proparallel => 'r',
prorettype => 'int8', proargtypes => '', prosrc => 'pg_stat_get_buf_alloc' },
+{ oid => '8459', descr => 'statistics: counts of all IO operations done to all IO paths by each type of backend.',
+ proname => 'pg_stat_get_buffers', provolatile => 's', proisstrict => 'f',
+ prorows => '52', proretset => 't',
+ proparallel => 'r', prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,text,int8,int8,int8,int8,timestamptz}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{backend_type,io_path,alloc,extend,fsync,write,stats_reset}',
+ prosrc => 'pg_stat_get_buffers' },
+
{ oid => '1136', descr => 'statistics: information about WAL activity',
proname => 'pg_stat_get_wal', proisstrict => 'f', provolatile => 's',
proparallel => 'r', prorettype => 'record', proargtypes => '',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 90a3016065..6785fb3813 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -338,6 +338,8 @@ typedef enum BackendType
B_LOGGER,
} BackendType;
+#define BACKEND_NUM_TYPES (B_LOGGER + 1)
+
extern BackendType MyBackendType;
extern const char *GetBackendTypeDesc(BackendType backendType);
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index bcd3588ea2..ac8aca2c61 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -72,6 +72,7 @@ typedef enum StatMsgType
PGSTAT_MTYPE_ARCHIVER,
PGSTAT_MTYPE_BGWRITER,
PGSTAT_MTYPE_CHECKPOINTER,
+ PGSTAT_MTYPE_IO_PATH_OPS,
PGSTAT_MTYPE_WAL,
PGSTAT_MTYPE_SLRU,
PGSTAT_MTYPE_FUNCSTAT,
@@ -138,6 +139,7 @@ typedef enum PgStat_Shared_Reset_Target
{
RESET_ARCHIVER,
RESET_BGWRITER,
+ RESET_BUFFERS,
RESET_WAL
} PgStat_Shared_Reset_Target;
@@ -331,6 +333,51 @@ typedef struct PgStat_MsgDropdb
} PgStat_MsgDropdb;
+/*
+ * Structure for counting all types of IO ops in the stats collector
+ */
+typedef struct PgStatIOOps
+{
+ PgStat_Counter allocs;
+ PgStat_Counter extends;
+ PgStat_Counter fsyncs;
+ PgStat_Counter writes;
+} PgStatIOOps;
+
+/*
+ * Structure for counting all IO ops on all types of buffers.
+ */
+typedef struct PgStatIOPathOps
+{
+ PgStatIOOps io_path_ops[IOPATH_NUM_TYPES];
+} PgStatIOPathOps;
+
+/*
+ * Sent by a backend to the stats collector to report all IO Ops for all IO
+ * Paths for a given type of a backend. This will happen when the backend exits
+ * or when stats are reset.
+ */
+typedef struct PgStat_MsgIOPathOps
+{
+ PgStat_MsgHdr m_hdr;
+
+ BackendType backend_type;
+ PgStatIOPathOps iop;
+} PgStat_MsgIOPathOps;
+
+/*
+ * Structure used by stats collector to keep track of all types of exited
+ * backends' IO Ops for all IO Paths as well as all stats from live backends at
+ * the time of stats reset. resets is populated using a reset message sent to
+ * the stats collector.
+ */
+typedef struct PgStat_BackendIOPathOps
+{
+ TimestampTz stat_reset_timestamp;
+ PgStatIOPathOps ops[BACKEND_NUM_TYPES];
+ PgStatIOPathOps resets[BACKEND_NUM_TYPES];
+} PgStat_BackendIOPathOps;
+
/* ----------
* PgStat_MsgResetcounter Sent by the backend to tell the collector
* to reset counters
@@ -351,6 +398,7 @@ typedef struct PgStat_MsgResetsharedcounter
{
PgStat_MsgHdr m_hdr;
PgStat_Shared_Reset_Target m_resettarget;
+ PgStat_MsgIOPathOps m_backend_resets;
} PgStat_MsgResetsharedcounter;
/* ----------
@@ -703,6 +751,7 @@ typedef union PgStat_Msg
PgStat_MsgArchiver msg_archiver;
PgStat_MsgBgWriter msg_bgwriter;
PgStat_MsgCheckpointer msg_checkpointer;
+ PgStat_MsgIOPathOps msg_io_path_ops;
PgStat_MsgWal msg_wal;
PgStat_MsgSLRU msg_slru;
PgStat_MsgFuncstat msg_funcstat;
@@ -879,6 +928,7 @@ typedef struct PgStat_GlobalStats
PgStat_CheckpointerStats checkpointer;
PgStat_BgWriterStats bgwriter;
+ PgStat_BackendIOPathOps buffers;
} PgStat_GlobalStats;
/*
@@ -1118,14 +1168,17 @@ extern void pgstat_twophase_postabort(TransactionId xid, uint16 info,
extern void pgstat_send_archiver(const char *xlog, bool failed);
extern void pgstat_send_bgwriter(void);
+extern void pgstat_send_buffers(void);
extern void pgstat_send_checkpointer(void);
extern void pgstat_send_wal(bool force);
+extern void pgstat_sum_io_path_ops(PgStatIOOps *dest, IOOps *src);
/* ----------
* Support functions for the SQL-callable functions to
* generate the pgstat* views.
* ----------
*/
+extern PgStat_BackendIOPathOps *pgstat_fetch_exited_backend_buffers(void);
extern PgStat_StatDBEntry *pgstat_fetch_stat_dbentry(Oid dbid);
extern PgStat_StatTabEntry *pgstat_fetch_stat_tabentry(Oid relid);
extern PgStat_StatFuncEntry *pgstat_fetch_stat_funcentry(Oid funcid);
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 33fcaf5c9a..7e385135db 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -310,10 +310,10 @@ extern void ScheduleBufferTagForWriteback(WritebackContext *context, BufferTag *
/* freelist.c */
extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy,
- uint32 *buf_state);
+ uint32 *buf_state, bool *from_ring);
extern void StrategyFreeBuffer(BufferDesc *buf);
extern bool StrategyRejectBuffer(BufferAccessStrategy strategy,
- BufferDesc *buf);
+ BufferDesc *buf, bool *from_ring);
extern int StrategySyncStart(uint32 *complete_passes, uint32 *num_buf_alloc);
extern void StrategyNotifyBgWriter(int bgwprocno);
diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h
index 8042b817df..2b99195e7b 100644
--- a/src/include/utils/backend_status.h
+++ b/src/include/utils/backend_status.h
@@ -13,6 +13,7 @@
#include "datatype/timestamp.h"
#include "libpq/pqcomm.h"
#include "miscadmin.h" /* for BackendType */
+#include "port/atomics.h"
#include "utils/backend_progress.h"
@@ -31,12 +32,48 @@ typedef enum BackendState
STATE_DISABLED
} BackendState;
+/* ----------
+ * IO Stats reporting utility types
+ * ----------
+ */
+
+typedef enum IOOp
+{
+ IOOP_ALLOC,
+ IOOP_EXTEND,
+ IOOP_FSYNC,
+ IOOP_WRITE,
+} IOOp;
+
+#define IOOP_NUM_TYPES (IOOP_WRITE + 1)
+
+typedef enum IOPath
+{
+ IOPATH_DIRECT,
+ IOPATH_LOCAL,
+ IOPATH_SHARED,
+ IOPATH_STRATEGY,
+} IOPath;
+
+#define IOPATH_NUM_TYPES (IOPATH_STRATEGY + 1)
+
/* ----------
* Shared-memory data structures
* ----------
*/
+/*
+ * Structure for counting all types of IOOps for a live backend.
+ */
+typedef struct IOOps
+{
+ pg_atomic_uint64 allocs;
+ pg_atomic_uint64 extends;
+ pg_atomic_uint64 fsyncs;
+ pg_atomic_uint64 writes;
+} IOOps;
+
/*
* PgBackendSSLStatus
*
@@ -168,6 +205,16 @@ typedef struct PgBackendStatus
/* query identifier, optionally computed using post_parse_analyze_hook */
uint64 st_query_id;
+
+ /*
+ * Stats on all IO Ops for all IO Paths for this backend. When the
+ * pg_stat_buffers view is queried and when stats are reset, one backend
+ * will read io_path_stats from all live backends and combine them with
+ * io_path_stats from exited backends for each backend type. When this
+ * backend exits, it will send io_path_stats to the stats collector to be
+ * persisted.
+ */
+ IOOps io_path_stats[IOPATH_NUM_TYPES];
} PgBackendStatus;
@@ -274,6 +321,7 @@ extern PGDLLIMPORT int pgstat_track_activity_query_size;
* ----------
*/
extern PGDLLIMPORT PgBackendStatus *MyBEEntry;
+extern PGDLLIMPORT PgBackendStatus *BackendStatusArray;
/* ----------
@@ -289,6 +337,10 @@ extern void CreateSharedBackendStatus(void);
* ----------
*/
+/* Utility functions */
+extern const char *GetIOPathDesc(IOPath io_path);
+
+
/* Initialization functions */
extern void pgstat_beinit(void);
extern void pgstat_bestart(void);
@@ -296,7 +348,35 @@ extern void pgstat_bestart(void);
extern void pgstat_clear_backend_activity_snapshot(void);
/* Activity reporting functions */
+typedef struct PgStatIOPathOps PgStatIOPathOps;
+
+static inline void
+pgstat_inc_ioop(IOOp io_op, IOPath io_path)
+{
+ IOOps *io_ops;
+ PgBackendStatus *beentry = MyBEEntry;
+
+ Assert(beentry);
+
+ io_ops = &beentry->io_path_stats[io_path];
+ switch (io_op)
+ {
+ case IOOP_ALLOC:
+ inc_counter(&io_ops->allocs);
+ break;
+ case IOOP_EXTEND:
+ inc_counter(&io_ops->extends);
+ break;
+ case IOOP_FSYNC:
+ inc_counter(&io_ops->fsyncs);
+ break;
+ case IOOP_WRITE:
+ inc_counter(&io_ops->writes);
+ break;
+ }
+}
extern void pgstat_report_activity(BackendState state, const char *cmd_str);
+extern void pgstat_report_live_backend_io_path_ops(PgStatIOPathOps *backend_io_path_ops);
extern void pgstat_report_query_id(uint64 query_id, bool force);
extern void pgstat_report_tempfile(size_t filesize);
extern void pgstat_report_appname(const char *appname);
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2fa00a3c29..5e5a0324ee 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1828,6 +1828,14 @@ pg_stat_bgwriter| SELECT pg_stat_get_bgwriter_timed_checkpoints() AS checkpoints
pg_stat_get_buf_fsync_backend() AS buffers_backend_fsync,
pg_stat_get_buf_alloc() AS buffers_alloc,
pg_stat_get_bgwriter_stat_reset_time() AS stats_reset;
+pg_stat_buffers| SELECT b.backend_type,
+ b.io_path,
+ b.alloc,
+ b.extend,
+ b.fsync,
+ b.write,
+ b.stats_reset
+ FROM pg_stat_get_buffers() b(backend_type, io_path, alloc, extend, fsync, write, stats_reset);
pg_stat_database| SELECT d.oid AS datid,
d.datname,
CASE
--
2.30.2
