On Fri, Mar 10, 2023 at 9:54 AM Michael Paquier <mich...@paquier.xyz> wrote: > > Hmm. I think this patch ought to have a result simpler than what's > proposed here. > > First, do we really have to begin marking the functions as non-STRICT > to abide with the treatment of NULL as a special value? The part that > I've found personally the most annoying with these functions is that > an incorrect bound leads to a random failure, particularly when such > queries are used for monitoring. I would simplify the whole with two > simple rules, as of: > - Keeping all the functions strict. > - When end_lsn is a LSN in the future of the current LSN inserted or > replayed, adjust its value to be the exactly GetXLogReplayRecPtr() or > GetFlushRecPtr(). This way, monitoring tools can use a value ahead, > at will. > - Failing if start_lsn > end_lsn. > - Failing if start_lsn refers to a position older than what exists is > still fine by me.
Done this way in the attached v5 patch. > I would also choose to remove > pg_get_wal_records_info_till_end_of_wal() from the SQL interface in > 1.1 to limit the confusion arount it, but keep a few lines of code so > as we are still able to use it when pg_walinspect 1.0 is the version > enabled with CREATE EXTENSION. > > In short, pg_get_wal_records_info_till_end_of_wal() should be able to > use exactly the same code as pg_get_wal_records_info(), still you need > to keep *two* functions for their prosrc with PG_FUNCTION_ARGS as > arguments so as 1.0 would work when dropped in place. The result, it > seems to me, mostly comes to simplify ValidateInputLSNs() and remove > its till_end_of_wal argument. This has already been taken care of in the previous patches, e.g. v3 and v4 and so in the latest v5 patch. > +-- Removed function > +SELECT > pg_get_functiondef('pg_get_wal_records_info_till_end_of_wal'::regproc); > +ERROR: function "pg_get_wal_records_info_till_end_of_wal" does not exist > +LINE 1: SELECT pg_get_functiondef('pg_get_wal_records_info_till_end_... > > It seems to me that you should just replace all that and anything > depending on pg_get_functiondef() with a \dx+ pg_walinspect, that > would list all the objects part of the extension for the specific > version you want to test. Not sure that there is a need to list the > full function definitions, either. That just bloats the tests. Agreed and used \dx+. One can anyways look at the function definitions and compare for knowing what's changed. > I think, however, that it is critical to test in oldextversions.out > the *executions* of the functions, so as we make sure that they don't > crash. The patch is missing that. You mean, we need to test the till_end_of_wal functions that were removed in the latest version 1.1 but they must work if the extension is installed with 1.0? If yes, I now added them. > +-- Invalid input LSNs > +SELECT * FROM pg_get_wal_record_info('0/0'); -- ERROR > +ERROR: invalid input LSN Removed InvalidRecPtr checks for input/start LSN because anyways the functions will fail with ERROR: could not read WAL at LSN 0/0. Any comments on the attached v5 patch? -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
From 63360cfa8c2a3141a1360a966bc7e210eb57810b Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Date: Fri, 10 Mar 2023 10:54:20 +0000 Subject: [PATCH v5] Rework pg_walinspect functions --- contrib/pg_walinspect/Makefile | 2 +- .../pg_walinspect/expected/oldextversions.out | 55 +++++ .../pg_walinspect/expected/pg_walinspect.out | 60 ++++- contrib/pg_walinspect/meson.build | 1 + .../pg_walinspect/pg_walinspect--1.0--1.1.sql | 4 + contrib/pg_walinspect/pg_walinspect.c | 213 ++++++++---------- contrib/pg_walinspect/sql/oldextversions.sql | 29 +++ contrib/pg_walinspect/sql/pg_walinspect.sql | 74 +++--- doc/src/sgml/pgwalinspect.sgml | 88 ++++---- 9 files changed, 307 insertions(+), 219 deletions(-) create mode 100644 contrib/pg_walinspect/expected/oldextversions.out create mode 100644 contrib/pg_walinspect/sql/oldextversions.sql diff --git a/contrib/pg_walinspect/Makefile b/contrib/pg_walinspect/Makefile index 7033878a79..22090f7716 100644 --- a/contrib/pg_walinspect/Makefile +++ b/contrib/pg_walinspect/Makefile @@ -9,7 +9,7 @@ PGFILEDESC = "pg_walinspect - functions to inspect contents of PostgreSQL Write- EXTENSION = pg_walinspect DATA = pg_walinspect--1.0.sql pg_walinspect--1.0--1.1.sql -REGRESS = pg_walinspect +REGRESS = pg_walinspect oldextversions REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_walinspect/walinspect.conf diff --git a/contrib/pg_walinspect/expected/oldextversions.out b/contrib/pg_walinspect/expected/oldextversions.out new file mode 100644 index 0000000000..4ad85392f0 --- /dev/null +++ b/contrib/pg_walinspect/expected/oldextversions.out @@ -0,0 +1,55 @@ +-- test old extension version entry points +DROP EXTENSION pg_walinspect; +CREATE EXTENSION pg_walinspect WITH VERSION '1.0'; +-- List what version 1.0 contains +\dx+ pg_walinspect + Objects in extension "pg_walinspect" + Object description +----------------------------------------------------------- + function pg_get_wal_record_info(pg_lsn) + function pg_get_wal_records_info(pg_lsn,pg_lsn) + function pg_get_wal_records_info_till_end_of_wal(pg_lsn) + function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) + function pg_get_wal_stats_till_end_of_wal(pg_lsn,boolean) +(5 rows) + +-- =================================================================== +-- Tests to check if the removed functions work when older version of +-- extension is explicitly installed. +-- =================================================================== +-- Make sure checkpoints don't interfere with the test. +SELECT 'init' FROM pg_create_physical_replication_slot('regress_pg_walinspect_slot', true, false); + ?column? +---------- + init +(1 row) + +CREATE TABLE sample_tbl(col1 int, col2 int); +SELECT pg_current_wal_lsn() AS wal_lsn1 \gset +INSERT INTO sample_tbl SELECT * FROM generate_series(1, 2); +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info_till_end_of_wal(:'wal_lsn1'); + ok +---- + t +(1 row) + +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1'); + ok +---- + t +(1 row) + +-- Move to new version 1.1 +ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; +-- List what version 1.1 contains +\dx+ pg_walinspect + Objects in extension "pg_walinspect" + Object description +-------------------------------------------------- + function pg_get_wal_block_info(pg_lsn,pg_lsn) + function pg_get_wal_record_info(pg_lsn) + function pg_get_wal_records_info(pg_lsn,pg_lsn) + function pg_get_wal_stats(pg_lsn,pg_lsn,boolean) +(4 rows) + +DROP EXTENSION pg_walinspect; diff --git a/contrib/pg_walinspect/expected/pg_walinspect.out b/contrib/pg_walinspect/expected/pg_walinspect.out index e0eb7ca08f..d3ddc2409c 100644 --- a/contrib/pg_walinspect/expected/pg_walinspect.out +++ b/contrib/pg_walinspect/expected/pg_walinspect.out @@ -1,11 +1,12 @@ CREATE EXTENSION pg_walinspect; --- Make sure checkpoints don't interfere with the test. +-- Make sure checkpoints don't interfere with the test SELECT 'init' FROM pg_create_physical_replication_slot('regress_pg_walinspect_slot', true, false); ?column? ---------- init (1 row) +-- Generate WAL and captures LSNs as needed CREATE TABLE sample_tbl(col1 int, col2 int); SELECT pg_current_wal_lsn() AS wal_lsn1 \gset INSERT INTO sample_tbl SELECT * FROM generate_series(1, 2); @@ -14,38 +15,55 @@ INSERT INTO sample_tbl SELECT * FROM generate_series(3, 4); -- =================================================================== -- Tests for input validation -- =================================================================== -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info(:'wal_lsn2', :'wal_lsn1'); -- ERROR +-- Invalid input LSN or LSN that server doesn't have WAL for +SELECT * FROM pg_get_wal_record_info('0/0'); -- ERROR +ERROR: could not read WAL at LSN 0/0 +-- Invalid start LSN or start LSN that server doesn't have WAL for +SELECT * FROM pg_get_wal_records_info('0/0', :'wal_lsn1'); -- ERROR +ERROR: could not read WAL at LSN 0/0 +SELECT * FROM pg_get_wal_stats('0/0', :'wal_lsn1'); -- ERROR +ERROR: could not read WAL at LSN 0/0 +SELECT * FROM pg_get_wal_block_info('0/0', :'wal_lsn1'); -- ERROR +ERROR: could not read WAL at LSN 0/0 +-- Start LSN >= End LSN +SELECT * FROM pg_get_wal_records_info(:'wal_lsn2', :'wal_lsn1'); -- ERROR ERROR: WAL start LSN must be less than end LSN -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats(:'wal_lsn2', :'wal_lsn1'); -- ERROR +SELECT * FROM pg_get_wal_stats(:'wal_lsn2', :'wal_lsn1'); -- ERROR +ERROR: WAL start LSN must be less than end LSN +SELECT * FROM pg_get_wal_block_info(:'wal_lsn2', :'wal_lsn1'); -- ERROR ERROR: WAL start LSN must be less than end LSN -- =================================================================== -- Tests for all function executions -- =================================================================== -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_record_info(:'wal_lsn1'); +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_record_info(:'wal_lsn1'); ok ---- t (1 row) -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2'); +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2'); ok ---- t (1 row) -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info_till_end_of_wal(:'wal_lsn1'); +-- Get info till end of WAL. End LSN is way higher, adjust it to the latest LSN +-- that server knows. +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', 'FFFFFFFF/FFFFFFFF'); ok ---- t (1 row) -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2'); +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2'); ok ---- t (1 row) -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1'); +-- Get info till end of WAL. End LSN is way higher, adjust it to the latest LSN +-- that server knows. +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats(:'wal_lsn1', 'FFFFFFFF/FFFFFFFF'); ok ---- t @@ -64,7 +82,7 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2' -- =================================================================== -- Test for filtering out WAL records based on resource_manager and --- record_type +-- record_type. -- =================================================================== SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2') WHERE resource_manager = 'Heap' AND record_type = 'INSERT'; @@ -80,7 +98,7 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2' SELECT pg_current_wal_lsn() AS wal_lsn3 \gset UPDATE sample_tbl SET col1 = col1 + 1 WHERE col1 = 1; SELECT pg_current_wal_lsn() AS wal_lsn4 \gset --- Check if we get block data from WAL record. +-- Check if we get block data from WAL record SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn3', :'wal_lsn4') WHERE relfilenode = :'sample_tbl_oid' AND blockdata IS NOT NULL; ok @@ -88,12 +106,21 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn3', :'wal_lsn4') t (1 row) --- Force full-page image on the next update. +-- Check till end of WAL if we get block data from WAL record. End LSN is way +-- higher, adjust it to the latest LSN that server knows. +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn3', 'FFFFFFFF/FFFFFFFF') + WHERE relfilenode = :'sample_tbl_oid'; + ok +---- + t +(1 row) + +-- Force full-page image on the next update SELECT pg_current_wal_lsn() AS wal_lsn5 \gset CHECKPOINT; UPDATE sample_tbl SET col1 = col1 + 1 WHERE col1 = 2; SELECT pg_current_wal_lsn() AS wal_lsn6 \gset --- Check if we get FPI from WAL record. +-- Check if we get FPI from WAL record SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn5', :'wal_lsn6') WHERE relfilenode = :'sample_tbl_oid' AND fpi IS NOT NULL; ok @@ -101,6 +128,15 @@ SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn5', :'wal_lsn6') t (1 row) +-- Check till end of WAL if we get FPI from WAL record. End LSN is way higher, +-- adjust it to the latest LSN that server knoows. +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn5', 'FFFFFFFF/FFFFFFFF') + WHERE relfilenode = :'sample_tbl_oid'; + ok +---- + t +(1 row) + -- =================================================================== -- Tests for permissions -- =================================================================== diff --git a/contrib/pg_walinspect/meson.build b/contrib/pg_walinspect/meson.build index bf7b79b1b7..80059f6119 100644 --- a/contrib/pg_walinspect/meson.build +++ b/contrib/pg_walinspect/meson.build @@ -30,6 +30,7 @@ tests += { 'regress': { 'sql': [ 'pg_walinspect', + 'oldextversions', ], # Disabled because these tests require "wal_level=replica", which # some runningcheck users do not have (e.g. buildfarm clients). diff --git a/contrib/pg_walinspect/pg_walinspect--1.0--1.1.sql b/contrib/pg_walinspect/pg_walinspect--1.0--1.1.sql index e674ef25aa..a9c9ddda0e 100644 --- a/contrib/pg_walinspect/pg_walinspect--1.0--1.1.sql +++ b/contrib/pg_walinspect/pg_walinspect--1.0--1.1.sql @@ -3,6 +3,10 @@ -- complain if script is sourced in psql, rather than via ALTER EXTENSION \echo Use "ALTER EXTENSION pg_walinspect UPDATE TO '1.1'" to load this file. \quit +/* Drop the unneeded functions */ +DROP FUNCTION pg_get_wal_records_info_till_end_of_wal(pg_lsn); +DROP FUNCTION pg_get_wal_stats_till_end_of_wal(pg_lsn, boolean); + -- -- pg_get_wal_block_info() -- diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index ee88dc4992..f604112623 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -38,15 +38,14 @@ PG_FUNCTION_INFO_V1(pg_get_wal_records_info_till_end_of_wal); PG_FUNCTION_INFO_V1(pg_get_wal_stats); PG_FUNCTION_INFO_V1(pg_get_wal_stats_till_end_of_wal); -static bool IsFutureLSN(XLogRecPtr lsn, XLogRecPtr *curr_lsn); +static XLogRecPtr GetCurrentLSN(void); static XLogReaderState *InitXLogReaderState(XLogRecPtr lsn); static XLogRecord *ReadNextXLogRecord(XLogReaderState *xlogreader); static void GetWALRecordInfo(XLogReaderState *record, Datum *values, bool *nulls, uint32 ncols); -static XLogRecPtr ValidateInputLSNs(bool till_end_of_wal, - XLogRecPtr start_lsn, XLogRecPtr end_lsn); -static void GetWALRecordsInfo(FunctionCallInfo fcinfo, XLogRecPtr start_lsn, - XLogRecPtr end_lsn); +static void GetInputLSNs(FunctionCallInfo fcinfo, XLogRecPtr *start_lsn, + XLogRecPtr *end_lsn, bool till_end_of_wal); +static void GetWALRecordsInfo(FunctionCallInfo fcinfo, bool till_end_of_wal); static void GetXLogSummaryStats(XLogStats *stats, ReturnSetInfo *rsinfo, Datum *values, bool *nulls, uint32 ncols, bool stats_per_record); @@ -55,32 +54,29 @@ static void FillXLogStatsRow(const char *name, uint64 n, uint64 total_count, uint64 fpi_len, uint64 total_fpi_len, uint64 tot_len, uint64 total_len, Datum *values, bool *nulls, uint32 ncols); -static void GetWalStats(FunctionCallInfo fcinfo, XLogRecPtr start_lsn, - XLogRecPtr end_lsn, bool stats_per_record); +static void GetWalStats(FunctionCallInfo fcinfo, bool till_end_of_wal); static void GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record); /* - * Check if the given LSN is in future. Also, return the LSN up to which the - * server has WAL. + * Return the LSN up to which the server has WAL. */ -static bool -IsFutureLSN(XLogRecPtr lsn, XLogRecPtr *curr_lsn) +static XLogRecPtr +GetCurrentLSN(void) { + XLogRecPtr curr_lsn; + /* * We determine the current LSN of the server similar to how page_read * callback read_local_xlog_page_no_wait does. */ if (!RecoveryInProgress()) - *curr_lsn = GetFlushRecPtr(NULL); + curr_lsn = GetFlushRecPtr(NULL); else - *curr_lsn = GetXLogReplayRecPtr(NULL); + curr_lsn = GetXLogReplayRecPtr(NULL); - Assert(!XLogRecPtrIsInvalid(*curr_lsn)); + Assert(!XLogRecPtrIsInvalid(curr_lsn)); - if (lsn >= *curr_lsn) - return true; - - return false; + return curr_lsn; } /* @@ -355,8 +351,13 @@ GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record) * to a record. Decompression is applied to the full page images, if * necessary. * - * This function emits an error if a future start or end WAL LSN i.e. WAL LSN - * the database system doesn't know about is specified. + * This function emits an error if the start LSN is in future i.e. LSN the + * database system doesn't know about. + * + * This function adjusts future end LSN to the last flushed LSN when not in + * recovery or the last replayed LSN when in recovery. + * + * This function emits an error, if start LSN is past the end LSN. */ Datum pg_get_wal_block_info(PG_FUNCTION_ARGS) @@ -367,10 +368,7 @@ pg_get_wal_block_info(PG_FUNCTION_ARGS) MemoryContext old_cxt; MemoryContext tmp_cxt; - start_lsn = PG_GETARG_LSN(0); - end_lsn = PG_GETARG_LSN(1); - - end_lsn = ValidateInputLSNs(false, start_lsn, end_lsn); + GetInputLSNs(fcinfo, &start_lsn, &end_lsn, false); InitMaterializedSRF(fcinfo, 0); @@ -405,8 +403,8 @@ pg_get_wal_block_info(PG_FUNCTION_ARGS) /* * Get WAL record info. * - * This function emits an error if a future WAL LSN i.e. WAL LSN the database - * system doesn't know about is specified. + * This function emits an error if input LSN is in future i.e. LSN the database + * system doesn't know about. */ Datum pg_get_wal_record_info(PG_FUNCTION_ARGS) @@ -422,20 +420,14 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) HeapTuple tuple; lsn = PG_GETARG_LSN(0); + curr_lsn = GetCurrentLSN(); - if (IsFutureLSN(lsn, &curr_lsn)) - { - /* - * GetFlushRecPtr or GetXLogReplayRecPtr gives "end+1" LSN of the last - * record flushed or replayed respectively. But let's use the LSN up - * to "end" in user facing message. - */ + if (lsn >= curr_lsn) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("cannot accept future input LSN"), errdetail("Last known WAL LSN on the database system is at %X/%X.", LSN_FORMAT_ARGS(curr_lsn)))); - } /* Build a tuple descriptor for our result type. */ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) @@ -462,54 +454,54 @@ pg_get_wal_record_info(PG_FUNCTION_ARGS) } /* - * Validate the input LSNs and compute end LSN for till_end_of_wal versions. + * Get start and end LSN. Adjust end LSN if needed. */ -static XLogRecPtr -ValidateInputLSNs(bool till_end_of_wal, XLogRecPtr start_lsn, - XLogRecPtr end_lsn) +static void +GetInputLSNs(FunctionCallInfo fcinfo, XLogRecPtr *start_lsn, + XLogRecPtr *end_lsn, bool till_end_of_wal) { XLogRecPtr curr_lsn; - if (IsFutureLSN(start_lsn, &curr_lsn)) - { - /* - * GetFlushRecPtr or GetXLogReplayRecPtr gives "end+1" LSN of the last - * record flushed or replayed respectively. But let's use the LSN up - * to "end" in user facing message. - */ + *start_lsn = PG_GETARG_LSN(0); + curr_lsn = GetCurrentLSN(); + + if (*start_lsn > curr_lsn) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("cannot accept future start LSN"), errdetail("Last known WAL LSN on the database system is at %X/%X.", LSN_FORMAT_ARGS(curr_lsn)))); - } if (till_end_of_wal) - end_lsn = curr_lsn; + *end_lsn = curr_lsn; + else + { + *end_lsn = PG_GETARG_LSN(1); - if (end_lsn > curr_lsn) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("cannot accept future end LSN"), - errdetail("Last known WAL LSN on the database system is at %X/%X.", - LSN_FORMAT_ARGS(curr_lsn)))); + /* + * Adjust end LSN to what the system knows at this point instead of + * raising errors. This helps to achieve till end of WAL functionality. + */ + if (*end_lsn > curr_lsn) + *end_lsn = curr_lsn; + } - if (start_lsn >= end_lsn) + /* Return false, when start LSN is past the end LSN. */ + if (*start_lsn >= *end_lsn) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("WAL start LSN must be less than end LSN"))); - - return end_lsn; } /* * Get info and data of all WAL records between start LSN and end LSN. */ static void -GetWALRecordsInfo(FunctionCallInfo fcinfo, XLogRecPtr start_lsn, - XLogRecPtr end_lsn) +GetWALRecordsInfo(FunctionCallInfo fcinfo, bool till_end_of_wal) { #define PG_GET_WAL_RECORDS_INFO_COLS 11 + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; XLogReaderState *xlogreader; ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; Datum values[PG_GET_WAL_RECORDS_INFO_COLS] = {0}; @@ -517,6 +509,8 @@ GetWALRecordsInfo(FunctionCallInfo fcinfo, XLogRecPtr start_lsn, MemoryContext old_cxt; MemoryContext tmp_cxt; + GetInputLSNs(fcinfo, &start_lsn, &end_lsn, till_end_of_wal); + InitMaterializedSRF(fcinfo, 0); xlogreader = InitXLogReaderState(start_lsn); @@ -554,42 +548,18 @@ GetWALRecordsInfo(FunctionCallInfo fcinfo, XLogRecPtr start_lsn, /* * Get info and data of all WAL records between start LSN and end LSN. * - * This function emits an error if a future start or end WAL LSN i.e. WAL LSN - * the database system doesn't know about is specified. - */ -Datum -pg_get_wal_records_info(PG_FUNCTION_ARGS) -{ - XLogRecPtr start_lsn; - XLogRecPtr end_lsn; - - start_lsn = PG_GETARG_LSN(0); - end_lsn = PG_GETARG_LSN(1); - - end_lsn = ValidateInputLSNs(false, start_lsn, end_lsn); - - GetWALRecordsInfo(fcinfo, start_lsn, end_lsn); - - PG_RETURN_VOID(); -} - -/* - * Get info and data of all WAL records from start LSN till end of WAL. + * This function emits an error if the start LSN is in future i.e. LSN the + * database system doesn't know about. + * + * This function adjusts future end LSN to the last flushed LSN when not in + * recovery or the last replayed LSN when in recovery. * - * This function emits an error if a future start i.e. WAL LSN the database - * system doesn't know about is specified. + * This function emits an error, if start LSN is past the end LSN. */ Datum -pg_get_wal_records_info_till_end_of_wal(PG_FUNCTION_ARGS) +pg_get_wal_records_info(PG_FUNCTION_ARGS) { - XLogRecPtr start_lsn; - XLogRecPtr end_lsn = InvalidXLogRecPtr; - - start_lsn = PG_GETARG_LSN(0); - - end_lsn = ValidateInputLSNs(true, start_lsn, end_lsn); - - GetWALRecordsInfo(fcinfo, start_lsn, end_lsn); + GetWALRecordsInfo(fcinfo, false); PG_RETURN_VOID(); } @@ -757,16 +727,22 @@ GetXLogSummaryStats(XLogStats *stats, ReturnSetInfo *rsinfo, * Get WAL stats between start LSN and end LSN. */ static void -GetWalStats(FunctionCallInfo fcinfo, XLogRecPtr start_lsn, - XLogRecPtr end_lsn, bool stats_per_record) +GetWalStats(FunctionCallInfo fcinfo, bool till_end_of_wal) { #define PG_GET_WAL_STATS_COLS 9 + XLogRecPtr start_lsn; + XLogRecPtr end_lsn; + bool stats_per_record; XLogReaderState *xlogreader; XLogStats stats = {0}; ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; Datum values[PG_GET_WAL_STATS_COLS] = {0}; bool nulls[PG_GET_WAL_STATS_COLS] = {0}; + GetInputLSNs(fcinfo, &start_lsn, &end_lsn, till_end_of_wal); + + stats_per_record = PG_GETARG_BOOL(2); + InitMaterializedSRF(fcinfo, 0); xlogreader = InitXLogReaderState(start_lsn); @@ -792,46 +768,49 @@ GetWalStats(FunctionCallInfo fcinfo, XLogRecPtr start_lsn, /* * Get stats of all WAL records between start LSN and end LSN. * - * This function emits an error if a future start or end WAL LSN i.e. WAL LSN - * the database system doesn't know about is specified. + * This function emits an error if the start LSN is in future i.e. LSN the + * database system doesn't know about. + * + * This function adjusts future end LSN to the last flushed LSN when not in + * recovery or the last replayed LSN when in recovery. + * + * This function emits an error, if start LSN is past the end LSN. */ Datum pg_get_wal_stats(PG_FUNCTION_ARGS) { - XLogRecPtr start_lsn; - XLogRecPtr end_lsn; - bool stats_per_record; - - start_lsn = PG_GETARG_LSN(0); - end_lsn = PG_GETARG_LSN(1); - stats_per_record = PG_GETARG_BOOL(2); - - end_lsn = ValidateInputLSNs(false, start_lsn, end_lsn); - - GetWalStats(fcinfo, start_lsn, end_lsn, stats_per_record); + GetWalStats(fcinfo, false); PG_RETURN_VOID(); } /* - * Get stats of all WAL records from start LSN till end of WAL. + * NB: Following till_end_of_wal functions have been removed in newer versions + * of extension. However, we keep them around for backward compatibility. Which + * means, these functions work if someone explicitly installs the older + * extension version (using CREATE EXTENSION pg_walinspect WITH VERSION '1.0';) + * containing them. * - * This function emits an error if a future start i.e. WAL LSN the database - * system doesn't know about is specified. + * If definitions of these functions are removed completely, the extension + * fails to install. XXX: We might consider removing them eventually after + * enough extension versions are out and we do away with version '1.0' + * completely. + * + * In newer versions, one can use pg_get_wal_records_info()/pg_get_wal_stats() + * for the same till_end_of_wal functionality. */ Datum -pg_get_wal_stats_till_end_of_wal(PG_FUNCTION_ARGS) +pg_get_wal_records_info_till_end_of_wal(PG_FUNCTION_ARGS) { - XLogRecPtr start_lsn; - XLogRecPtr end_lsn = InvalidXLogRecPtr; - bool stats_per_record; + GetWALRecordsInfo(fcinfo, true); - start_lsn = PG_GETARG_LSN(0); - stats_per_record = PG_GETARG_BOOL(1); - - end_lsn = ValidateInputLSNs(true, start_lsn, end_lsn); + PG_RETURN_VOID(); +} - GetWalStats(fcinfo, start_lsn, end_lsn, stats_per_record); +Datum +pg_get_wal_stats_till_end_of_wal(PG_FUNCTION_ARGS) +{ + GetWalStats(fcinfo, true); PG_RETURN_VOID(); } diff --git a/contrib/pg_walinspect/sql/oldextversions.sql b/contrib/pg_walinspect/sql/oldextversions.sql new file mode 100644 index 0000000000..79a382acb4 --- /dev/null +++ b/contrib/pg_walinspect/sql/oldextversions.sql @@ -0,0 +1,29 @@ +-- test old extension version entry points + +DROP EXTENSION pg_walinspect; +CREATE EXTENSION pg_walinspect WITH VERSION '1.0'; + +-- List what version 1.0 contains +\dx+ pg_walinspect + +-- =================================================================== +-- Tests to check if the removed functions work when older version of +-- extension is explicitly installed. +-- =================================================================== +-- Make sure checkpoints don't interfere with the test. +SELECT 'init' FROM pg_create_physical_replication_slot('regress_pg_walinspect_slot', true, false); + +CREATE TABLE sample_tbl(col1 int, col2 int); +SELECT pg_current_wal_lsn() AS wal_lsn1 \gset +INSERT INTO sample_tbl SELECT * FROM generate_series(1, 2); + +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info_till_end_of_wal(:'wal_lsn1'); +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1'); + +-- Move to new version 1.1 +ALTER EXTENSION pg_walinspect UPDATE TO '1.1'; + +-- List what version 1.1 contains +\dx+ pg_walinspect + +DROP EXTENSION pg_walinspect; diff --git a/contrib/pg_walinspect/sql/pg_walinspect.sql b/contrib/pg_walinspect/sql/pg_walinspect.sql index 01a120f398..8f2cdd09f9 100644 --- a/contrib/pg_walinspect/sql/pg_walinspect.sql +++ b/contrib/pg_walinspect/sql/pg_walinspect.sql @@ -1,77 +1,88 @@ CREATE EXTENSION pg_walinspect; --- Make sure checkpoints don't interfere with the test. +-- Make sure checkpoints don't interfere with the test SELECT 'init' FROM pg_create_physical_replication_slot('regress_pg_walinspect_slot', true, false); +-- Generate WAL and captures LSNs as needed CREATE TABLE sample_tbl(col1 int, col2 int); - SELECT pg_current_wal_lsn() AS wal_lsn1 \gset - INSERT INTO sample_tbl SELECT * FROM generate_series(1, 2); - SELECT pg_current_wal_lsn() AS wal_lsn2 \gset - INSERT INTO sample_tbl SELECT * FROM generate_series(3, 4); -- =================================================================== -- Tests for input validation -- =================================================================== +-- Invalid input LSN or LSN that server doesn't have WAL for +SELECT * FROM pg_get_wal_record_info('0/0'); -- ERROR -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info(:'wal_lsn2', :'wal_lsn1'); -- ERROR +-- Invalid start LSN or start LSN that server doesn't have WAL for +SELECT * FROM pg_get_wal_records_info('0/0', :'wal_lsn1'); -- ERROR +SELECT * FROM pg_get_wal_stats('0/0', :'wal_lsn1'); -- ERROR +SELECT * FROM pg_get_wal_block_info('0/0', :'wal_lsn1'); -- ERROR -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats(:'wal_lsn2', :'wal_lsn1'); -- ERROR +-- Start LSN >= End LSN +SELECT * FROM pg_get_wal_records_info(:'wal_lsn2', :'wal_lsn1'); -- ERROR +SELECT * FROM pg_get_wal_stats(:'wal_lsn2', :'wal_lsn1'); -- ERROR +SELECT * FROM pg_get_wal_block_info(:'wal_lsn2', :'wal_lsn1'); -- ERROR -- =================================================================== -- Tests for all function executions -- =================================================================== +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_record_info(:'wal_lsn1'); +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2'); -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_record_info(:'wal_lsn1'); +-- Get info till end of WAL. End LSN is way higher, adjust it to the latest LSN +-- that server knows. +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', 'FFFFFFFF/FFFFFFFF'); -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2'); +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2'); -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_records_info_till_end_of_wal(:'wal_lsn1'); - -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats(:'wal_lsn1', :'wal_lsn2'); - -SELECT COUNT(*) >= 0 AS ok FROM pg_get_wal_stats_till_end_of_wal(:'wal_lsn1'); +-- Get info till end of WAL. End LSN is way higher, adjust it to the latest LSN +-- that server knows. +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_stats(:'wal_lsn1', 'FFFFFFFF/FFFFFFFF'); -- =================================================================== -- Test for filtering out WAL records of a particular table -- =================================================================== - SELECT oid AS sample_tbl_oid FROM pg_class WHERE relname = 'sample_tbl' \gset - SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2') WHERE block_ref LIKE concat('%', :'sample_tbl_oid', '%') AND resource_manager = 'Heap'; -- =================================================================== -- Test for filtering out WAL records based on resource_manager and --- record_type +-- record_type. -- =================================================================== - SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_records_info(:'wal_lsn1', :'wal_lsn2') WHERE resource_manager = 'Heap' AND record_type = 'INSERT'; -- =================================================================== -- Tests to get block information from WAL record -- =================================================================== - -- Update table to generate some block data SELECT pg_current_wal_lsn() AS wal_lsn3 \gset UPDATE sample_tbl SET col1 = col1 + 1 WHERE col1 = 1; SELECT pg_current_wal_lsn() AS wal_lsn4 \gset --- Check if we get block data from WAL record. +-- Check if we get block data from WAL record SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn3', :'wal_lsn4') WHERE relfilenode = :'sample_tbl_oid' AND blockdata IS NOT NULL; +-- Check till end of WAL if we get block data from WAL record. End LSN is way +-- higher, adjust it to the latest LSN that server knows. +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn3', 'FFFFFFFF/FFFFFFFF') + WHERE relfilenode = :'sample_tbl_oid'; --- Force full-page image on the next update. +-- Force full-page image on the next update SELECT pg_current_wal_lsn() AS wal_lsn5 \gset CHECKPOINT; UPDATE sample_tbl SET col1 = col1 + 1 WHERE col1 = 2; SELECT pg_current_wal_lsn() AS wal_lsn6 \gset --- Check if we get FPI from WAL record. +-- Check if we get FPI from WAL record SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn5', :'wal_lsn6') WHERE relfilenode = :'sample_tbl_oid' AND fpi IS NOT NULL; +-- Check till end of WAL if we get FPI from WAL record. End LSN is way higher, +-- adjust it to the latest LSN that server knoows. +SELECT COUNT(*) >= 1 AS ok FROM pg_get_wal_block_info(:'wal_lsn5', 'FFFFFFFF/FFFFFFFF') + WHERE relfilenode = :'sample_tbl_oid'; -- =================================================================== -- Tests for permissions @@ -80,29 +91,22 @@ CREATE ROLE regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- no - SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_records_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- no - SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- no - SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- no -- Functions accessible by users with role pg_read_server_files - GRANT pg_read_server_files TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes - SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_records_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes - SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes - SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes @@ -111,46 +115,34 @@ REVOKE pg_read_server_files FROM regress_pg_walinspect; -- Superuser can grant execute to other users GRANT EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) TO regress_pg_walinspect; - GRANT EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn) TO regress_pg_walinspect; - GRANT EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) TO regress_pg_walinspect; - GRANT EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn) TO regress_pg_walinspect; SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_record_info(pg_lsn)', 'EXECUTE'); -- yes - SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_records_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes - SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_stats(pg_lsn, pg_lsn, boolean) ', 'EXECUTE'); -- yes - SELECT has_function_privilege('regress_pg_walinspect', 'pg_get_wal_block_info(pg_lsn, pg_lsn) ', 'EXECUTE'); -- yes REVOKE EXECUTE ON FUNCTION pg_get_wal_record_info(pg_lsn) FROM regress_pg_walinspect; - REVOKE EXECUTE ON FUNCTION pg_get_wal_records_info(pg_lsn, pg_lsn) FROM regress_pg_walinspect; - REVOKE EXECUTE ON FUNCTION pg_get_wal_stats(pg_lsn, pg_lsn, boolean) FROM regress_pg_walinspect; - REVOKE EXECUTE ON FUNCTION pg_get_wal_block_info(pg_lsn, pg_lsn) FROM regress_pg_walinspect; -- =================================================================== -- Clean up -- =================================================================== - DROP ROLE regress_pg_walinspect; - SELECT pg_drop_replication_slot('regress_pg_walinspect_slot'); - DROP TABLE sample_tbl; diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 3b19863dce..d9aac121d5 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@ -61,8 +61,8 @@ Gets WAL record information of a given LSN. If the given LSN isn't at the start of a WAL record, it gives the information of the next available valid WAL record; or an error if no such record is found. - For example, usage of the function is as - follows: + It will raise an error, if given LSN is in future (i.e. the LSN server + doesn't know about). For example, usage of the function is as follows: <screen> postgres=# SELECT * FROM pg_get_wal_record_info('0/1E826E98'); -[ RECORD 1 ]----+---------------------------------------------------- @@ -85,7 +85,7 @@ block_ref | blkref #0: rel 1663/5/60221 fork main blk 2 <varlistentry id="pgwalinspect-funcs-pg-get-wal-records-info"> <term> <function> - pg_get_wal_records_info(start_lsn pg_lsn, end_lsn pg_lsn) + pg_get_wal_records_info(start_lsn pg_lsn, end_lsn pg_lsn DEFAULT NULL) returns setof record </function> </term> @@ -94,9 +94,13 @@ block_ref | blkref #0: rel 1663/5/60221 fork main blk 2 <para> Gets information of all the valid WAL records between <replaceable>start_lsn</replaceable> and <replaceable>end_lsn</replaceable>. - Returns one row per WAL record. If <replaceable>start_lsn</replaceable> - or <replaceable>end_lsn</replaceable> are not yet available, the - function will raise an error. For example: + Returns one row per WAL record. If a future <replaceable>end_lsn</replaceable> + (i.e. the LSN server doesn't know about) is specified, it returns + informaton till end of WAL. It will raise an error, if the server + doesn't have WAL available at given <replaceable>start_lsn</replaceable> + or if the <replaceable>start_lsn</replaceable> is in future or is past + the <replaceable>end_lsn</replaceable>. For example, usage of the + function is as follows: <screen> postgres=# SELECT * FROM pg_get_wal_records_info('0/1E913618', '0/1E913740') LIMIT 1; -[ RECORD 1 ]----+-------------------------------------------------------------- @@ -116,27 +120,10 @@ block_ref | </listitem> </varlistentry> - <varlistentry id="pgwalinspect-funcs-pg-get-wal-records-info-till-end-of-wal"> - <term> - <function> - pg_get_wal_records_info_till_end_of_wal(start_lsn pg_lsn) - returns setof record - </function> - </term> - - <listitem> - <para> - This function is the same as <function>pg_get_wal_records_info()</function>, - except that it gets information of all the valid WAL records from - <replaceable>start_lsn</replaceable> till the end of WAL. - </para> - </listitem> - </varlistentry> - <varlistentry id="pgwalinspect-funcs-pg-get-wal-stats"> <term> <function> - pg_get_wal_stats(start_lsn pg_lsn, end_lsn pg_lsn, per_record boolean DEFAULT false) + pg_get_wal_stats(start_lsn pg_lsn, end_lsn pg_lsn DEFAULT NULL, per_record boolean DEFAULT false) returns setof record </function> </term> @@ -149,9 +136,13 @@ block_ref | <replaceable>resource_manager</replaceable> type. When <replaceable>per_record</replaceable> is set to <literal>true</literal>, it returns one row per <replaceable>record_type</replaceable>. - If <replaceable>start_lsn</replaceable> - or <replaceable>end_lsn</replaceable> are not yet available, the - function will raise an error. For example: + If a future <replaceable>end_lsn</replaceable> (i.e. the LSN server + doesn't know about) is specified, it returns stats till end of WAL. It + will raise an error, if the server doesn't have WAL available at given + <replaceable>start_lsn</replaceable> or if the + <replaceable>start_lsn</replaceable> is in future or is past the + <replaceable>end_lsn</replaceable>. For example, usage of the function is + as follows: <screen> postgres=# SELECT * FROM pg_get_wal_stats('0/1E847D00', '0/1E84F500') WHERE count > 0 LIMIT 1 AND @@ -171,23 +162,6 @@ combined_size_percentage | 2.8634072910530795 </listitem> </varlistentry> - <varlistentry id="pgwalinspect-funcs-pg-get-wal-stats-till-end-of-wal"> - <term> - <function> - pg_get_wal_stats_till_end_of_wal(start_lsn pg_lsn, per_record boolean DEFAULT false) - returns setof record - </function> - </term> - - <listitem> - <para> - This function is the same as <function>pg_get_wal_stats()</function>, - except that it gets statistics of all the valid WAL records from - <replaceable>start_lsn</replaceable> till end of WAL. - </para> - </listitem> - </varlistentry> - <varlistentry> <term> <function>pg_get_wal_block_info(start_lsn pg_lsn, end_lsn pg_lsn) returns setof record</function> @@ -202,9 +176,13 @@ combined_size_percentage | 2.8634072910530795 and their information associated with all the valid WAL records between <replaceable>start_lsn</replaceable> and <replaceable>end_lsn</replaceable>. Returns one row per block registered - in a WAL record. If <replaceable>start_lsn</replaceable> or - <replaceable>end_lsn</replaceable> are not yet available, the function - will raise an error. For example: + in a WAL record. If a future <replaceable>end_lsn</replaceable> + (i.e. the LSN server doesn't know about) is specified, it returns + information till end of WAL. It will raise an error, if the server + doesn't have WAL available at given <replaceable>start_lsn</replaceable> + or if the <replaceable>start_lsn</replaceable> is in future or is past + the <replaceable>end_lsn</replaceable>. For example, usage of the + function is as follows: <screen> postgres=# SELECT lsn, blockid, reltablespace, reldatabase, relfilenode, relblocknumber, forkname, @@ -227,8 +205,22 @@ fpiinfo | {HAS_HOLE,APPLY} </para> </listitem> </varlistentry> - </variablelist> + + <note> + <para> + Note that <function>pg_get_wal_records_info_till_end_of_wal</function> and + <function>pg_get_wal_stats_till_end_of_wal</function> functions have been + removed in the <filename>pg_walinspect</filename> version + <literal>1.1</literal>. The same functionality can be achieved with + <function>pg_get_wal_records_info</function> and + <function>pg_get_wal_stats</function> functions by specifying a future + <replaceable>end_lsn</replaceable>. However, <function>till_end_of_wal</function> + functions will still work if the extension is installed explicitly with + version <literal>1.0</literal>. + </para> + </note> + </sect2> <sect2 id="pgwalinspect-author"> -- 2.34.1