Re: pg_waldump

2023-12-19 Thread Fabrice Chapuis
Ok thanks for all these precisions Regards Fabrice On Tue, Dec 19, 2023 at 2:00 PM Matthias van de Meent < boekewurm+postg...@gmail.com> wrote: > On Tue, 19 Dec 2023, 12:27 Fabrice Chapuis, > wrote: > > > > Hi, > > Is it possible to visualize the DDL with the

Re: pg_waldump

2023-12-19 Thread Matthias van de Meent
On Tue, 19 Dec 2023, 12:27 Fabrice Chapuis, wrote: > > Hi, > Is it possible to visualize the DDL with the pg_waldump tool. I created a > postgres user but I cannot find the creation command in the wals Not really, no. PostgreSQL does not log DDL or DML as such in WAL. Essentially

pg_waldump

2023-12-19 Thread Fabrice Chapuis
Hi, Is it possible to visualize the DDL with the pg_waldump tool. I created a postgres user but I cannot find the creation command in the wals Thanks for help Fabrice

Re: pg_waldump vs. all-zeros WAL files; server creation of such files

2023-08-13 Thread Michael Paquier
On Sat, Aug 12, 2023 at 08:15:31PM -0700, Noah Misch wrote: > The attached 010_zero.pl, when run as part of the pg_waldump test suite, fails > at today's master (c36b636) and v15 (1bc19df). It passes at v14 (5a32af3). > Command "pg_waldump --start 0/0100 --end 0/0100010

pg_waldump vs. all-zeros WAL files; server creation of such files

2023-08-12 Thread Noah Misch
The attached 010_zero.pl, when run as part of the pg_waldump test suite, fails at today's master (c36b636) and v15 (1bc19df). It passes at v14 (5a32af3). Command "pg_waldump --start 0/0100 --end 0/01000100" fails as follows: pg_waldump: error: WAL segment size must be

Re: pg_waldump: add test for coverage

2023-07-05 Thread Peter Eisentraut
ow records with transaction ID XID --follow is a bit tricky to test because you need to leave pg_waldump running in the background for a while, or something like that. --timeline and --xid can be tested but would need some work on the underlying test data (such as creating more than one time

Re: pg_waldump: add test for coverage

2023-06-29 Thread Tristen Raab
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested Hello, I've reviewed your latest v3 patches on Ubuntu 23.04. Both pa

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-28 Thread Michael Paquier
On Wed, Jun 28, 2023 at 09:20:27AM +0900, Kyotaro Horiguchi wrote: > At Tue, 27 Jun 2023 18:58:39 -0500, David Christensen > wrote in >>> Adjusted as per the v2 attached. >> >> +1 > > +1 Okay, cool. Both of you seem happy with it, so I have applied it. Thanks for the quick checks. -- Michael

Re: pg_waldump: add test for coverage

2023-06-27 Thread Peter Eisentraut
dback from many people. I think having some more test coverage for pg_waldump would be good, so I encourage you to continue working on this. I made an updated patch that incorporates many of your ideas and code, just made it a bit more compact, and added more tests for various command-line op

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread Kyotaro Horiguchi
At Tue, 27 Jun 2023 18:58:39 -0500, David Christensen wrote in > > Adjusted as per the v2 attached. > > +1 +1 regards. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread David Christensen
> Adjusted as per the v2 attached. +1

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread Michael Paquier
erm in the comment, and adjusted the XXX to be about the LSN numbers. Adjusted as per the v2 attached. -- Michael From f0936a95651f767e0f56ff507db5d5a02c7629b2 Mon Sep 17 00:00:00 2001 From: Michael Paquier Date: Wed, 28 Jun 2023 08:46:26 +0900 Subject: [PATCH v2] Add timeline to file names g

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread Michael Paquier
On Tue, Jun 27, 2023 at 11:53:10AM -0500, David Christensen wrote: > Patch looks good, but agreed that that comment should also be fixed. Okay, thanks for checking! -- Michael signature.asc Description: PGP signature

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread David Christensen
On Tue, Jun 27, 2023 at 1:12 AM Michael Paquier wrote: > > Hi all, > (Fujii-san and David in CC.) > > Fujii-san has reported on Twitter that we had better add the TLI > number to what pg_waldump --save-fullpage generates for the file names > of the blocks, as it coul

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread Kyotaro Horiguchi
Of course, it's wrong. At Tue, 27 Jun 2023 16:39:52 +0900 (JST), Kyotaro Horiguchi wrote in > I meant that the name is structured as - TLIh-TLIl._, which + LSNh-LSNl._, which > appears to be inconsistent with the comment. (And I'm not sure what > "TLOID" is..) > > regards. > > -- > K

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread Kyotaro Horiguchi
At Tue, 27 Jun 2023 15:58:38 +0900, Michael Paquier wrote in > On Tue, Jun 27, 2023 at 03:44:04PM +0900, Kyotaro Horiguchi wrote: > > The file name was "LSNh-LSNl.spcOid.dbOid.relNumber.blk_forkname", but > > the comment in the TAP script read as: > > > > -# -.DBOID.TLOID.NODEOI

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-26 Thread Michael Paquier
On Tue, Jun 27, 2023 at 03:44:04PM +0900, Kyotaro Horiguchi wrote: > +# - Timeline number in hex format. > > Arn't we reffering to it as "Timeline ID"? (I remember there was a > discussion about redefining the "timeline ID" to use non-orderable > IDs. That is, making it non-numbers.) Using ID is

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-26 Thread Kyotaro Horiguchi
At Tue, 27 Jun 2023 15:12:43 +0900, Michael Paquier wrote in > Hi all, > (Fujii-san and David in CC.) > > Fujii-san has reported on Twitter that we had better add the TLI > number to what pg_waldump --save-fullpage generates for the file names > of the blocks, as it could b

Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-26 Thread Michael Paquier
Hi all, (Fujii-san and David in CC.) Fujii-san has reported on Twitter that we had better add the TLI number to what pg_waldump --save-fullpage generates for the file names of the blocks, as it could be possible that we overwrite some blocks. This information can be added thanks to ws_tli, that

Re: pg_waldump: add test for coverage

2023-06-14 Thread Peter Eisentraut
more test coverage for pg_waldump would be good, so I encourage you to continue working on this. I made an updated patch that incorporates many of your ideas and code, just made it a bit more compact, and added more tests for various command-line options. This moves the test coverage of pg_wa

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-27 Thread Bharath Rupireddy
On Mon, Dec 26, 2022 at 4:18 PM Bharath Rupireddy wrote: > > On Mon, Dec 26, 2022 at 12:59 PM Michael Paquier wrote: > > +1. I think this feature will also be useful in pg_walinspect. Just for the record - here's the pg_walinspect function to extract FPIs from WAL records - https://www.postgresq

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread Michael Paquier
On Mon, Dec 26, 2022 at 04:18:18PM +0530, Bharath Rupireddy wrote: > +1. I think this feature will also be useful in pg_walinspect. > However, I'm a bit concerned that it can flood the running database > disk - if someone generates a lot of FPI files. pg_read_file() and pg_wald

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread Michael Paquier
On Mon, Dec 26, 2022 at 02:39:03PM -0600, Justin Pryzby wrote: > fclose() should be tested, too: Sure. Done that too, and applied the change after a last lookup. -- Michael signature.asc Description: PGP signature

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread Justin Pryzby
On Mon, Dec 26, 2022 at 04:28:52PM +0900, Michael Paquier wrote: > Comments? > + file = fopen(filename, PG_BINARY_W); > + if (!file) > + pg_fatal("could not open file \"%s\": %m", filename); > + > + if (fwrite(page, BLCKSZ, 1, file) != 1) > +

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread David Christensen
> On Dec 26, 2022, at 1:29 AM, Michael Paquier wrote: > > On Sat, Dec 24, 2022 at 06:23:29PM +0530, Bharath Rupireddy wrote: >> Thanks for the patch. I've made the above change as well as renamed >> the test file name to be save_fpi.pl, everything else remains the same >> as v11. Here's the v12

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread Bharath Rupireddy
On Mon, Dec 26, 2022 at 12:59 PM Michael Paquier wrote: > > I have done a review of that, and here are my notes: > - The variable names were a bit inconsistent, so I have switched most > of the new code to use "fullpage". > > - The new test has been renamed. > > - RestoreBlockImage() would report

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-25 Thread Michael Paquier
Mon Sep 17 00:00:00 2001 From: Michael Paquier Date: Mon, 26 Dec 2022 16:28:00 +0900 Subject: [PATCH v13] Teach pg_waldump to extract FPIs from the WAL stream Extracts full-page images from the WAL stream into a given target directory. These images are subject to the same filtering rules as nor

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-24 Thread Bharath Rupireddy
1 with yours and Michael's latest feedback. Thanks for the patch. I've made the above change as well as renamed the test file name to be save_fpi.pl, everything else remains the same as v11. Here's the v12 patch which LGTM. I'll mark it as RfC - https://commitfest.postgresql.org/41

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-23 Thread David Christensen
ed for the test to work. Enclosing v11 with yours and Michael's latest feedback. Best, David v11-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-st.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-23 Thread David Christensen
On Mon, Dec 19, 2022 at 12:23 AM Michael Paquier wrote: > > On Thu, Dec 15, 2022 at 05:17:46PM -0600, David Christensen wrote: > > On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier > > wrote: > > This v10 should incorporate your feedback as well as Bharath's. > > Thanks for the new version. I ha

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-23 Thread David Christensen
On Wed, Dec 21, 2022 at 5:47 AM Bharath Rupireddy wrote: > > On Fri, Dec 16, 2022 at 4:47 AM David Christensen > wrote: > > > > On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier > > wrote: > > > > > > On Wed, Dec 14, 2022 at 04:44:34PM -0600, David Christensen wrote: > > > > I can get one sent i

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-21 Thread Bharath Rupireddy
On Fri, Dec 16, 2022 at 4:47 AM David Christensen wrote: > > On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier wrote: > > > > On Wed, Dec 14, 2022 at 04:44:34PM -0600, David Christensen wrote: > > > I can get one sent in tomorrow. > > This v10 should incorporate your feedback as well as Bharath's.

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-18 Thread Michael Paquier
On Thu, Dec 15, 2022 at 05:17:46PM -0600, David Christensen wrote: > On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier wrote: > This v10 should incorporate your feedback as well as Bharath's. Thanks for the new version. I have minor comments. >> It seems to me that you could allow things to work

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-15 Thread David Christensen
7;, < +SELECT 'init' FROM > pg_create_physical_replication_slot('regress_pg_waldump_slot', true, false); > +CREATE TABLE test_table AS SELECT generate_series(1,100) a; > +CHECKPOINT; -- required to force FPI for next writes > +UPDATE test_table SET a = a + 1; > Using

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-14 Thread Michael Paquier
On Wed, Dec 14, 2022 at 04:44:34PM -0600, David Christensen wrote: > I can get one sent in tomorrow. -XLogRecordHasFPW(XLogReaderState *record) +XLogRecordHasFPI(XLogReaderState *record) This still refers to a FPW, so let's leave that out as well as any renamings of this kind.. + if (config.sav

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-14 Thread David Christensen
Hi Bharath, I can get one sent in tomorrow. Thanks, David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-14 Thread Bharath Rupireddy
100?) qq[] or quotes is being used. I > > have no strong opinion here, I'll leave it to the other reviewers or > > committer. > > I'm inclined to leave it just for (personal) readability, but can > change if there's a strong consensus against. > > > > > 11. &

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-17 Thread Peter Geoghegan
On Thu, Nov 17, 2022 at 9:02 AM Peter Geoghegan wrote: > Plan is to commit this later on today, barring objections. Pushed, thanks. -- Peter Geoghegan

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-17 Thread Peter Geoghegan
On Wed, Nov 16, 2022 at 4:34 PM Peter Geoghegan wrote: > WFM. Attached is v3. Plan is to commit this later on today, barring objections. Thanks -- Peter Geoghegan v3-0001-Standardize-rmgrdesc-recovery-conflict-XID-output.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-17 Thread David Christensen
+# verify filename formats matches w/--save-fpi > > > +for my $fullpath (glob "$tmp_folder/raw/*") > > > Do we need to look for the exact match of the file that gets created > > > in the save-fpi path? While checking for this is great, it makes the > > >

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-16 Thread Peter Geoghegan
On Wed, Nov 16, 2022 at 4:25 PM Andres Freund wrote: > > Anyway, worth calling this out directly in these comments IMV. We're > > addressing two closely related things that assign opposite meanings to > > InvalidTransactionId, which is rather confusing. > > It makes sense to call this out, but I'd

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-16 Thread Andres Freund
Hi, On 2022-11-16 15:37:40 -0800, Peter Geoghegan wrote: > On Wed, Nov 16, 2022 at 3:27 PM Andres Freund wrote: > > What are "snapshotConflictHorizon format XIDs"? I guess you mean format in > > the > > sense of having the semantics of snapshotConflictHorizon? > > Yes. That is the only possible

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-16 Thread Peter Geoghegan
On Wed, Nov 16, 2022 at 3:27 PM Andres Freund wrote: > The "(also...) formulation seems a bit odd. How about "an obsolescent heap > tuple that the caller is physically removing, e.g. via HOT pruning or index > deletion." or such? Okay, WFM. > > + * snapshotConflictHorizon format values are how a

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-16 Thread Andres Freund
Hi, On 2022-11-16 14:14:30 -0800, Peter Geoghegan wrote: > /* > - * If 'tuple' contains any visible XID greater than latestRemovedXid, > - * ratchet forwards latestRemovedXid to the greatest one found. > - * This is used as the basis for generating Hot Standby conflicts, so > - * if a tuple was n

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-16 Thread Peter Geoghegan
On Tue, Nov 15, 2022 at 8:48 PM Peter Geoghegan wrote: > Okay, let's go with snapshotConflictHorizon. I'll use that name in the > next revision, which I should be able to post tomorrow. Attached is a somewhat cleaned up version that uses that symbol name for everything. -- Peter Geoghegan v2-0

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-16 Thread Bharath Rupireddy
quot;$tmp_folder/raw/*") > > Do we need to look for the exact match of the file that gets created > > in the save-fpi path? While checking for this is great, it makes the > > test code non-portable (may not work on Windows or other platforms, > > no?) and complex? This way, y

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Andres Freund
On 2022-11-15 20:48:56 -0800, Peter Geoghegan wrote: > On Tue, Nov 15, 2022 at 5:29 PM Andres Freund wrote: > > If we want to focus on the mvcc affects we could just go for something like > > snapshotConflictHorizon or such. > > Okay, let's go with snapshotConflictHorizon. I'll use that name in t

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Peter Geoghegan
On Tue, Nov 15, 2022 at 5:29 PM Andres Freund wrote: > If we want to focus on the mvcc affects we could just go for something like > snapshotConflictHorizon or such. Okay, let's go with snapshotConflictHorizon. I'll use that name in the next revision, which I should be able to post tomorrow. --

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 13:54:24 -0800, Peter Geoghegan wrote: > On Tue, Nov 15, 2022 at 12:29 PM Andres Freund wrote: > > ... I strongly dislike latestCommittedXid. That seems at least as misleading > > as latestRemovedXid and has the danger of confusion with latestCompletedXid > > as you mention. >

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Peter Geoghegan
On Tue, Nov 15, 2022 at 12:29 PM Andres Freund wrote: > ... I strongly dislike latestCommittedXid. That seems at least as misleading > as latestRemovedXid and has the danger of confusion with latestCompletedXid > as you mention. > How about latestAffectedXid? I get why you don't care for latestC

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Andres Freund
Hi, I like the idea of this, but: On 2022-11-15 10:24:05 -0800, Peter Geoghegan wrote: > I'm not necessarily that attached to the name latestCommittedXid. It > is more accurate, but it's also a little bit too similar to another > common XID symbol name, latestCompletedXid. Can anyone suggest an >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-15 Thread David Christensen
Enclosed is v9. - code style consistency (FPI instead of FPW) internally. - cleanup of no-longer needed checksum-related pieces from code and tests. - test cleanup/simplification. - other comment cleanup. Passes all CI checks. Best, David v9-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL

Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Peter Geoghegan
always the responsibility of code that runs during original execution (details of which will vary by record type). Consistency makes all of this fairly explicit, and makes it easier to use tools like pg_waldump to debug recovery conflicts -- the user can grep for the same generic symbol name and see

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-15 Thread David Christensen
one. > > Thanks. It mostly is in good shape. However, few more comments: > > 1. > +if it does not exist. The images saved will be subject to the same > +filtering and limiting criteria as display records, but in this > +mode pg_waldump will not output

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-15 Thread Bharath Rupireddy
does not exist. The images saved will be subject to the same +filtering and limiting criteria as display records, but in this +mode pg_waldump will not output any other +information. May I know what's the intention of the statement 'The images saved '? If

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-14 Thread David Christensen
Enclosed is v8, which uses the replication slot method to retain WAL as well as fsync'ing the output directory when everything is done. v8-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-str.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-14 Thread David Christensen
/* we have our extracted FPI, let's save it now */ > > > > > After extracting the page from the WAL record, do we need to perform a > > > > > checksum on it? > > > > > > I think you just need to do the following, this will ensure the > >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-11 Thread David Christensen
On Fri, Nov 11, 2022 at 8:15 AM Justin Pryzby wrote: > > On Wed, Nov 09, 2022 at 02:37:29PM -0600, David Christensen wrote: > > On Wed, Nov 9, 2022 at 2:08 PM David Christensen > > wrote: > > > Justin sez: > > > > I was wondering if there's any reason to do "CREATE DATABASE". The vast > > > > m

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-11 Thread Justin Pryzby
On Wed, Nov 09, 2022 at 02:37:29PM -0600, David Christensen wrote: > On Wed, Nov 9, 2022 at 2:08 PM David Christensen > wrote: > > Justin sez: > > > I was wondering if there's any reason to do "CREATE DATABASE". The vast > > > majority of TAP tests don't. > > > > > > $ git grep -ho 'safe_psql[^

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-11 Thread Bharath Rupireddy
On Thu, Nov 10, 2022 at 9:52 PM David Christensen wrote: > > > > > 2. I'm unable to understand the use-case for --fixup-fpi option. > > > > pg_waldump is supposed to be just WAL reader, and must not return any > > > > modified information, with

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-10 Thread David Christensen
able to understand the use-case for --fixup-fpi option. > > > pg_waldump is supposed to be just WAL reader, and must not return any > > > modified information, with --fixup-fpi option, the patch violates this > > > principle i.e. it sets page LSN and returns. Without

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-10 Thread Bharath Rupireddy
On Thu, Nov 10, 2022 at 1:31 AM David Christensen wrote: > Thanks for providing the v7 patch, please see my comments and responses below. > > 2. I'm unable to understand the use-case for --fixup-fpi option. > > pg_waldump is supposed to be just WAL reader, and must not ret

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread David Christensen
blew up when I did that; rather than hunt it down, I just left it in. :-) Enclosed is v7, with changes thus suggested thus far. v7-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-str.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread David Christensen
> > 6. > > +if (dir_status == 0 && mkdir(config.save_fpw_path, 0700) < 0) > > Use pg_dir_create_mode instead of hard-coded 0007? > > I think I thought of that when I first looked at the patch ... but, I'm > not sure, since it says: > > src/include/common/file_perm.h-/* Modes for creating d

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread David Christensen
e fine in the same patch, yes? > 2. I'm unable to understand the use-case for --fixup-fpi option. > pg_waldump is supposed to be just WAL reader, and must not return any > modified information, with --fixup-fpi option, the patch violates this > principle i.e. it sets page LSN and retur

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread Alvaro Herrera
On 2022-Nov-09, Justin Pryzby wrote: > On Wed, Nov 09, 2022 at 06:00:40PM +0530, Bharath Rupireddy wrote: > > 1. For ease of review, please split the test patch to 0002. > > This is just my opinion, but .. why ? Since it's easy to > filter/skip/display a file, I don't think it's usually useful

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread Justin Pryzby
On Wed, Nov 09, 2022 at 06:00:40PM +0530, Bharath Rupireddy wrote: > On Wed, Nov 9, 2022 at 5:08 AM David Christensen > wrote: > > > > Enclosed is v6, which squashes your refactor and adds the additional > > recent suggestions. > > Thanks for working on this feature. Here are some comments for n

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread Bharath Rupireddy
code and tests after these and all other comments are addressed. 1. For ease of review, please split the test patch to 0002. 2. I'm unable to understand the use-case for --fixup-fpi option. pg_waldump is supposed to be just WAL reader, and must not return any modified information, with --fi

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-08 Thread sho kato
The following review has been posted through the commitfest application: make installcheck-world: tested, failed Implements feature: tested, failed Spec compliant: tested, failed Documentation:tested, failed Hello, I tested this patch on Linux and there is no problem.

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-08 Thread David Christensen
Enclosed is v6, which squashes your refactor and adds the additional recent suggestions. Thanks! v6-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-str.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-08 Thread David Christensen
On Tue, Nov 8, 2022 at 4:45 PM Justin Pryzby wrote: > > On Mon, Nov 07, 2022 at 05:01:01PM -0600, David Christensen wrote: > > Hi Justin et al, > > > > Enclosed is v5 of this patch which now passes the CirrusCI checks for > > all supported OSes. I went ahead and reworked the test a bit so it's a >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-08 Thread Justin Pryzby
so, should the test case call pg_relation_filenode() rather than using relfilenode directly ? Is it a problem that the test code assumes pagesize=8192 ? -- Justin >From d4cb23bc2f0edcc65b0bef75fd2e8d6e5aeda21f Mon Sep 17 00:00:00 2001 From: David Christensen Date: Wed, 20 Apr 2022 19:59:35 -

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-07 Thread David Christensen
Hi Justin et al, Enclosed is v5 of this patch which now passes the CirrusCI checks for all supported OSes. I went ahead and reworked the test a bit so it's a little more amenable to the OS-agnostic approach for testing. Best, David v5-0001-Teach-pg_waldump-to-extract-FPIs-from-th

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-04 Thread David Christensen
On Fri, Nov 4, 2022 at 1:38 PM Justin Pryzby wrote: > > > As I recall, that's due to relying on "cp". And "rsync", which > > > shouldn't be assumed to exist by regression tests). Will poke around other TAP tests to see if there's a more consistent interface, what perl version we can assume and a

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-04 Thread Justin Pryzby
On Fri, Nov 04, 2022 at 09:16:29AM -0500, David Christensen wrote: > On Nov 4, 2022, at 9:02 AM, Justin Pryzby wrote: > > On Fri, Nov 04, 2022 at 11:52:59AM +0900, Ian Lawrence Barwick wrote: > >> 2022年5月3日(火) 8:45 David Christensen : > >>> > >>> ...and pushing a couple fixups pointed out by cfbo

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-04 Thread David Christensen
On Nov 4, 2022, at 9:02 AM, Justin Pryzby wrote: > > On Fri, Nov 04, 2022 at 11:52:59AM +0900, Ian Lawrence Barwick wrote: >> 2022年5月3日(火) 8:45 David Christensen : >>> >>> ...and pushing a couple fixups pointed out by cfbot, so here's v4. >> >> cfbot reports the patch no longer applies [1]. As

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-04 Thread Justin Pryzby
On Fri, Nov 04, 2022 at 11:52:59AM +0900, Ian Lawrence Barwick wrote: > 2022年5月3日(火) 8:45 David Christensen : > > > > ...and pushing a couple fixups pointed out by cfbot, so here's v4. > > cfbot reports the patch no longer applies [1]. As CommitFest 2022-11 is > currently underway, this would be a

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-03 Thread Ian Lawrence Barwick
2022年5月3日(火) 8:45 David Christensen : > > ...and pushing a couple fixups pointed out by cfbot, so here's v4. Hi cfbot reports the patch no longer applies [1]. As CommitFest 2022-11 is currently underway, this would be an excellent time to update the patch. [1] http://cfbot.cputube.org/patch_40_3

Re: pg_waldump: add test for coverage

2022-09-22 Thread Andres Freund
Hi, On 2022-08-23 10:50:08 +0900, Dong Wook Lee wrote: > I wrote a test for coverage. Unfortunately the test doesn't seem to pass on windows, and hasn't ever done so: https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/39/3834 Due to the merge of the meson patchset, you should al

Re: pg_waldump: add test for coverage

2022-09-05 Thread Peter Eisentraut
s successfully and check its standard out. For example, instead of # test pg_waldump with -F (main) IPC::Run::run [ 'pg_waldump', "$wal_dump_path", '-F', 'main' ], '>', \$stdout, '2>', \$stderr; isnt($stdout, '

pg_waldump: add test for coverage

2022-08-22 Thread Dong Wook Lee
Hi Hackers, I wrote a test for coverage. Unfortunately, it seems to take quite a while to run the test. I want to improve these execution times, but I don't know exactly what to do. Therefore, I want to hear feedback from many people. --- Regards, Dong Wook Lee v1_add_test_pg_waldump.patch Descri

Re: pg_waldump got an error with waldump file generated by initdb

2022-07-10 Thread Justin Pryzby
On Sun, Jul 10, 2022 at 02:39:00PM -0700, Andres Freund wrote: > On 2022-07-10 21:51:04 +0900, Dong Wook Lee wrote: > > I don't know if this is an error. > > when I do ./initdb -D ../data and execute pg_waldump like this, In the last > > part I got an error. > >

Re: pg_waldump got an error with waldump file generated by initdb

2022-07-10 Thread Andres Freund
Hi, On 2022-07-10 21:51:04 +0900, Dong Wook Lee wrote: > I don't know if this is an error. > when I do ./initdb -D ../data and execute pg_waldump like this, In the last > part I got an error. > > ``` > ./pg_waldump ../data/pg_wal/00010001 > ``` >

pg_waldump got an error with waldump file generated by initdb

2022-07-10 Thread Dong Wook Lee
Hi, hackers I don't know if this is an error. when I do ./initdb -D ../data and execute pg_waldump like this, In the last part I got an error. ``` ./pg_waldump ../data/pg_wal/00010001 ``` pg_waldump: error: error in WAL record at 0/140: invalid record length at 0/14

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-06-14 Thread Michael Paquier
On Tue, May 03, 2022 at 10:34:41AM +0200, Alvaro Herrera wrote: > I remember Greg Mullane posted a tool that attempts to correct page CRC > mismatches[1]. This new tool might be useful to feed healing attempts, > too. (It's of course not in any way a *solution*, because the page > might have been

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-05-03 Thread Alvaro Herrera
On 2022-Apr-27, Michael Paquier wrote: > On Tue, Apr 26, 2022 at 01:15:05PM -0500, David Christensen wrote: > > True. :-) This does seem like a tool geared towards "expert mode", so > > maybe we just assume if you need it you know what you're doing? > > This is definitely an expert mode toy. I r

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-05-02 Thread David Christensen
t; Would appreciate any additional feedback here. > > Best, > > David v4-pg_waldump-save-fpi.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-05-02 Thread David Christensen
es to pg_waldump.c through pgindent/perltidy to make them adhere to project standards. Threw in a rebase as well. Would appreciate any additional feedback here. Best, David v3-pg_waldump-save-fpi.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread Michael Paquier
On Tue, Apr 26, 2022 at 01:15:05PM -0500, David Christensen wrote: > True. :-) This does seem like a tool geared towards "expert mode", so > maybe we just assume if you need it you know what you're doing? This is definitely an expert mode toy. -- Michael signature.asc Description: PGP signature

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread David Christensen
On Mon, Apr 25, 2022 at 9:42 PM Michael Paquier wrote: > On Mon, Apr 25, 2022 at 10:24:52AM -0500, David Christensen wrote: > > On Mon, Apr 25, 2022 at 6:03 AM Bharath Rupireddy > > wrote: > >> Thanks for working on this. I'm just thinking if we can use these FPIs > >> to repair the corrupted pag

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread David Christensen
On Mon, Apr 25, 2022 at 9:54 PM Michael Paquier wrote: > On Mon, Apr 25, 2022 at 10:11:10AM -0500, David Christensen wrote: > > On Mon, Apr 25, 2022 at 1:11 AM Michael Paquier wrote: > >> I don't think that there is any need to rely on a new logic if there > >> is already some code in place able

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread Michael Paquier
On Mon, Apr 25, 2022 at 10:11:10AM -0500, David Christensen wrote: > On Mon, Apr 25, 2022 at 1:11 AM Michael Paquier wrote: >> I don't think that there is any need to rely on a new logic if there >> is already some code in place able to do the same work. See >> verify_dir_is_empty_or_create() in

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread Michael Paquier
On Mon, Apr 25, 2022 at 10:24:52AM -0500, David Christensen wrote: > On Mon, Apr 25, 2022 at 6:03 AM Bharath Rupireddy > wrote: >> Thanks for working on this. I'm just thinking if we can use these FPIs >> to repair the corrupted pages? I would like to understand more >> detailed usages of the FPIs

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread David Christensen
On Mon, Apr 25, 2022 at 6:03 AM Bharath Rupireddy wrote: > Thanks for working on this. I'm just thinking if we can use these FPIs > to repair the corrupted pages? I would like to understand more > detailed usages of the FPIs other than inspecting with pageinspect. My main use case was for being a

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread David Christensen
ere is an extension doing this [1] but having the feature > included in pg_waldump would be great. Cool, glad to see there is some interest; definitely some overlap in forensics inside and outside the database both, as there are different use cases for both. David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread David Christensen
On Mon, Apr 25, 2022 at 1:11 AM Michael Paquier wrote: > > On Sat, Apr 23, 2022 at 01:43:36PM -0500, David Christensen wrote: > > Hi Matthias, great point. Enclosed is a revised version of the patch > > that adds the fork identifier to the end if it's a non-main fork. > > Like Alvaro, I have seen

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread Bharath Rupireddy
On Sat, Apr 23, 2022 at 4:21 AM David Christensen wrote: > > Hi -hackers, > > Enclosed is a patch to allow extraction/saving of FPI from the WAL > stream via pg_waldump. > > Description from the commit: > > Extracts full-page images from the WAL stream into a target

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-24 Thread Michael Paquier
On Sat, Apr 23, 2022 at 01:43:36PM -0500, David Christensen wrote: > Hi Matthias, great point. Enclosed is a revised version of the patch > that adds the fork identifier to the end if it's a non-main fork. Like Alvaro, I have seen cases where this would have been really handy. So +1 from me, as

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-23 Thread David Christensen
rough `.` numerical in- or postfix in > the filename. > > -Matthias Hi Matthias, great point. Enclosed is a revised version of the patch that adds the fork identifier to the end if it's a non-main fork. Best, David v2-pg_waldump-save-fpi.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-23 Thread Matthias van de Meent
On Sat, 23 Apr 2022 at 00:51, David Christensen wrote: > > Hi -hackers, > > Enclosed is a patch to allow extraction/saving of FPI from the WAL > stream via pg_waldump. > > Description from the commit: > > Extracts full-page images from the WAL stream into a target direc

  1   2   3   >