Re: Adding comments to help understand psql hidden queries

2024-04-04 Thread David Christensen
On Thu, Apr 4, 2024 at 9:32 AM Peter Eisentraut wrote: > > On 03.04.24 19:16, David Christensen wrote: > > I removed _() in the output of the query/stars since there'd be no > > sensible existing translations for the constructed string, which > > included the query st

Re: Adding comments to help understand psql hidden queries

2024-04-03 Thread David Christensen
I got Greg's blessing on squashing the commits down, and now including a v4 with additional improvements on the output formatting front. Main changes: - all generated comments are the same width - width has been bumped to 80 - removed _() functions for consumers of the new output functions This

Re: Adding comments to help understand psql hidden queries

2024-03-22 Thread David Christensen
On Fri, Mar 22, 2024 at 9:47 AM Greg Sabino Mullane wrote: > > On Thu, Mar 21, 2024 at 6:20 PM Peter Eisentraut wrote: >> >> lines are supposed to align vertically. With your patch, the first line >> would have variable length depending on the command. > > > Yes, that is a good point. Aligning

Re: Avoiding inadvertent debugging mode for pgbench

2024-03-21 Thread David Christensen
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:tested, passed Did a quick review of this one; CFbot is now happy, local

Re: Adding comments to help understand psql hidden queries

2024-03-21 Thread David Christensen
Created the CF entry in commitfest 48 but didn't see it was already in 47; closing the CFEntry in 48. (Doesn't appear to be a different status than "Withdrawn"...)

Re: Adding comments to help understand psql hidden queries

2024-03-21 Thread David Christensen
Hi Jim, Thanks for the feedback. Enclosed is a v2 of this series(?) rebased and with that warning fixed; @Greg Sabino Mullane I just created a commit on your behalf with the message to hackers. I'm also creating a commit-fest entry for this thread. Best, David

Re: Constant Splitting/Refactoring

2024-03-13 Thread David Christensen
Here is a version 2 of this patch, rebased atop 97d85be365. As before, this is a cleanup/prerequisite patch series for the page features/reserved page size patches[1]. (Said patch series is going to be updated shortly.) This splits each of the 4 constants that care about page size into

Re: Constant Splitting/Refactoring

2024-02-14 Thread David Christensen
This should target PG 18, but that status is not available in the CF app yet, so just making a note here.

Constant Splitting/Refactoring

2024-02-07 Thread David Christensen
As part of the reserved page space/page features[1] work, there is a need to convert some of the existing constants into variables, since the size of those will no longer be fixed at compile time. At FOSDEM, there was some interest expressed about seeing what a cleanup patch like that would look

Re: Adding comments to help understand psql hidden queries

2024-02-01 Thread David Christensen
On Thu, Feb 1, 2024 at 4:34 PM Greg Sabino Mullane wrote: > > The use of the --echo-hidden flag in psql is used to show people the way psql > performs its magic for its backslash commands. None of them has more magic > than "\d relation", but it suffers from needing a lot of separate queries to

Re: [PATCHES] Post-special page storage TDE support

2023-12-22 Thread David Christensen
Hi again! Per some offline discussion with Stephen, I've continued to work on some modifications here; this particular patchset is intended to facilitate review by highlighting the mechanical nature of many of these changes. As such, I have taken the following approach to this rework: 0001 -

Re: [PATCHES] Post-special page storage TDE support

2023-11-29 Thread David Christensen
On Tue, Nov 7, 2023 at 6:20 PM Andres Freund wrote: > Hi, > > - IMO the patch touches many places it shouldn't need to touch, because of > essentially renaming a lot of existing macro names to *Limit, > necessitating modifying a lot of users. I think instead the few places > that > care

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread David Christensen
On Mon, Nov 13, 2023 at 2:52 PM Andres Freund wrote: > > > This scheme would open up space per page that would now be available for > > plenty of other things; the encoding in the header and the corresponding > > available space in the footer would seem to open up quite a few options > > now,

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread David Christensen
On Mon, Nov 13, 2023 at 2:27 PM Andres Freund wrote: > Hi, > > On 2023-11-08 18:47:56 -0800, Peter Geoghegan wrote: > > On Wed, Nov 8, 2023 at 6:04 AM Stephen Frost wrote: > > > In conversations with folks (my memory specifically is a discussion > with > > > Peter G, added to CC, and my

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread David Christensen
On Tue, Nov 7, 2023 at 6:20 PM Andres Freund wrote: > Hi, > > On 2023-05-09 17:08:26 -0500, David Christensen wrote: > > From 965309ea3517fa734c4bc89c144e2031cdf6c0c3 Mon Sep 17 00:00:00 2001 > > From: David Christensen > > Date: Tue, 9 May 2023 16:56:15 -0500 >

Re: [PATCHES] Post-special page storage TDE support

2023-11-08 Thread David Christensen
On Wed, Nov 8, 2023 at 8:04 AM Stephen Frost wrote: > Greetings, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2023-05-09 17:08:26 -0500, David Christensen wrote: > > > From 965309ea3517fa734c4bc89c144e2031cdf6c0c3 Mon Sep 17 00:00:00 2001 > > > From: D

Re: Moving forward with TDE [PATCH v3]

2023-11-08 Thread David Christensen
On Tue, Nov 7, 2023 at 5:49 PM Andres Freund wrote: > Hi, > > On 2023-11-06 09:56:37 -0500, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > I still am quite quite unconvinced that using the LSN as a nonce is a > good > > > design decision. > > > > This is a really

Re: Moving forward with TDE [PATCH v3]

2023-11-08 Thread David Christensen
On Tue, Nov 7, 2023 at 6:47 PM Andres Freund wrote: > Hi, > > On 2023-11-06 11:26:44 +0100, Matthias van de Meent wrote: > > On Sat, 4 Nov 2023 at 03:38, Andres Freund wrote: > > > On 2023-11-02 22:09:40 +0100, Matthias van de Meent wrote: > > > > I'm quite surprised at the significant number

Re: Moving forward with TDE [PATCH v3]

2023-11-06 Thread David Christensen
On Fri, Nov 3, 2023 at 9:53 PM Andres Freund wrote: > On 2023-11-02 19:32:28 -0700, Andres Freund wrote: > > > From 327e86d52be1df8de9c3a324cb06b85ba5db9604 Mon Sep 17 00:00:00 2001 > > > From: David Christensen > > > Date: Fri, 29 Sep 2023 15:16:00 -0400 >

Re: Moving forward with TDE [PATCH v3]

2023-11-06 Thread David Christensen
Hi, thanks for the detailed feedback here. I do think it's worth addressing the question Stephen raised as far as what we use for the IV[1]; whether LSN or something else entirely, and if so what. The choice of LSN here is fairly fundamental to the existing implementation, so if we decide to do

Re: Moving forward with TDE [PATCH v3]

2023-10-31 Thread David Christensen
On Tue, Oct 31, 2023 at 4:30 PM Bruce Momjian wrote: > On Tue, Oct 31, 2023 at 04:23:17PM -0500, David Christensen wrote: > > Greetings, > > > > I am including an updated version of this patch series; it has been > rebased > > onto 6ec62b7799 and reworked

Re: Initdb-time block size specification

2023-09-05 Thread David Christensen
On Tue, Sep 5, 2023 at 2:52 PM Hannu Krosing wrote: > > Something I also asked at this years Unconference - Do we currently > have Build Farm animals testing with different page sizes ? > > I'd say that testing all sizes from 4KB up (so 4, 8, 16, 32) should be > done at least before each release

Re: Initdb-time block size specification

2023-09-05 Thread David Christensen
On Tue, Sep 5, 2023 at 9:04 AM Robert Haas wrote: > > On Sat, Sep 2, 2023 at 3:09 PM Tomas Vondra > wrote: > > Except that no one is forcing you to actually go 130mph or 32mph, right? > > You make it seem like this patch forces people to use some other page > > size, but that's clearly not what

Re: Initdb-time block size specification

2023-08-31 Thread David Christensen
> I was definitely hand-waving additional implementation here for > non-native 128 bit support; the modulus algorithm as presented > requires 4 times the space as the divisor, so a uint16 implementation > should work for all 64-bit machines. Certainly open to other ideas or > implementations,

Re: Initdb-time block size specification

2023-08-31 Thread David Christensen
> + * pg_fastmod - calculates the modulus of a 32-bit number against a constant > + * divisor without using the division operator > + */ > +static inline uint32 pg_fastmod(uint32 n, uint32 divisor, uint64 fastinv) > +{ > +#ifdef HAVE_INT128 > + uint64_t lowbits = fastinv * n; > + return

Re: Initdb-time block size specification

2023-08-31 Thread David Christensen
Enclosed are TPC-H results for 1GB shared_buffers, 64MB work_mem on a 64GB laptop with SSD storage; everything else is default settings. TL;DR: unpatched version: 17.30 seconds, patched version: 17.15; there are some slight variations in runtime, but seems to be within the noise level at this

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 4:17 PM Andres Freund wrote: > > Hi, > > On 2023-06-30 15:05:54 -0500, David Christensen wrote: > > > I am fairly certain this is going to be causing substantial performance > > > regressions. I think we should reject this even if we

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 4:27 PM Tomas Vondra wrote: > On 6/30/23 23:11, Andres Freund wrote: > > ... > > > > If we really wanted to do this - but I don't think we do - I'd argue for > > working on the buildsystem support to build the postgres binary multiple > > times, for 4, 8, 16 kB BLCKSZ and

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 4:12 PM Tomas Vondra wrote: > > > > On 6/30/23 22:05, David Christensen wrote: > > On Fri, Jun 30, 2023 at 2:39 PM Andres Freund wrote: > >> > >> ... > >> > >> Besides this, I've not really heard any convincing justif

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 3:29 PM Andres Freund wrote: >> And indeed. Comparing e.g. TPC-H, I see *massive* regressions. Some queries > are the same, sobut others regress by up to 70% (although more commonly around > 10-20%). Hmm, that is definitely not good. > That's larger than I thought,

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 2:39 PM Andres Freund wrote: > > Hi, > > On 2023-06-30 12:35:09 -0500, David Christensen wrote: > > As discussed somewhat at PgCon, enclosed is v1 of a patch to provide > > variable block sizes; basically instead of BLCKSZ being a compile-time &g

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 1:14 PM Tomas Vondra wrote: > Do we really want to prefix the option with CLUSTER_? That seems to just > add a lot of noise into the patch, and I don't see much value in this > rename. I'd prefer keeping BLCKSZ and tweak just the couple places that > need "default" to use

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread David Christensen
> Adjusted as per the v2 attached. +1

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread David Christensen
On Tue, Jun 27, 2023 at 1:12 AM Michael Paquier wrote: > > Hi all, > (Fujii-san and David in CC.) > > Fujii-san has reported on Twitter that we had better add the TLI > number to what pg_waldump --save-fullpage generates for the file names > of the blocks, as it could be possible that we

Re: [PATCHES] Post-special page storage TDE support

2023-05-30 Thread David Christensen
On Fri, May 12, 2023 at 7:48 PM Stephen Frost wrote: > > Greetings, > > * David Christensen (david.christen...@crunchydata.com) wrote: > > Refreshing this with HEAD as of today, v4. > > Thanks for updating this! Thanks for the patience in my response here. > &g

Re: Kerberos delegation support in libpq and postgres_fdw

2023-04-07 Thread David Christensen
Ok, based on the interdiff there, I'm happy with that last change. Marking as Ready For Committer. Best, David

Re: Kerberos delegation support in libpq and postgres_fdw

2023-04-07 Thread David Christensen
Reviewed v8; largely looking good, though I notice this hunk, which may arguably be a bug fix, but doesn't appear to be relevant to this specific patch, so could probably be debated independently (and if a bug, should probably be backpatched): diff --git a/contrib/postgres_fdw/option.c

Re: Kerberos delegation support in libpq and postgres_fdw

2023-04-05 Thread David Christensen
On Wed, Apr 5, 2023 at 3:30 PM Stephen Frost wrote: > Greetings, > > * David Christensen (david...@pgguru.net) wrote: > > Did a code review pass here; here is some feedback. > > Thanks! > > > + /* If password was used to connect,

Re: Kerberos delegation support in libpq and postgres_fdw

2023-04-03 Thread David Christensen
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested Did a code review pass here; here is some feedback. + /* If

Re: Improving btree performance through specializing by key shape, take 2

2023-01-12 Thread David Christensen
Hi Matthias, I'm going to look at this patch series if you're still interested. What was the status of your final performance testing for the 0008 patch alone vs the specialization series? Last I saw on the thread you were going to see if the specialization was required or not. Best, David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread David Christensen
> On Dec 26, 2022, at 1:29 AM, Michael Paquier wrote: > > On Sat, Dec 24, 2022 at 06:23:29PM +0530, Bharath Rupireddy wrote: >> Thanks for the patch. I've made the above change as well as renamed >> the test file name to be save_fpi.pl, everything else remains the same >> as v11. Here's the v12

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-23 Thread David Christensen
On Fri, Dec 23, 2022 at 12:57 PM David Christensen wrote: > > On Wed, Dec 21, 2022 at 5:47 AM Bharath Rupireddy > wrote: [snip] > > 2. +$node->init(extra => ['-k'], allows_streaming => 1); > > When enabled with allows_streaming, there are a bunch of things tha

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-23 Thread David Christensen
On Mon, Dec 19, 2022 at 12:23 AM Michael Paquier wrote: > > On Thu, Dec 15, 2022 at 05:17:46PM -0600, David Christensen wrote: > > On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier > > wrote: > > This v10 should incorporate your feedback as well as Bharath's. > >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-23 Thread David Christensen
On Wed, Dec 21, 2022 at 5:47 AM Bharath Rupireddy wrote: > > On Fri, Dec 16, 2022 at 4:47 AM David Christensen > wrote: > > > > On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier > > wrote: > > > > > > On Wed, Dec 14, 2022 at 04:44:34PM -0600, David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-15 Thread David Christensen
On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier wrote: > > On Wed, Dec 14, 2022 at 04:44:34PM -0600, David Christensen wrote: > > I can get one sent in tomorrow. This v10 should incorporate your feedback as well as Bharath's. > -XLogRecordHasFPW(XLogReaderState *record) >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-14 Thread David Christensen
Hi Bharath, I can get one sent in tomorrow. Thanks, David

Re: Moving forward with TDE

2022-11-17 Thread David Christensen
Hi Dilip, Thanks for the feedback here. I will review the docs changes and add to my tree. Best, David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-17 Thread David Christensen
On Wed, Nov 16, 2022 at 3:30 AM Bharath Rupireddy wrote: > 1. > -if (config.filter_by_fpw && !XLogRecordHasFPW(xlogreader_state)) > +if (config.filter_by_fpw && !XLogRecordHasFPI(xlogreader_state)) > These changes are not related to this feature, hence renaming those >

Re: Moving forward with TDE

2022-11-17 Thread David Christensen
Hi Jacob, Thanks, I've added this patch in my tree [1]. (For now, just adding fixes and the like atop the original separate patches, but will eventually get things winnowed down into probably the same 12 parts the originals were reviewed in. Best, David [1]

ScanSourceDatabasePgClass

2022-11-16 Thread David Christensen
Hi Robert, In 9c08aea6a you introduce the block-by-block strategy for creating a copy of the database. In the main loop, this utilizes this call: buf = ReadBufferWithoutRelcache(rlocator, MAIN_FORKNUM, blkno, RBM_NORMAL, bstrategy, false); Here, the last parameter is "false" for the permanence

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-15 Thread David Christensen
Enclosed is v9. - code style consistency (FPI instead of FPW) internally. - cleanup of no-longer needed checksum-related pieces from code and tests. - test cleanup/simplification. - other comment cleanup. Passes all CI checks. Best, David

Re: Moving forward with TDE

2022-11-15 Thread David Christensen
> On Nov 15, 2022, at 1:08 PM, Jacob Champion wrote: > > On Mon, Oct 24, 2022 at 9:29 AM David Christensen > wrote: >> I would love to open a discussion about how to move forward and get >> some of these features built out. The historical threads here are >&

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-15 Thread David Christensen
On Tue, Nov 15, 2022 at 4:41 AM Bharath Rupireddy wrote: > > On Tue, Nov 15, 2022 at 1:29 AM David Christensen > wrote: > > > > Enclosed is v8, which uses the replication slot method to retain WAL > > as well as fsync'ing the output directory when everything is

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-14 Thread David Christensen
Enclosed is v8, which uses the replication slot method to retain WAL as well as fsync'ing the output directory when everything is done. v8-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-str.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-14 Thread David Christensen
On Fri, Nov 11, 2022 at 4:57 AM Bharath Rupireddy wrote: > > Well if it's not the same output then I guess you're right and there's > > not a use for the `--fixup` mode. By the same token, I'd say > > calculating/setting the checksum also wouldn't need to be done, we > > should just include the

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-11 Thread David Christensen
On Fri, Nov 11, 2022 at 8:15 AM Justin Pryzby wrote: > > On Wed, Nov 09, 2022 at 02:37:29PM -0600, David Christensen wrote: > > On Wed, Nov 9, 2022 at 2:08 PM David Christensen > > wrote: > > > Justin sez: > > > > I was wondering if there's any re

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-10 Thread David Christensen
On Thu, Nov 10, 2022 at 2:33 AM Bharath Rupireddy wrote: > > On Thu, Nov 10, 2022 at 1:31 AM David Christensen > wrote: > > > > Thanks for providing the v7 patch, please see my comments and responses below. Hi Bharath, Thanks for the feedback. > > > 2. I'm una

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread David Christensen
On Wed, Nov 9, 2022 at 2:08 PM David Christensen wrote: > Justin sez: > > I was wondering if there's any reason to do "CREATE DATABASE". The vast > > majority of TAP tests don't. > > > > $ git grep -ho 'safe_psql[^ ]*' '*pl' |sort |uniq -c |sort -nr

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread David Christensen
> > 6. > > +if (dir_status == 0 && mkdir(config.save_fpw_path, 0700) < 0) > > Use pg_dir_create_mode instead of hard-coded 0007? > > I think I thought of that when I first looked at the patch ... but, I'm > not sure, since it says: > > src/include/common/file_perm.h-/* Modes for creating

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread David Christensen
On Wed, Nov 9, 2022 at 6:30 AM Bharath Rupireddy wrote: > > On Wed, Nov 9, 2022 at 5:08 AM David Christensen > wrote: > > > > Enclosed is v6, which squashes your refactor and adds the additional > > recent suggestions. > > Thanks for working on this feature.

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-08 Thread David Christensen
Enclosed is v6, which squashes your refactor and adds the additional recent suggestions. Thanks! v6-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-str.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-08 Thread David Christensen
On Tue, Nov 8, 2022 at 4:45 PM Justin Pryzby wrote: > > On Mon, Nov 07, 2022 at 05:01:01PM -0600, David Christensen wrote: > > Hi Justin et al, > > > > Enclosed is v5 of this patch which now passes the CirrusCI checks for > > all supported OSes. I went ahead and r

Re: [PATCHES] Post-special page storage TDE support

2022-11-08 Thread David Christensen
Looking into some CF bot failures which didn't show up locally. Will send a v3 when resolved.

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-07 Thread David Christensen
Hi Justin et al, Enclosed is v5 of this patch which now passes the CirrusCI checks for all supported OSes. I went ahead and reworked the test a bit so it's a little more amenable to the OS-agnostic approach for testing. Best, David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-04 Thread David Christensen
On Fri, Nov 4, 2022 at 1:38 PM Justin Pryzby wrote: > > > As I recall, that's due to relying on "cp". And "rsync", which > > > shouldn't be assumed to exist by regression tests). Will poke around other TAP tests to see if there's a more consistent interface, what perl version we can assume and

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-04 Thread David Christensen
On Nov 4, 2022, at 9:02 AM, Justin Pryzby wrote: > > On Fri, Nov 04, 2022 at 11:52:59AM +0900, Ian Lawrence Barwick wrote: >> 2022年5月3日(火) 8:45 David Christensen : >>> >>> ...and pushing a couple fixups pointed out by cfbot, so here's v4. >> >> cf

Re: [PATCHES] Post-special page storage TDE support

2022-10-28 Thread David Christensen
Hi Matthias, > Did you read the related thread with related discussion from last June, "Re: > better page-level checksums" [0]? In that I argued that space at the end of a > page is already allocated for the AM, and that reserving variable space at > the end of the page for non-AM usage is

Re: [PATCHES] Post-special page storage TDE support

2022-10-25 Thread David Christensen
> > Explicitly > > locking (assuming you stay in your lane) should only need to guard > > against access from other > > backends of this type if using shared buffers, so will be use-case > > dependent. > > I'm not sure what you mean here? I'm mainly pointing out that the specific code that

[PATCH] comment fixes for delayChkptFlags

2022-10-14 Thread David Christensen
Enclosed is a trivial fix for a typo and misnamed field I noted when doing some code review. Best, David 0001-fix-comment-typos-for-delayChkptFlags.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-05-02 Thread David Christensen
...and pushing a couple fixups pointed out by cfbot, so here's v4. On Mon, May 2, 2022 at 8:42 AM David Christensen wrote: > > Enclosed is v3 of this patch; this adds two modes for this feature, > one with the raw page `--save-fullpage/-W` and one with the > LSN+checksum fixups `--s

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-05-02 Thread David Christensen
Enclosed is v3 of this patch; this adds two modes for this feature, one with the raw page `--save-fullpage/-W` and one with the LSN+checksum fixups `--save-fullpage-fixup/-X`. I've added at least some basic sanity-checking of the underlying feature, as well as run the test file and the changes to

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread David Christensen
On Mon, Apr 25, 2022 at 9:42 PM Michael Paquier wrote: > On Mon, Apr 25, 2022 at 10:24:52AM -0500, David Christensen wrote: > > On Mon, Apr 25, 2022 at 6:03 AM Bharath Rupireddy > > wrote: > >> Thanks for working on this. I'm just thinking if we can use these FPIs >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread David Christensen
On Mon, Apr 25, 2022 at 9:54 PM Michael Paquier wrote: > On Mon, Apr 25, 2022 at 10:11:10AM -0500, David Christensen wrote: > > On Mon, Apr 25, 2022 at 1:11 AM Michael Paquier wrote: > >> I don't think that there is any need to rely on a new logic if there > >> is alre

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread David Christensen
On Mon, Apr 25, 2022 at 6:03 AM Bharath Rupireddy wrote: > Thanks for working on this. I'm just thinking if we can use these FPIs > to repair the corrupted pages? I would like to understand more > detailed usages of the FPIs other than inspecting with pageinspect. My main use case was for being

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread David Christensen
On Mon, Apr 25, 2022 at 2:00 AM Drouvot, Bertrand wrote: > > Hi, > > On 4/25/22 8:11 AM, Michael Paquier wrote: > > On Sat, Apr 23, 2022 at 01:43:36PM -0500, David Christensen wrote: > >> Hi Matthias, great point. Enclosed is a revised version of the patch > &g

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread David Christensen
On Mon, Apr 25, 2022 at 1:11 AM Michael Paquier wrote: > > On Sat, Apr 23, 2022 at 01:43:36PM -0500, David Christensen wrote: > > Hi Matthias, great point. Enclosed is a revised version of the patch > > that adds the fork identifier to the end if it's a non-main fork. >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-23 Thread David Christensen
On Sat, Apr 23, 2022 at 9:49 AM Matthias van de Meent wrote: > Regardless of my (lack of) opinion on the inclusion of this patch in > PG (I did not significantly review this patch); I noticed that you do > not yet identify the 'fork' of the FPI in the file name. > > A lack of fork identifier in

[PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-22 Thread David Christensen
Hi -hackers, Enclosed is a patch to allow extraction/saving of FPI from the WAL stream via pg_waldump. Description from the commit: Extracts full-page images from the WAL stream into a target directory, which must be empty or not exist. These images are subject to the same filtering rules as

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-24 Thread David Christensen
> On Mar 24, 2022, at 4:12 PM, Thomas Munro wrote: > > On Fri, Mar 25, 2022 at 1:43 AM David Christensen > wrote: >>>> On Mar 24, 2022, at 6:43 AM, Thomas Munro wrote: >>> On Fri, Mar 25, 2022 at 12:26 AM Thomas Munro >>> wrote: >>>&

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-24 Thread David Christensen
> On Mar 24, 2022, at 6:43 AM, Thomas Munro wrote: > > On Fri, Mar 25, 2022 at 12:26 AM Thomas Munro wrote: >>> On Fri, Mar 25, 2022 at 12:01 AM Peter Eisentraut >>> wrote: >>> Or even: Why are we exposing fork *numbers* in the user interface? >>> Even low-level tools such as pageinspect

Re: [PATCH] Proof of concept for GUC improvements

2022-03-22 Thread David Christensen
> On Mar 21, 2022, at 7:53 PM, Tom Lane wrote: > > Andres Freund writes: >> My impression is that there's not a lot of enthusiasm for the concept? If >> that's true we maybe ought to mark the CF entry as rejected? > > Yeah, I'm kind of leaning that way too. I don't see how we can >

Re: [PATCH] pgbench: add multiconnect option

2022-03-22 Thread David Christensen
On Sat, Mar 19, 2022 at 11:43 AM Fabien COELHO wrote: > > Hi Sami, > > > Pgbench is a simple benchmark tool by design, and I wonder if adding > > a multiconnect feature will cause pgbench to be used incorrectly. > > Maybe, but I do not see how it would be worse that what pgbench already >

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-21 Thread David Christensen
On Mon, Mar 21, 2022 at 4:39 PM Thomas Munro wrote: [snip] I guess you did this because init fork references aren't really > expected in the WAL, but I think it's more consistent to allow up to > MAX_FORKNUM, not least because your documentation mentions 3 as a > valid value. So I adjust this

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-21 Thread David Christensen
Updated to include the V3 fixes as well as the unsigned int/enum fix. > v4-0001-Add-additional-filtering-options-to-pg_waldump.patch Description: Binary data

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-21 Thread David Christensen
On Sun, Mar 20, 2022 at 11:56 PM Thomas Munro wrote: > On Mon, Mar 21, 2022 at 4:36 PM Thomas Munro > wrote: > > On Sat, Feb 26, 2022 at 7:58 AM David Christensen > > wrote: > > > Attached is V2 with additional feedback from this email, as well as > the specifi

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-25 Thread David Christensen
Bharath Rupireddy writes: > On Fri, Feb 25, 2022 at 12:36 AM David Christensen > wrote: >> >> Greetings, >> >> This patch adds the ability to specify a RelFileNode and optional BlockNum >> to limit output of >> pg_waldump records to only those which

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-25 Thread David Christensen
On Fri, Feb 25, 2022 at 7:08 AM Japin Li wrote: > > On Fri, 25 Feb 2022 at 20:48, David Christensen < > david.christen...@crunchydata.com> wrote: > >> Cool. I think we can report an error instead of reading wal files, > >> if the tablespace, database, or rel

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-25 Thread David Christensen
On Fri, Feb 25, 2022 at 7:33 AM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > Thanks for the patch. This is not adding something that users can't do > right now, but definitely improves the usability of the pg_waldump as > it avoids external filterings. Also, it can give

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-25 Thread David Christensen
> Cool. I think we can report an error instead of reading wal files, > if the tablespace, database, or relation is invalid. Does there any > WAL record that has invalid tablespace, database, or relation OID? The only sort of validity check we could do here is range checking for the underlying

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-24 Thread David Christensen
Added to commitfest as: https://commitfest.postgresql.org/37/3565/

[PATCH] add relation and block-level filtering to pg_waldump

2022-02-24 Thread David Christensen
depending on how the blocks are specified. This currently affects only the main fork, but we could presumably add the option to filter by fork as well, if that is considered useful. Best, David >From 9194b2cb07172e636030b9b4e979b7f2caf7cbc0 Mon Sep 17 00:00:00 2001 From: David Christensen D

Re: DELETE CASCADE

2022-01-31 Thread David Christensen
On Sun, Jan 30, 2022 at 9:47 PM Julien Rouhaud wrote: > Hi, > > On Tue, Jan 25, 2022 at 10:26:53PM +0800, Julien Rouhaud wrote: > > > > It's been almost 4 months since your last email, and almost 2 weeks > since the > > notice that this patch doesn't apply anymore. Without update in the next >

Re: [PATCH] Proof of concept for GUC improvements

2022-01-12 Thread David Christensen
> Hi, > > According to the cfbot, the patch doesn't apply anymore and needs a > rebase: http://cfbot.cputube.org/patch_36_3290.log V4 rebased attached. special-guc-values-v4.patch Description: Binary data

Re: CREATE ROLE IF NOT EXISTS

2021-11-22 Thread David Christensen
On Mon, Nov 22, 2021 at 6:49 AM Daniel Gustafsson wrote: > > On 10 Nov 2021, at 18:14, David Christensen < > david.christen...@crunchydata.com> wrote: > > > Modulo other issues/discussions, here is a version of this patch.. > > This patch fails to compile since

Re: CREATE ROLE IF NOT EXISTS

2021-11-10 Thread David Christensen
Modulo other issues/discussions, here is a version of this patch that implements CREATE OR REPLACE ROLE just by handing off to AlterRole if it's determined that the role already exists; presumably any/all additional considerations would need to be added in both places were there a separate code

Re: CREATE ROLE IF NOT EXISTS

2021-11-09 Thread David Christensen
On Tue, Nov 9, 2021 at 10:22 AM Stephen Frost wrote: > Greetings, > > * David Christensen (david.christen...@crunchydata.com) wrote: > > Well, the CREATE OR REPLACE via just setting the role's attributes > > explicitly based on what you passed it could work (not stric

Re: CREATE ROLE IF NOT EXISTS

2021-11-09 Thread David Christensen
On Tue, Nov 9, 2021 at 9:55 AM Mark Dilger wrote: > > On Nov 9, 2021, at 7:36 AM, David Christensen < > david.christen...@crunchydata.com> wrote: > > > > If CINE semantics are at issue, what about the CREATE OR REPLACE > semantics with some sort of merge into th

Re: CREATE ROLE IF NOT EXISTS

2021-11-09 Thread David Christensen
On Mon, Nov 8, 2021 at 1:22 PM Mark Dilger wrote: > > On Nov 8, 2021, at 10:38 AM, Stephen Frost wrote: > > > I don't quite follow this. The entire point of Alice writing a script > > that uses IF NOT EXISTS is to have that command not fail if, indeed, > > that role already exists, but for the

Re: CREATE ROLE IF NOT EXISTS

2021-11-03 Thread David Christensen
> > > This fails the roleattributes test in "make check", with what seems to be a > trivial change in the output. Can you please submit a rebased version > fixing > the test? > Updated version attached. David CREATE-ROLE-IF-NOT-EXISTS-v2.patch Description: Binary data

Re: [PATCH] Proof of concept for GUC improvements

2021-11-03 Thread David Christensen
> On Nov 3, 2021, at 5:35 AM, Daniel Gustafsson wrote: > >  >> >>> On 15 Oct 2021, at 23:54, Cary Huang wrote: >> >> I scanned through the GUC list and found that the following parameters can >> potentially be categorized in the "special_disabled0" group, just for your >> reference. > > >

  1   2   >