out of memory in crosstab()

2022-11-15 Thread Amit Langote
Hi, A customer seems to have run into $subject. Here's a reproducer they shared: CREATE TABLE test (id integer, category integer, rate numeric); INSERT INTO test SELECT x.id, y.category, random() * 10 AS rate FROM generate_series(1, 100) AS x(id) INNER JOIN generate_series(1,

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread John Naylor
On Wed, Nov 16, 2022 at 12:33 PM Masahiko Sawada wrote: > > On Wed, Nov 16, 2022 at 1:46 PM John Naylor > wrote: > > > > > > On Tue, Nov 15, 2022 at 11:59 AM Masahiko Sawada wrote: > > > Thanks! Please let me know if there is something I can help with. > > > > I didn't get very far because the

Re: Avoid overhead open-close indexes (catalog updates)

2022-11-15 Thread Michael Paquier
On Wed, Nov 16, 2022 at 06:58:01AM +0900, Michael Paquier wrote: > This one has been left out on purpose. I was tempting to use > WithInfo() with a CatalogIndexState opened optionally but I got the > impression that it makes the code a bit harder to follow and > AddRoleMems() is already complex

Re: HOT chain validation in verify_heapam()

2022-11-15 Thread Himanshu Upadhyaya
On Thu, Nov 10, 2022 at 3:38 AM Andres Freund wrote: > > > + } > > + > > + /* > > + * Loop over offset and populate predecessor array from > all entries > > + * that are present in successor array. > > + */ > > +

Re: [PoC] Let libpq reject unexpected authentication requests

2022-11-15 Thread Michael Paquier
On Thu, Oct 20, 2022 at 11:36:34AM -0700, Jacob Champion wrote: > Maybe I should just add a basic Assert here, to trip if someone adds a > new SASL mechanism, and point that lucky person to this thread with a > comment? I am beginning to look at the last version proposed, which has been marked as

Re: Small miscellaneous fixes

2022-11-15 Thread Michael Paquier
On Tue, Oct 04, 2022 at 08:23:16AM -0300, Ranier Vilela wrote: > Both are correct, I missed the pqsignal calls. > > Attached patch to change this. The change for pgbench is missing and this is only changing pg_test_fsync. Switching to sig_atomic_t would be fine on non-WIN32 as these are used in

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Andres Freund
On 2022-11-15 20:48:56 -0800, Peter Geoghegan wrote: > On Tue, Nov 15, 2022 at 5:29 PM Andres Freund wrote: > > If we want to focus on the mvcc affects we could just go for something like > > snapshotConflictHorizon or such. > > Okay, let's go with snapshotConflictHorizon. I'll use that name in

Re: HOT chain validation in verify_heapam()

2022-11-15 Thread Andres Freund
Hi, On 2022-11-14 15:07:05 -0800, Peter Geoghegan wrote: > I'd really like to know if the scary HOT chain freezing scenario is > possible, for the very obvious reason. Have you tried to write a test > case for that? I tried. Unfortunately, even if the bug exists, we currently don't have the

Re: when the startup process doesn't (logging startup delays)

2022-11-15 Thread Bharath Rupireddy
On Tue, Nov 15, 2022 at 10:55 PM Robert Haas wrote: > > On Tue, Nov 15, 2022 at 8:33 AM Bharath Rupireddy > wrote: > > Please review the v2 patch. > > It seems to me that this will call disable_startup_progress_timeout > once per WAL record, which seems like an unnecessary expense. How > about

Re: Switching XLog source from archive to streaming when primary available

2022-11-15 Thread Bharath Rupireddy
On Wed, Nov 16, 2022 at 9:38 AM Ian Lawrence Barwick wrote: > > While reviewing the patch backlog, we have determined that this patch adds > one or more TAP tests but has not added the test to the "meson.build" file. Thanks for pointing it out. Yeah, the test wasn't picking up on meson builds. I

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread Masahiko Sawada
On Wed, Nov 16, 2022 at 2:17 PM John Naylor wrote: > > > > On Wed, Nov 16, 2022 at 11:46 AM John Naylor > wrote: > > > > > > On Tue, Nov 15, 2022 at 11:59 AM Masahiko Sawada > > wrote: > > > Thanks! Please let me know if there is something I can help with. > > > > I didn't get very far

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread Masahiko Sawada
On Wed, Nov 16, 2022 at 1:46 PM John Naylor wrote: > > > On Tue, Nov 15, 2022 at 11:59 AM Masahiko Sawada > wrote: > > Thanks! Please let me know if there is something I can help with. > > I didn't get very far because the tests fail on 0004 in rt_verify_node: > > TRAP: failed

Re: Allow file inclusion in pg_hba and pg_ident files

2022-11-15 Thread Julien Rouhaud
Le mer. 16 nov. 2022 à 13:01, Ian Lawrence Barwick a écrit : > 2022年11月14日(月) 14:41 Michael Paquier : > > > > On Sat, Nov 12, 2022 at 04:13:53PM +0800, Julien Rouhaud wrote: > > > It's looks good to me. I agree that file name and line number should > be enough > > > to diagnose any unexpected

Re: Memory leak in adjust_data_dir

2022-11-15 Thread Japin Li
On Wed, 16 Nov 2022 at 12:19, Tom Lane wrote: > Japin Li writes: >> Hi, hackers, > > ITYM pgsql-hackers, this is off-topic here. > Sorry for typo the email address. >> When I'm reviewing patch [1], I find there is a memory leak in >> adjust_data_dir(), the cmd was allocated by psprintf(),

Re: New strategies for freezing, advancing relfrozenxid early

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 19:02:12 -0800, Peter Geoghegan wrote: > From 352867c5027fae6194ab1c6480cd326963e201b1 Mon Sep 17 00:00:00 2001 > From: Peter Geoghegan > Date: Sun, 12 Jun 2022 15:46:08 -0700 > Subject: [PATCH v6 1/6] Add page-level freezing to VACUUM. > > Teach VACUUM to decide on whether or

Re: pg_rewind: warn when checkpoint hasn't happened after promotion

2022-11-15 Thread kuroda . keisuke
Hi, hackers The issue here is pg_rewind looks into control file to determine the soruce timeline, because the control file is not updated until the first checkpoint ends after promotion finishes, even though file blocks are already diverged. Even in that case history file for the new timeline

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread John Naylor
On Wed, Nov 16, 2022 at 11:46 AM John Naylor wrote: > > > On Tue, Nov 15, 2022 at 11:59 AM Masahiko Sawada wrote: > > Thanks! Please let me know if there is something I can help with. > > I didn't get very far because the tests fail on 0004 in rt_verify_node: > > TRAP: failed

Re: Allow file inclusion in pg_hba and pg_ident files

2022-11-15 Thread Ian Lawrence Barwick
2022年11月14日(月) 14:41 Michael Paquier : > > On Sat, Nov 12, 2022 at 04:13:53PM +0800, Julien Rouhaud wrote: > > It's looks good to me. I agree that file name and line number should be > > enough > > to diagnose any unexpected error. > > Thanks for checking. I have looked at 0001 and 0002 again

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Peter Geoghegan
On Tue, Nov 15, 2022 at 5:29 PM Andres Freund wrote: > If we want to focus on the mvcc affects we could just go for something like > snapshotConflictHorizon or such. Okay, let's go with snapshotConflictHorizon. I'll use that name in the next revision, which I should be able to post tomorrow. --

Re: vacuumlo: add test to vacuumlo for test coverage

2022-11-15 Thread Ian Lawrence Barwick
2022年9月3日(土) 17:28 Dong Wook Lee : > > Hi hackers, > I write a tiny patch about vacuumlo to improve test coverage. > I hope my work is meaningful. Hi While reviewing the patch backlog, we have determined that this patch adds one or more TAP tests but has not added the test to the "meson.build"

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread John Naylor
On Tue, Nov 15, 2022 at 11:59 AM Masahiko Sawada wrote: > Thanks! Please let me know if there is something I can help with. I didn't get very far because the tests fail on 0004 in rt_verify_node: TRAP: failed Assert("n4->chunks[i - 1] < n4->chunks[i]"), File: "../src/backend/lib/radixtree.c",

Re: logical decoding and replication of sequences, take 2

2022-11-15 Thread Ian Lawrence Barwick
2022年11月12日(土) 7:49 Tomas Vondra : > > Hi, > > I noticed on cfbot the patch no longer applies, so here's a rebased > version. Most of the breakage was due to the column filtering reworks, > grammar changes etc. A lot of bitrot, but mostly mechanical stuff. (...) Hi Thanks for the update patch.

Re: Testing autovacuum wraparound (including failsafe)

2022-11-15 Thread Ian Lawrence Barwick
2022年6月30日(木) 10:40 Masahiko Sawada : > > Hi, > > On Tue, Feb 1, 2022 at 11:58 AM Masahiko Sawada wrote: > > > > On Fri, Jun 11, 2021 at 10:19 AM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2021-06-10 16:42:01 +0300, Anastasia Lubennikova wrote: > > > > Cool. Thank you for working on

Re: Hash index build performance tweak from sorting

2022-11-15 Thread Simon Riggs
On Wed, 21 Sept 2022 at 02:32, David Rowley wrote: > > I took this patch for a spin and saw a 2.5% performance increase using > the random INT test that Tom posted. The index took an average of > 7227.47 milliseconds on master and 7045.05 with the patch applied. Thanks for the review, apologies

Re: SUBTRANS: Minimizing calls to SubTransSetParent()

2022-11-15 Thread Dilip Kumar
On Wed, Nov 16, 2022 at 8:41 AM Simon Riggs wrote: > > > No, that's not what XidInMVCCSnapshot does. If snapshot->suboverflowed > > is set (ie, somebody somewhere/somewhen overflowed), then it does > > SubTransGetTopmostTransaction and searches only the xips with the result. > > This behavior

Re: Suppressing useless wakeups in walreceiver

2022-11-15 Thread Nathan Bossart
On Wed, Nov 16, 2022 at 04:57:08PM +1300, Thomas Munro wrote: > On Tue, Nov 15, 2022 at 5:49 PM Nathan Bossart > wrote: >> Another option might be to just force initial reply/feedback messages when >> streaming starts. The attached patch improves src/test/recovery test >> runtime just like the

Re: Add test module for Custom WAL Resource Manager feature

2022-11-15 Thread Jeff Davis
On Wed, 2022-11-16 at 10:27 +0900, Michael Paquier wrote: > On Wed, Nov 16, 2022 at 10:26:32AM +0900, Michael Paquier wrote: > > Not many buildfarm members test 32b builds, but lapwing does. > > Well, it didn't take long: >

Re: Switching XLog source from archive to streaming when primary available

2022-11-15 Thread Ian Lawrence Barwick
2022年10月18日(火) 11:02 Bharath Rupireddy : > > On Tue, Oct 11, 2022 at 8:40 AM Nathan Bossart > wrote: > > > > On Mon, Oct 10, 2022 at 11:33:57AM +0530, Bharath Rupireddy wrote: > > > On Mon, Oct 10, 2022 at 3:17 AM Nathan Bossart > > > wrote: > > >> I wonder if it would be better to simply

Re: Skipping schema changes in publication

2022-11-15 Thread Ian Lawrence Barwick
2022年11月7日(月) 22:39 vignesh C : > > On Fri, 4 Nov 2022 at 08:19, Ian Lawrence Barwick wrote: > > > > Hi > > > > cfbot reports the patch no longer applies [1]. As CommitFest 2022-11 is > > currently underway, this would be an excellent time to update the patch. > > > > [1]

Re: logical replication restrictions

2022-11-15 Thread Ian Lawrence Barwick
2022年11月14日(月) 10:09 Takamichi Osumi (Fujitsu) : > > On Tuesday, November 8, 2022 2:27 PM Kuroda, Hayato/黒田 隼人 > wrote: > > If you could not allocate a time to discuss this problem because of other > > important tasks or events, we would like to take over the thread and modify > > your patch. >

Re: Suppressing useless wakeups in walreceiver

2022-11-15 Thread Thomas Munro
On Tue, Nov 15, 2022 at 5:49 PM Nathan Bossart wrote: > Another option might be to just force initial reply/feedback messages when > streaming starts. The attached patch improves src/test/recovery test > runtime just like the previous one I posted. Yeah, looks good in my tests. I think we just

Re: closing file in adjust_data_dir

2022-11-15 Thread Ted Yu
On Tue, Nov 15, 2022 at 7:26 PM Japin Li wrote: > > On Wed, 16 Nov 2022 at 11:15, Ted Yu wrote: > > On Tue, Nov 15, 2022 at 7:12 PM Japin Li wrote: > >> After some rethinking, I find the origin code do not have problems. > >> > >> If fd is NULL or fgets() returns NULL, the process exits.

Re: closing file in adjust_data_dir

2022-11-15 Thread Japin Li
On Wed, 16 Nov 2022 at 11:15, Ted Yu wrote: > On Tue, Nov 15, 2022 at 7:12 PM Japin Li wrote: >> After some rethinking, I find the origin code do not have problems. >> >> If fd is NULL or fgets() returns NULL, the process exits. Otherwise, we >> call >> pclose() to close fd. The code isn't

Re: libpq compression (part 2)

2022-11-15 Thread Justin Pryzby
On Mon, Nov 14, 2022 at 07:44:24PM -0800, Andrey Borodin wrote: > patchset needs a heavy rebase. If no one shows up to fix it I'll do Despite what its git timestamp says, this is based on the most recent patch from January, which I've had floating around since then. It needed to be rebased over

Re: closing file in adjust_data_dir

2022-11-15 Thread Ted Yu
On Tue, Nov 15, 2022 at 7:12 PM Japin Li wrote: > > On Wed, 16 Nov 2022 at 10:52, Ted Yu wrote: > > On Tue, Nov 15, 2022 at 6:35 PM Japin Li wrote: > >> > >> fd = popen(cmd, "r"); > >> - if (fd == NULL || fgets(filename, sizeof(filename), fd) == NULL > || > >> pclose(fd) != 0) >

Re: closing file in adjust_data_dir

2022-11-15 Thread Japin Li
On Wed, 16 Nov 2022 at 10:52, Ted Yu wrote: > On Tue, Nov 15, 2022 at 6:35 PM Japin Li wrote: >> >> fd = popen(cmd, "r"); >> - if (fd == NULL || fgets(filename, sizeof(filename), fd) == NULL || >> pclose(fd) != 0) >> + if (fd == NULL || fgets(filename, sizeof(filename), fd)

Re: SUBTRANS: Minimizing calls to SubTransSetParent()

2022-11-15 Thread Simon Riggs
On Wed, 16 Nov 2022 at 00:09, Tom Lane wrote: > > Simon Riggs writes: > > On Tue, 15 Nov 2022 at 21:03, Tom Lane wrote: > >> The subxidStatus.overflowed check quoted above has a similar sort > >> of myopia: it's checking whether our current transaction has > >> already suboverflowed. But (a)

Re: closing file in adjust_data_dir

2022-11-15 Thread Ted Yu
On Tue, Nov 15, 2022 at 6:35 PM Japin Li wrote: > > On Wed, 16 Nov 2022 at 10:06, Ted Yu wrote: > >> Hi, > > That check is a few line above: > > > > + if (fd == NULL || fgets(filename, sizeof(filename), fd) == NULL) > > { > > > > Cheers > > Thanks for the explanation. Comment on

Re: Allow file inclusion in pg_hba and pg_ident files

2022-11-15 Thread Julien Rouhaud
On Tue, Nov 15, 2022 at 08:46:55AM +0900, Michael Paquier wrote: > On Mon, Nov 14, 2022 at 03:47:27PM +0800, Julien Rouhaud wrote: > > > > If you have an include_dir directive and multiple files have wrong > > permission > > (or maybe broken symlink or something like that), you will get multiple

Re: Slow standby snapshot

2022-11-15 Thread Simon Riggs
On Wed, 16 Nov 2022 at 00:15, Tom Lane wrote: > > Andres Freund writes: > > On 2022-11-15 23:14:42 +, Simon Riggs wrote: > >> Hence more frequent compression is effective at reducing the overhead. > >> But too frequent compression slows down the startup process, which > >> can't then keep

Re: closing file in adjust_data_dir

2022-11-15 Thread Japin Li
On Wed, 16 Nov 2022 at 10:06, Ted Yu wrote: >> Hi, > That check is a few line above: > > + if (fd == NULL || fgets(filename, sizeof(filename), fd) == NULL) > { > > Cheers Thanks for the explanation. Comment on v2 patch. fd = popen(cmd, "r"); - if (fd == NULL ||

Re: List of Bitmapset (was Re: ExecRTCheckPerms() and many prunable partitions)

2022-11-15 Thread Amit Langote
On Wed, Nov 16, 2022 at 3:25 AM Tom Lane wrote: > Amit Langote writes: > > On Mon, Nov 14, 2022 at 11:57 PM Tom Lane wrote: > >> + * The new member is identified by the zero-based index of the List > >> + * element it should go into, and the bit number to be set therein. > > > The comment

Re: Documentation for building with meson

2022-11-15 Thread Justin Pryzby
On Wed, Nov 16, 2022 at 10:52:35AM +0900, Ian Lawrence Barwick wrote: > 2022年10月20日(木) 11:43 Justin Pryzby : > > > > On Wed, Oct 19, 2022 at 11:35:10AM -0700, samay sharma wrote: > > > Creating a new thread focussed on adding docs for building Postgres with > > > meson. This is a spinoff from the

Re: closing file in adjust_data_dir

2022-11-15 Thread Japin Li
On Wed, 16 Nov 2022 at 10:02, Japin Li wrote: > I think we should check whether fd is NULL or not, otherwise, segmentation > fault maybe occur. > > + if (pclose(fd) != 0) > + { > + write_stderr(_("%s: could not close the file following command > \"%s\"\n"), progname, cmd);

Re: closing file in adjust_data_dir

2022-11-15 Thread Ted Yu
On Tue, Nov 15, 2022 at 6:02 PM Japin Li wrote: > > On Wed, 16 Nov 2022 at 02:43, Ted Yu wrote: > > Hi, > > I was looking at the commit: > > > > commit 2fe3bdbd691a5d11626308e7d660440be6c210c8 > > Author: Peter Eisentraut > > Date: Tue Nov 15 15:35:37 2022 +0100 > > > > Check return

Re: hash_xlog_split_allocate_page: failed to acquire cleanup lock

2022-11-15 Thread Amit Kapila
On Mon, Nov 14, 2022 at 11:18 PM Andres Freund wrote: > > On 2022-11-14 16:06:27 +0530, Amit Kapila wrote: > > Pushed. > > Thanks. > Please find the attached patch to remove the buffer cleanup check on the new bucket page. I think we should do this only for the HEAD. Do you have any suggestions

Re: closing file in adjust_data_dir

2022-11-15 Thread Japin Li
On Wed, 16 Nov 2022 at 02:43, Ted Yu wrote: > Hi, > I was looking at the commit: > > commit 2fe3bdbd691a5d11626308e7d660440be6c210c8 > Author: Peter Eisentraut > Date: Tue Nov 15 15:35:37 2022 +0100 > > Check return value of pclose() correctly > > In src/bin/pg_ctl/pg_ctl.c : > > if

Re: Assertion failure in SnapBuildInitialSnapshot()

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 16:20:00 +0530, Amit Kapila wrote: > On Tue, Nov 15, 2022 at 8:08 AM Andres Freund wrote: > > nor do we enforce in an obvious place that we > > don't already hold a snapshot. > > > > We have a check for (FirstXactSnapshot == NULL) in >

Re: Documentation for building with meson

2022-11-15 Thread Ian Lawrence Barwick
2022年10月20日(木) 11:43 Justin Pryzby : > > On Wed, Oct 19, 2022 at 11:35:10AM -0700, samay sharma wrote: > > Creating a new thread focussed on adding docs for building Postgres with > > meson. This is a spinoff from the original thread [1] and I've attempted to > > address all the feedback provided

Re: ssl tests aren't concurrency safe due to get_free_port()

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 15:56:37 -0500, Andrew Dunstan wrote: > On 2022-11-06 Su 11:30, Andrew Dunstan wrote: > > > > One possible addition would be to add removing the reservation files in > > an END handler. That would be pretty simple. > > > > > > > Here's a version with that. I suggest we try it

Re: [DOCS] Stats views and functions not in order?

2022-11-15 Thread Peter Smith
On Thu, Nov 10, 2022 at 10:04 AM David G. Johnston wrote: > ... >> > So ... how do we proceed? >> > >> >> To proceed with the existing patches I need some guidance on exactly >> which of the changes can be considered improvements versus which ones >> are maybe just trading one 'random' order for

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 13:54:24 -0800, Peter Geoghegan wrote: > On Tue, Nov 15, 2022 at 12:29 PM Andres Freund wrote: > > ... I strongly dislike latestCommittedXid. That seems at least as misleading > > as latestRemovedXid and has the danger of confusion with latestCompletedXid > > as you mention. >

Re: Add test module for Custom WAL Resource Manager feature

2022-11-15 Thread Michael Paquier
On Wed, Nov 16, 2022 at 10:26:32AM +0900, Michael Paquier wrote: > Not many buildfarm members test 32b builds, but lapwing does. Well, it didn't take long: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=lapwing=2022-11-16%2000%3A40%3A11 -- Michael signature.asc Description: PGP

Re: Add test module for Custom WAL Resource Manager feature

2022-11-15 Thread Michael Paquier
On Tue, Nov 15, 2022 at 04:29:08PM -0800, Jeff Davis wrote: > Committed with some significant revisions (ae168c794f): > > * changed to insert a deterministic message, rather than a random > one, which allows more complete testing > * fixed a couple bugs > * used a static initializer for the

Re: Slow standby snapshot

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 19:46:26 -0500, Tom Lane wrote: > Andres Freund writes: > > To me it sounds like known_assigned_xids_lck is pointless and the talk about > > memory barriers a red herring, since all modifications have to happen with > > ProcArrayLock held exlusively and all reads with

Re: pg_basebackup's --gzip switch misbehaves

2022-11-15 Thread Michael Paquier
On Tue, Nov 15, 2022 at 11:09:54AM +0100, Daniel Gustafsson wrote: >> On 15 Nov 2022, at 00:58, Michael Paquier wrote: >> >> On Mon, Nov 14, 2022 at 03:27:14PM +0100, Daniel Gustafsson wrote: >>> Ugh, yes, that's what it should say. >> >> A split sounds fine by me. On top of what Tom has

Re: Unit tests for SLRU

2022-11-15 Thread Michael Paquier
On Tue, Nov 15, 2022 at 02:43:06PM +0400, Pavel Borisov wrote: > I've looked through the patch again. I agree it looks better and can > be committed. > Mark it as RfC now. Okay, applied, then. The SQL function names speak by themselves, even if some of them refer to pages but they act on

Re: Slow standby snapshot

2022-11-15 Thread Tom Lane
Andres Freund writes: > To me it sounds like known_assigned_xids_lck is pointless and the talk about > memory barriers a red herring, since all modifications have to happen with > ProcArrayLock held exlusively and all reads with ProcArrayLock held in share > mode. It can't be legal to modify

Re: Slow standby snapshot

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 19:15:15 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2022-11-15 23:14:42 +, Simon Riggs wrote: > >> Hence more frequent compression is effective at reducing the overhead. > >> But too frequent compression slows down the startup process, which > >> can't then keep

Re: Slow standby snapshot

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 23:14:42 +, Simon Riggs wrote: > On Tue, 15 Nov 2022 at 23:06, Tom Lane wrote: > > > > BTW, while nosing around this code I came across this statement > > (procarray.c, about line 4550 in HEAD): > > > > * ... To add XIDs to the array, we just insert > > * them into slots

Re: Slow standby snapshot

2022-11-15 Thread Tom Lane
Simon Riggs writes: > but that is not related to the main issues: > * COMMITs: xids are removed from the array by performing a binary > search - this gets more and more expensive as the array gets wider > * SNAPSHOTs: scanning the array for snapshots becomes more expensive > as the array gets

Re: Add test module for Custom WAL Resource Manager feature

2022-11-15 Thread Jeff Davis
On Mon, 2022-11-14 at 09:34 +0530, Bharath Rupireddy wrote: > Thanks. I would like to keep it simple. > > I've added some more comments and attached v2 patch herewith. Please > review. Committed with some significant revisions (ae168c794f): * changed to insert a deterministic message, rather

Re: allowing for control over SET ROLE

2022-11-15 Thread Jeff Davis
On Tue, 2022-11-15 at 12:07 -0500, Robert Haas wrote: > If anyone else wants to comment, or if either of those people want to > comment further, please speak up soon. Did you have some thoughts on: https://postgr.es/m/a41d606d03b629c2ef0ed274ae3b04a2c266.ca...@j-davis.com Regards,

Re: Unit tests for SLRU

2022-11-15 Thread Michael Paquier
On Tue, Nov 15, 2022 at 11:39:20AM +0100, Daniel Gustafsson wrote: > + /* write given data to the page */ > + strncpy(TestSlruCtl->shared->page_buffer[slotno], data, BLCKSZ - 1); > > Would it make sense to instead use pg_pwrite to closer match the code being > tested? Hmm. I am not

Re: Slow standby snapshot

2022-11-15 Thread Tom Lane
Andres Freund writes: > On 2022-11-15 23:14:42 +, Simon Riggs wrote: >> Hence more frequent compression is effective at reducing the overhead. >> But too frequent compression slows down the startup process, which >> can't then keep up. >> So we're just looking for an optimal frequency of

Re: Meson doesn't define HAVE_LOCALE_T for mscv

2022-11-15 Thread Juan José Santamaría Flecha
On Tue, Nov 15, 2022 at 9:02 PM Andres Freund wrote: > > On 2022-11-15 15:35:31 +0100, Juan José Santamaría Flecha wrote: > > On Tue, Nov 15, 2022 at 1:49 AM Andres Freund > wrote: > > > Hm. Is it right that the changes are only done for msvc? win32_port.h > > > defines the types for mingw as

Re: SUBTRANS: Minimizing calls to SubTransSetParent()

2022-11-15 Thread Tom Lane
Simon Riggs writes: > On Tue, 15 Nov 2022 at 21:03, Tom Lane wrote: >> The subxidStatus.overflowed check quoted above has a similar sort >> of myopia: it's checking whether our current transaction has >> already suboverflowed. But (a) that doesn't prove it won't suboverflow >> later, and (b)

Re: Slow standby snapshot

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 23:14:42 +, Simon Riggs wrote: > * COMMITs: xids are removed from the array by performing a binary > search - this gets more and more expensive as the array gets wider > * SNAPSHOTs: scanning the array for snapshots becomes more expensive > as the array gets wider > > Hence

Re: Meson doesn't define HAVE_LOCALE_T for mscv

2022-11-15 Thread Juan José Santamaría Flecha
On Tue, Nov 15, 2022 at 8:53 PM Andres Freund wrote: > > I don't think we should print mingw - that's really just redundant with > gcc. But including host_system seems like a good idea. Not sure why I > didn't > do that. > > I'll open a new thread for this. Also, I think this is skipping

Re: Meson add host_system to PG_VERSION_STR

2022-11-15 Thread Michael Paquier
On Wed, Nov 16, 2022 at 12:08:56AM +0100, Juan José Santamaría Flecha wrote: > As mentioned here [1] it might be interesting to complete the returned > information by version() when compiled with meson by including the > host_system. The meson build provides extra_version, which would be able to

Re: meson oddities

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 16:08:35 -0500, Tom Lane wrote: > Andrew Dunstan writes: > > On 2022-11-15 Tu 14:04, Andres Freund wrote: > >> I don't think it's a virtue to break the layout of the platform by > >> e.g. installing 64bit libs into the directory containing 32bit libs. > > > You might end up

Bug in row_number() optimization

2022-11-15 Thread Sergey Shinderuk
Hello, We've accidentally found a subtle bug introduced by commit 9d9c02ccd1aea8e9131d8f4edb21bf1687e40782 Author: David Rowley Date: Fri Apr 8 10:34:36 2022 +1200 Teach planner and executor about monotonic window funcs On a 32-bit system Valgrind reports use-after-free when running

Re: Slow standby snapshot

2022-11-15 Thread Simon Riggs
On Tue, 15 Nov 2022 at 23:06, Tom Lane wrote: > > BTW, while nosing around this code I came across this statement > (procarray.c, about line 4550 in HEAD): > > * ... To add XIDs to the array, we just insert > * them into slots to the right of the head pointer and then advance the head > *

Re: Add palloc_aligned() to allow arbitrary power of 2 memory alignment

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 16:58:10 -0500, Greg Stark wrote: > So I think it's kind of cute that you've implemented these as agnostic > wrappers that work with any allocator ... but why? > > I would have expected the functionality to just be added directly to > the allocator to explicitly request whole

Meson add host_system to PG_VERSION_STR

2022-11-15 Thread Juan José Santamaría Flecha
Hello all, As mentioned here [1] it might be interesting to complete the returned information by version() when compiled with meson by including the host_system. [1] https://www.postgresql.org/message-id/20221115195318.5v5ynapmkusgyzks%40awork3.anarazel.de Regards, Juan José Santamaría Flecha

Re: Slow standby snapshot

2022-11-15 Thread Tom Lane
BTW, while nosing around this code I came across this statement (procarray.c, about line 4550 in HEAD): * ... To add XIDs to the array, we just insert * them into slots to the right of the head pointer and then advance the head * pointer. This wouldn't require any lock at all, except that on

Re: SUBTRANS: Minimizing calls to SubTransSetParent()

2022-11-15 Thread Simon Riggs
On Tue, 15 Nov 2022 at 21:03, Tom Lane wrote: > > Simon Riggs writes: > > New version attached. > > I looked at this patch and I do not see how it can possibly be safe. I grant you it is complex, so please bear with me. > The most direct counterexample arises from the fact that >

Re: Slow standby snapshot

2022-11-15 Thread Tom Lane
Simon Riggs writes: > I've cleaned up the comments and used a #define for the constant. > IMHO this is ready for commit. I will add it to the next CF. I looked at this a little. It's a simple enough patch, and if it solves the problem then I sure like it better than the previous ideas in this

Re: closing file in adjust_data_dir

2022-11-15 Thread Ted Yu
On Tue, Nov 15, 2022 at 10:43 AM Ted Yu wrote: > Hi, > I was looking at the commit: > > commit 2fe3bdbd691a5d11626308e7d660440be6c210c8 > Author: Peter Eisentraut > Date: Tue Nov 15 15:35:37 2022 +0100 > > Check return value of pclose() correctly > > In src/bin/pg_ctl/pg_ctl.c : > >

Re: Add palloc_aligned() to allow arbitrary power of 2 memory alignment

2022-11-15 Thread Greg Stark
So I think it's kind of cute that you've implemented these as agnostic wrappers that work with any allocator ... but why? I would have expected the functionality to just be added directly to the allocator to explicitly request whole aligned pages which IIRC it's already capable of doing but just

Re: Avoid overhead open-close indexes (catalog updates)

2022-11-15 Thread Michael Paquier
On Tue, Nov 15, 2022 at 11:42:34AM -0300, Ranier Vilela wrote: > I find it very difficult not to have some tuple to be updated, > once inside CopyStatistics and the branch cost can get in the way, > but I don't object with your solution. The code assumes that it is a possibility. > Missed

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Peter Geoghegan
On Tue, Nov 15, 2022 at 12:29 PM Andres Freund wrote: > ... I strongly dislike latestCommittedXid. That seems at least as misleading > as latestRemovedXid and has the danger of confusion with latestCompletedXid > as you mention. > How about latestAffectedXid? I get why you don't care for

Re: archive modules

2022-11-15 Thread Michael Paquier
On Tue, Nov 15, 2022 at 12:57:49PM -0800, Nathan Bossart wrote: > On Tue, Nov 15, 2022 at 06:14:25PM +0100, Alvaro Herrera wrote: >> Hmm, really? It seems to me that we will have two slightly different >> behaviors in 15 and master, which may be confusing later on. I think >> it'd be better to

Re: allowing for control over SET ROLE

2022-11-15 Thread Nathan Bossart
On Tue, Nov 15, 2022 at 12:07:06PM -0500, Robert Haas wrote: > If anyone else wants to comment, or if either of those people want to > comment further, please speak up soon. Otherwise, I am going to press > forward with committing this. I don't think I have any further thoughts about the

Re: meson oddities

2022-11-15 Thread Tom Lane
Andrew Dunstan writes: > On 2022-11-15 Tu 14:04, Andres Freund wrote: >> I don't think it's a virtue to break the layout of the platform by >> e.g. installing 64bit libs into the directory containing 32bit libs. > You might end up surprising people who have installed from source for > years and

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2022-11-15 Thread David Rowley
Thank you for looking at the patch. On Fri, 4 Nov 2022 at 04:43, Ankit Kumar Pandey wrote: > I don't see any performance improvement in tests. Are you able to share what your test was? In order to see a performance improvement you're likely going to have to obtain a large number of locks in

Re: SUBTRANS: Minimizing calls to SubTransSetParent()

2022-11-15 Thread Tom Lane
Simon Riggs writes: > New version attached. I looked at this patch and I do not see how it can possibly be safe. The most direct counterexample arises from the fact that HeapCheckForSerializableConflictOut checks SubTransGetTopmostTransaction in some cases. You haven't tried to analyze when,

Re: archive modules

2022-11-15 Thread Nathan Bossart
On Tue, Nov 15, 2022 at 06:14:25PM +0100, Alvaro Herrera wrote: > On 2022-Nov-15, Nathan Bossart wrote: > >> On Tue, Nov 15, 2022 at 10:31:44AM +0100, Peter Eisentraut wrote: > >> > The surrounding code has changed a bit between PG15 and master, so if we >> > wanted to backpatch this, we'd need

Re: ssl tests aren't concurrency safe due to get_free_port()

2022-11-15 Thread Andrew Dunstan
On 2022-11-06 Su 11:30, Andrew Dunstan wrote: > > One possible addition would be to add removing the reservation files in > an END handler. That would be pretty simple. > > Here's a version with that. I suggest we try it out and see if anything breaks. cheers andrew -- Andrew Dunstan EDB:

Re: meson oddities

2022-11-15 Thread Andrew Dunstan
On 2022-11-15 Tu 14:04, Andres Freund wrote: >> But ISTM we shouldn't be presuming what packagers will do, and that >> there is some virtue in having a default layout under ${prefix} that is >> consistent across platforms, as is now the case with autoconf/configure. > I don't think it's a virtue

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-15 Thread Andres Freund
Hi, I like the idea of this, but: On 2022-11-15 10:24:05 -0800, Peter Geoghegan wrote: > I'm not necessarily that attached to the name latestCommittedXid. It > is more accurate, but it's also a little bit too similar to another > common XID symbol name, latestCompletedXid. Can anyone suggest an

Re: HOT chain validation in verify_heapam()

2022-11-15 Thread Robert Haas
On Tue, Nov 15, 2022 at 2:50 PM Andres Freund wrote: > On 2022-11-15 11:36:21 -0500, Robert Haas wrote: > > On Mon, Nov 14, 2022 at 5:02 PM Andres Freund wrote: > > > It seems like we should do a bit more validation within a chain of > > > tuples. E.g. that no live tuple can follow an !DidCommit

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-11-15 Thread Andres Freund
Hi, On 2022-11-04 09:25:52 +0100, Drouvot, Bertrand wrote: > > @@ -7023,29 +7048,63 @@ static void > CheckPointGuts(XLogRecPtr checkPointRedo, int flags) > { > CheckPointRelationMap(); > + > + pgstat_progress_update_param(PROGRESS_CHECKPOINT_PHASE, > +

Re: Moving forward with TDE

2022-11-15 Thread Jacob Champion
On Tue, Nov 15, 2022 at 11:39 AM David Christensen wrote: > Good to know about the next steps, thanks. You're welcome! > This was just a refresh of the old patches on the wiki to work as written on > HEAD. If there are known TODOs here this then that work is still needing to > be done. > > I

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-11-15 Thread Robert Haas
On Fri, Nov 4, 2022 at 4:27 AM Drouvot, Bertrand wrote: > Please find attached a rebase in v7. I don't think it's a good thing that this patch is using the progress-reporting machinery. The point of that machinery is that we want any backend to be able to report progress for any command it

Re: Meson doesn't define HAVE_LOCALE_T for mscv

2022-11-15 Thread Andres Freund
Hi, Hm, the quoting was odd, making me think you had written a separate email about the define issue. Hence the separate email... On 2022-11-15 15:35:31 +0100, Juan José Santamaría Flecha wrote: > On Tue, Nov 15, 2022 at 1:49 AM Andres Freund wrote: > > Hm. Is it right that the changes are

Re: Meson doesn't define HAVE_LOCALE_T for mscv

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 15:35:31 +0100, Juan José Santamaría Flecha wrote: > I've seen that when building with meson on MinGW the output for version() > is 'PostgreSQL 16devel on x86_64, compiled by gcc-12.2.0', which is not > wrong but I cannot tell that it was done on MinGW. Should we include the >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-15 Thread David Christensen
Enclosed is v9. - code style consistency (FPI instead of FPW) internally. - cleanup of no-longer needed checksum-related pieces from code and tests. - test cleanup/simplification. - other comment cleanup. Passes all CI checks. Best, David

Re: HOT chain validation in verify_heapam()

2022-11-15 Thread Andres Freund
Hi, On 2022-11-15 11:36:21 -0500, Robert Haas wrote: > On Mon, Nov 14, 2022 at 5:02 PM Andres Freund wrote: > > It seems like we should do a bit more validation within a chain of > > tuples. E.g. that no live tuple can follow an !DidCommit xmin? > > I think this check is already present in

Re: Moving forward with TDE

2022-11-15 Thread David Christensen
> On Nov 15, 2022, at 1:08 PM, Jacob Champion wrote: > > On Mon, Oct 24, 2022 at 9:29 AM David Christensen > wrote: >> I would love to open a discussion about how to move forward and get >> some of these features built out. The historical threads here are >> quite long and complicated; is

  1   2   >