Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-17 Thread John Naylor
On Wed, Sep 28, 2022 at 1:18 PM I wrote: > Along those lines, one thing I've been thinking about is the number of size classes. There is a tradeoff between memory efficiency and number of branches when searching/inserting. My current thinking is there is too much coupling between size class and

Re: contrib: auth_delay module

2022-11-17 Thread Julien Rouhaud
On Thu, Nov 17, 2022 at 05:37:51PM -0500, Tom Lane wrote: > =?UTF-8?B?5oiQ5LmL54SV?= writes: > > The attached patch is a contrib module to set login restrictions on users > > with > > too many authentication failure. The administrator could manage several GUC > > parameters to control the login

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-17 Thread Peter Smith
Here are some review comments for v47-0001 (This review is a WIP - I will post more comments for this patch next week) == .../replication/logical/applyparallelworker.c 1. + * Copyright (c) 2022, PostgreSQL Global Development Group + * + * IDENTIFICATION

Re: Avoid double lookup in pgstat_fetch_stat_tabentry()

2022-11-17 Thread Bharath Rupireddy
On Fri, Nov 18, 2022 at 10:32 AM Drouvot, Bertrand wrote: > > Hi hackers, > > Please find attached a patch proposal to avoid 2 calls to > pgstat_fetch_stat_tabentry_ext() in pgstat_fetch_stat_tabentry() in case > the relation is not a shared one and no statistics are found. > > Thanks Andres for

Re: Assertion failure in SnapBuildInitialSnapshot()

2022-11-17 Thread Amit Kapila
On Thu, Nov 17, 2022 at 11:15 PM Andres Freund wrote: > > On 2022-11-17 10:44:18 +0530, Amit Kapila wrote: > > On Wed, Nov 16, 2022 at 11:56 PM Andres Freund wrote: > > > On 2022-11-16 14:22:01 +0530, Amit Kapila wrote: > > > > On Wed, Nov 16, 2022 at 7:30 AM Andres Freund > > > > wrote: > > >

Re: Optimize join selectivity estimation by not reading MCV stats for unique join attributes

2022-11-17 Thread Tom Lane
Richard Guo writes: > Yes, it is. Using zero flag would short-cut get_attstatsslot() to just > return whether the slot type exists without loading it. Do you think we > need to emphasize this use case in the comments for 'flags'? Perhaps, it's not really obvious now. > I wonder whether we

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-17 Thread Amit Kapila
On Fri, Nov 18, 2022 at 10:31 AM Masahiko Sawada wrote: > > On Fri, Nov 18, 2022 at 1:47 PM Amit Kapila wrote: > > > > On Fri, Nov 18, 2022 at 8:01 AM Peter Smith wrote: > > > > > > On Fri, Nov 18, 2022 at 11:36 AM Masahiko Sawada > > > wrote: > > > > > > > ... > > > > --- > > > > The

Avoid double lookup in pgstat_fetch_stat_tabentry()

2022-11-17 Thread Drouvot, Bertrand
Hi hackers, Please find attached a patch proposal to avoid 2 calls to pgstat_fetch_stat_tabentry_ext() in pgstat_fetch_stat_tabentry() in case the relation is not a shared one and no statistics are found. Thanks Andres for the suggestion done in [1]. [1]:

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-17 Thread Masahiko Sawada
On Fri, Nov 18, 2022 at 1:47 PM Amit Kapila wrote: > > On Fri, Nov 18, 2022 at 8:01 AM Peter Smith wrote: > > > > On Fri, Nov 18, 2022 at 11:36 AM Masahiko Sawada > > wrote: > > > > > ... > > > --- > > > The streaming parameter has the new value "parallel" for "streaming" > > > option to

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-17 Thread Amit Kapila
On Fri, Nov 18, 2022 at 8:01 AM Peter Smith wrote: > > On Fri, Nov 18, 2022 at 11:36 AM Masahiko Sawada > wrote: > > > ... > > --- > > The streaming parameter has the new value "parallel" for "streaming" > > option to enable the parallel apply. It fits so far but I think the > > parallel apply

Re: Optimize join selectivity estimation by not reading MCV stats for unique join attributes

2022-11-17 Thread Richard Guo
On Fri, Nov 18, 2022 at 9:36 AM Tom Lane wrote: > Actually, looking at get_attstatslot, I realize it was already designed > to do that -- just pass zero for flags. So we could do it as attached. Yes, it is. Using zero flag would short-cut get_attstatsslot() to just return whether the slot

Re: Strange failure on mamba

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 17:47:50 -0500, Tom Lane wrote: > Yeah, that or some other NetBSD bug could be the explanation, too. > Without a stack trace it's hard to have any confidence about it, > but I've been unable to reproduce the problem outside the buildfarm. > (Which is a familiar refrain. I

Re: logical decoding and replication of sequences, take 2

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 22:13:23 +0100, Tomas Vondra wrote: > On 11/17/22 18:07, Andres Freund wrote: > > On 2022-11-17 12:39:49 +0100, Tomas Vondra wrote: > >> On 11/17/22 03:43, Andres Freund wrote: > >>> I assume that part of the initial sync would have to be a new sequence > >>> synchronization

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-17 Thread Peter Smith
On Fri, Nov 18, 2022 at 11:36 AM Masahiko Sawada wrote: > ... > --- > The streaming parameter has the new value "parallel" for "streaming" > option to enable the parallel apply. It fits so far but I think the > parallel apply feature doesn't necessarily need to be tied up with > streaming

Re: Fix some newly modified tab-complete changes

2022-11-17 Thread Michael Paquier
On Wed, Nov 16, 2022 at 08:29:24AM +, shiy.f...@fujitsu.com wrote: > I have fixed the problems you saw, and improved the patch as you suggested. > > Besides, I noticed that the tab completion for "ALTER DEFAULT PRIVILEGES ... > GRANT/REVOKE ..." missed "CREATE". Fix it in 0001 patch. > > And

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-17 Thread Amit Kapila
On Wed, Nov 16, 2022 at 1:50 PM houzj.f...@fujitsu.com wrote: > > On Tuesday, November 15, 2022 7:58 PM houzj.f...@fujitsu.com > wrote: > > I noticed that I didn't add CHECK_FOR_INTERRUPTS while retrying send message. > So, attach the new version which adds that. Also attach the 0004 patch that

Re: Optimize join selectivity estimation by not reading MCV stats for unique join attributes

2022-11-17 Thread Tom Lane
I wrote: > Tomas Vondra writes: >> Or perhaps what if we have a function that quickly determines if the >> attribute has MCV, without loading it? I'd bet the expensive part of >> get_attstatslot() is the deconstruct_array(). >> We could have a function that only does the first small loop over

Re: Getting rid of SQLValueFunction

2022-11-17 Thread Michael Paquier
On Tue, Oct 25, 2022 at 02:20:12PM +0900, Michael Paquier wrote: > Attached is a rebased patch set, as of the conflicts from 2e0d80c. So, this patch set has been sitting in the CF app for a few weeks now, and I would like to apply them to remove a bit of code from the executor. Please note that

Re: libpq compression (part 2)

2022-11-17 Thread Andrey Borodin
On Thu, Nov 17, 2022 at 7:09 AM Peter Eisentraut wrote: > > Note that the above code was just changed in dce92e59b1. Thanks! > I don't know > how that affects this patch set. With dce92e59b1 it would be much easier to find a bug in the compression patch. Some more notes about the patch. (sorry

Re: Typo in SH_LOOKUP and SH_LOOKUP_HASH comments

2022-11-17 Thread vignesh C
On Thu, 17 Nov 2022 at 17:48, Daniel Gustafsson wrote: > > > On 17 Nov 2022, at 10:56, vignesh C wrote: > > > When I was reviewing one of the patches, I found a typo in the > > comments of SH_LOOKUP and SH_LOOKUP_HASH. I felt "lookup up" should > > have been "lookup". > > Attached a patch to

Re: Reducing power consumption on idle servers

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 13:06:23 +0530, Bharath Rupireddy wrote: > I understand. I know it's a bit hard to measure the power savings, I'm > wondering if there's any info, maybe not necessarily related to > postgres, but in general how much power gets saved if a certain number > of waits/polls/system

Re: Optimize join selectivity estimation by not reading MCV stats for unique join attributes

2022-11-17 Thread Tom Lane
Tomas Vondra writes: > Or perhaps what if we have a function that quickly determines if the > attribute has MCV, without loading it? I'd bet the expensive part of > get_attstatslot() is the deconstruct_array(). > We could have a function that only does the first small loop over slots, > and

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-17 Thread Masahiko Sawada
On Tue, Nov 15, 2022 at 8:57 PM houzj.f...@fujitsu.com wrote: > > On Saturday, November 12, 2022 7:06 PM Amit Kapila > > > > On Fri, Nov 11, 2022 at 2:12 PM houzj.f...@fujitsu.com > > wrote: > > > > > > > Few comments on v46-0001: > > == > > > > Thanks for the comments. > >

Re: Optimize join selectivity estimation by not reading MCV stats for unique join attributes

2022-11-17 Thread Tomas Vondra
On 11/14/22 10:19, David Geier wrote: > Hi Tom, >> There won't *be* any MCV stats for a column that ANALYZE perceives to >> be unique, so I'm not quite sure where the claimed savings comes from. > > We save if one join attribute is unique while the other isn't. In that > case stored MCV stats

Re: allowing for control over SET ROLE

2022-11-17 Thread Jeff Davis
On Thu, 2022-11-17 at 16:52 -0500, Robert Haas wrote: > But I think the bigger > reason is that, in my opinion, this proposal is more generally > useful, > because it takes no position on why you wish to disallow SET ROLE. > You > can just disallow it in some cases and allow it in others, and

Re: Mingw task for Cirrus CI

2022-11-17 Thread Andres Freund
Hi, On 2022-10-24 14:13:06 +0300, Melih Mutlu wrote: > If you say so, then I think it's ready. I pushed it with a few changes: - I added an only_if, otherwise the task shows up (without running) even if you use ci-os-only: linux - I added -Dnls=disabled - It seemed to add about 1.5min to the

Re: redundant check of msg in does_not_exist_skipping

2022-11-17 Thread Li Japin
> On Nov 18, 2022, at 4:04 AM, Robert Haas wrote: > > On Thu, Nov 17, 2022 at 10:56 AM Tom Lane wrote: >> This is a completely bad idea. If it takes that level of analysis >> to see that msg can't be null, we should leave the test in place. >> Any future modification of either this code or

Re: Don't treate IndexStmt like AlterTable when DefineIndex is called from ProcessUtilitySlow.

2022-11-17 Thread 正华吕
Hi, thanks a lot for your reply! > Why do you think this is an improvement? I hit the issue in Greenplum Database (Massively Parallel Postgres), the MPP architecture is that coordinator dispatch statement to segments. The dispatch logic is quit different for AlterTable and CreateTableLike: *

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-17 Thread Peter Geoghegan
On Thu, Nov 17, 2022 at 9:02 AM Peter Geoghegan wrote: > Plan is to commit this later on today, barring objections. Pushed, thanks. -- Peter Geoghegan

Re: Strange failure on mamba

2022-11-17 Thread Thomas Munro
On Fri, Nov 18, 2022 at 11:35 AM Thomas Munro wrote: > I wonder if it's a runtime variant of the other problem. We do > load_file("libpqwalreceiver", false) before unblocking signals but > maybe don't resolve the symbols until calling them, or something like > that... Hmm, no, I take that back.

Re: Strange failure on mamba

2022-11-17 Thread Tom Lane
Thomas Munro writes: > On Fri, Nov 18, 2022 at 11:08 AM Tom Lane wrote: >> mamba has been showing intermittent failures in various replication >> tests since day one. > I wonder if it's a runtime variant of the other problem. We do > load_file("libpqwalreceiver", false) before unblocking

Re: contrib: auth_delay module

2022-11-17 Thread Tom Lane
=?UTF-8?B?5oiQ5LmL54SV?= writes: > The attached patch is a contrib module to set login restrictions on users > with > too many authentication failure. The administrator could manage several GUC > parameters to control the login restrictions which are listed below. > - set the wait time when

Re: Strange failure on mamba

2022-11-17 Thread Thomas Munro
On Fri, Nov 18, 2022 at 11:08 AM Tom Lane wrote: > Thomas Munro writes: > > I wonder why the walreceiver didn't start in > > 008_min_recovery_point_node_3.log here: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mamba=2022-11-16%2023%3A13%3A38 > > mamba has been showing intermittent

Re: Don't treate IndexStmt like AlterTable when DefineIndex is called from ProcessUtilitySlow.

2022-11-17 Thread Tom Lane
=?UTF-8?B?5q2j5Y2O5ZCV?= writes: > Recently read the code, I find that when calling DefineIndex > from ProcessUtilitySlow, is_alter_table will be set true if > this statement is came from expandTableLikeClause. Yeah. > Based on the above, I think we can always a false value > for

Re: allow segment size to be set to < 1GiB

2022-11-17 Thread Andrew Dunstan
On 2022-11-17 Th 10:48, Tom Lane wrote: > Andres Freund writes: >> On 2022-11-17 09:58:48 -0500, Andrew Dunstan wrote: >>> Are we going to impose some sane minimum, or leave it up to developers >>> to discover that for themselves? >> I don't think we should. It's actually useful to e.g. use 1

Re: Fix proposal for comparaison bugs in PostgreSQL::Version

2022-11-17 Thread Andrew Dunstan
On 2022-11-04 Fr 10:06, Jehan-Guillaume de Rorthais wrote: > On Thu, 3 Nov 2022 13:11:18 -0500 > Justin Pryzby wrote: > >> On Tue, Jun 28, 2022 at 06:17:40PM -0400, Andrew Dunstan wrote: >>> Nice catch, but this looks like massive overkill. I think we can very >>> simply fix the test in just a

Re: Strange failure on mamba

2022-11-17 Thread Tom Lane
Thomas Munro writes: > I wonder why the walreceiver didn't start in > 008_min_recovery_point_node_3.log here: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mamba=2022-11-16%2023%3A13%3A38 mamba has been showing intermittent failures in various replication tests since day one. My

Patch: Global Unique Index

2022-11-17 Thread Cary Huang
Patch: Global Unique Index “Global unique index” in our definition is a unique index on a partitioned table that can ensure cross-partition uniqueness using a non-partition key. This work is inspired by this email thread, “Proposal: Global Index” started back in 2019 (Link below). My

Strange failure on mamba

2022-11-17 Thread Thomas Munro
Hi, I wonder why the walreceiver didn't start in 008_min_recovery_point_node_3.log here: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mamba=2022-11-16%2023%3A13%3A38 There was the case of commit 8acd8f86, but that involved a deadlocked postmaster whereas this one still handled a

Re: allowing for control over SET ROLE

2022-11-17 Thread Robert Haas
On Tue, Nov 15, 2022 at 7:23 PM Jeff Davis wrote: > On Tue, 2022-11-15 at 12:07 -0500, Robert Haas wrote: > > If anyone else wants to comment, or if either of those people want to > > comment further, please speak up soon. > > Did you have some thoughts on: > >

Outdated url to Hunspell

2022-11-17 Thread Daniel Gustafsson
I happened to notice that the link to Hunspell in our documentation goes to the hunspell sourceforge page last updated in 2015. The project has since moved to Github [0] with hunspell.sourceforge.net redirecting there, I'm not sure exactly when but Wikipedia updated their link entry in 2016 [1]

Re: ubsan fails on 32bit builds

2022-11-17 Thread Thomas Munro
On Fri, Nov 18, 2022 at 9:13 AM Andres Freund wrote: > Agreed. I had started on a set of patches for some of the SHM_QUEUE uses, but > somehow we ended up a bit stuck on the naming of dlist_delete variant that > afterwards zeroes next/prev so we can replace SHMQueueIsDetached() uses. > > Should

Re: logical decoding and replication of sequences, take 2

2022-11-17 Thread Tomas Vondra
On 11/17/22 18:07, Andres Freund wrote: > Hi, > > On 2022-11-17 12:39:49 +0100, Tomas Vondra wrote: >> On 11/17/22 03:43, Andres Freund wrote: >>> On 2022-11-17 02:41:14 +0100, Tomas Vondra wrote: Well, yeah - we can either try to perform the stuff independently of the transactions

Re: Reducing power consumption on idle servers

2022-11-17 Thread Robert Haas
On Thu, Nov 17, 2022 at 2:55 AM Simon Riggs wrote: > No, it will have a direct effect only on people using promote_trigger_file > who do not read and act upon the deprecation notice before upgrading > by making a one line change to their failover scripts. TBH, I wonder if we shouldn't just nuke

Re: SUBTRANS: Minimizing calls to SubTransSetParent()

2022-11-17 Thread Tomas Vondra
On 11/17/22 18:29, Simon Riggs wrote: > On Thu, 17 Nov 2022 at 17:04, Simon Riggs > wrote: >> >> New version with greatly improved comments coming very soon. > >>> Perhaps it would be a good idea to split up the patch. The business >>> about making pg_subtrans flat rather than a tree seems

Re: ubsan fails on 32bit builds

2022-11-17 Thread Tom Lane
Andres Freund writes: > On 2022-11-17 14:20:47 -0500, Robert Haas wrote: >> Not that I object to a targeted fix > Should we backpatch this fix? Likely this doesn't cause active breakage > outside of 32bit builds under ubsan, but that's not an unreasonable thing to > want to do in the

Re: ubsan fails on 32bit builds

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 14:20:47 -0500, Robert Haas wrote: > On Wed, Nov 16, 2022 at 8:42 PM Andres Freund wrote: > > Afaict the problem is that > > proc = (PGPROC *) &(waitQueue->links); > > > > is a gross gross hack - this isn't actually a PGPROC, it's pointing to an > > SHM_QUEUE,

Re: Allow single table VACUUM in transaction block

2022-11-17 Thread Robert Haas
On Wed, Nov 16, 2022 at 5:14 PM Greg Stark wrote: > However I'm not a fan of commands that sometimes do one thing and > sometimes magically do something very different. I don't like the idea > that the same vacuum command would sometimes run in-process and > sometimes do this out of process

Re: redundant check of msg in does_not_exist_skipping

2022-11-17 Thread Robert Haas
On Thu, Nov 17, 2022 at 10:56 AM Tom Lane wrote: > This is a completely bad idea. If it takes that level of analysis > to see that msg can't be null, we should leave the test in place. > Any future modification of either this code or what it calls could > break the conclusion. +1. Also, even if

Re: Allow single table VACUUM in transaction block

2022-11-17 Thread Justin Pryzby
On Wed, Nov 16, 2022 at 05:14:07PM -0500, Greg Stark wrote: > I think this requesting autovacuum worker should be a distinct > command. Or at least an explicit option to vacuum. +1. I was going to suggest VACUUM (NOWAIT) .. -- Justin

Re: [PoC] configurable out of disk space elog level

2022-11-17 Thread Robert Haas
On Wed, Nov 16, 2022 at 7:59 AM Maxim Orlov wrote: > The customer, mentioned above, in this particular case, would be glad to be > able to have a mechanism to stop the cluster. > Again, in this concrete case. > > My proposal is to add a tablespace option in order to be able to configure > which

Re: ubsan fails on 32bit builds

2022-11-17 Thread Robert Haas
On Wed, Nov 16, 2022 at 8:42 PM Andres Freund wrote: > Afaict the problem is that > proc = (PGPROC *) &(waitQueue->links); > > is a gross gross hack - this isn't actually a PGPROC, it's pointing to an > SHM_QUEUE, but *not* one embedded in PGPROC. It kinda works because ->links >

Re: when the startup process doesn't (logging startup delays)

2022-11-17 Thread Robert Haas
On Thu, Nov 17, 2022 at 2:22 AM Bharath Rupireddy wrote: > Duplication is a problem that I agree with and I have an idea here - > how about introducing a new function, say EnableStandbyMode() that > sets StandbyMode to true and disables the startup progress timeout, > something like the attached?

Re: TAP output format in pg_regress

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 11:36:13 +0100, Daniel Gustafsson wrote: > > On 10 Nov 2022, at 11:44, Nikolay Shaplov wrote: > > Did not found quick way to include prove TAP harness right into Makefile, > > so I > > check dumped output, but it is not really important for now, I guess. > > I think we'll

Re: HOT chain validation in verify_heapam()

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 21:33:17 +0530, Himanshu Upadhyaya wrote: > On Tue, Nov 15, 2022 at 3:32 AM Andres Freund wrote: > > > Furthermore, it is > > > possible that successor[x] = successor[x'] since the page might be > > corrupted > > > and we haven't checked otherwise. > > > > > > predecessor[y] =

Re: CREATE UNLOGGED TABLE seq faults when debug_discard_caches=1

2022-11-17 Thread Tom Lane
I wrote: > I wonder whether we ought to back-patch f10f0ae42. We could > leave the RelationOpenSmgr macro in existence to avoid unnecessary > breakage of extension code, but stop using it within our own code. Concretely, about like this for v14 (didn't look at the older branches yet). I'm not

Re: Assertion failure in SnapBuildInitialSnapshot()

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 10:44:18 +0530, Amit Kapila wrote: > On Wed, Nov 16, 2022 at 11:56 PM Andres Freund wrote: > > On 2022-11-16 14:22:01 +0530, Amit Kapila wrote: > > > On Wed, Nov 16, 2022 at 7:30 AM Andres Freund wrote: > > > > On 2022-11-15 16:20:00 +0530, Amit Kapila wrote: > > > > > On Tue,

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-11-17 Thread Robert Haas
On Thu, Nov 17, 2022 at 11:24 AM Andres Freund wrote: > As an experiment, I added a progress report to BufferSync()'s first loop > (i.e. where it checks all buffers). On a 128GB shared_buffers cluster that > increases the time for a do-nothing checkpoint from ~235ms to ~280ms. If I > remove the

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-11-17 Thread Tom Lane
Andres Freund writes: > I think pgstat_progress_start_command() needs the changecount stuff, as does > pgstat_progress_update_multi_param(). But for anything updating a single > parameter at a time it really doesn't do anything useful on a platform that > doesn't tear 64bit writes (so it could be

Re: logical decoding and replication of sequences, take 2

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 12:39:49 +0100, Tomas Vondra wrote: > On 11/17/22 03:43, Andres Freund wrote: > > On 2022-11-17 02:41:14 +0100, Tomas Vondra wrote: > >> Well, yeah - we can either try to perform the stuff independently of the > >> transactions that triggered it, or we can try making it part of

Re: SUBTRANS: Minimizing calls to SubTransSetParent()

2022-11-17 Thread Simon Riggs
On Wed, 16 Nov 2022 at 15:44, Tom Lane wrote: > > Simon Riggs writes: > > On Wed, 16 Nov 2022 at 00:09, Tom Lane wrote: > >> No, that's not what XidInMVCCSnapshot does. If snapshot->suboverflowed > >> is set (ie, somebody somewhere/somewhen overflowed), then it does > >>

Re: [BUG] pg_dump blocked

2022-11-17 Thread Gilles Darold
Le 17/11/2022 à 17:59, Tom Lane a écrit : Gilles Darold writes: I have an incorrect behavior with pg_dump prior PG version 15. With PostgreSQL 15, thanks to commit e3fcbbd623b9ccc16cdbda374654d91a4727d173 the problem is gone but for older versions it persists with locks on partitioned tables.

Re: Standardizing how pg_waldump presents recovery conflict XID cutoffs

2022-11-17 Thread Peter Geoghegan
On Wed, Nov 16, 2022 at 4:34 PM Peter Geoghegan wrote: > WFM. Attached is v3. Plan is to commit this later on today, barring objections. Thanks -- Peter Geoghegan v3-0001-Standardize-rmgrdesc-recovery-conflict-XID-output.patch Description: Binary data

Re: [BUG] pg_dump blocked

2022-11-17 Thread Tom Lane
Gilles Darold writes: > I have an incorrect behavior with pg_dump prior PG version 15. With > PostgreSQL 15, thanks to commit e3fcbbd623b9ccc16cdbda374654d91a4727d173 > the problem is gone but for older versions it persists with locks on > partitioned tables. I didn't want to back-patch

[BUG] pg_dump blocked

2022-11-17 Thread Gilles Darold
Hi, I have an incorrect behavior with pg_dump prior PG version 15. With PostgreSQL 15, thanks to commit e3fcbbd623b9ccc16cdbda374654d91a4727d173 the problem is gone but for older versions it persists with locks on partitioned tables. When we try to dump a database where a table is locked,

Re: Moving forward with TDE

2022-11-17 Thread David Christensen
Hi Dilip, Thanks for the feedback here. I will review the docs changes and add to my tree. Best, David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-17 Thread David Christensen
On Wed, Nov 16, 2022 at 3:30 AM Bharath Rupireddy wrote: > 1. > -if (config.filter_by_fpw && !XLogRecordHasFPW(xlogreader_state)) > +if (config.filter_by_fpw && !XLogRecordHasFPI(xlogreader_state)) > These changes are not related to this feature, hence renaming those >

Re: allow segment size to be set to < 1GiB

2022-11-17 Thread Andres Freund
On 2022-11-17 10:48:52 -0500, Tom Lane wrote: > Yeah, I think we should allow setting it to 1 block. This switch is > only for testing purposes (I hope the docs make that clear). "This option is only for developers, to test segment related code."

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 09:03:32 -0500, Robert Haas wrote: > On Wed, Nov 16, 2022 at 2:52 PM Andres Freund wrote: > > I don't think we'd want every buffer write or whatnot go through the > > changecount mechanism, on some non-x86 platforms that could be noticable. > > But > > if we didn't stage the

Re: CREATE UNLOGGED TABLE seq faults when debug_discard_caches=1

2022-11-17 Thread Tom Lane
Spyridon Dimitrios Agathos writes: > while testing the developer settings of PSQL (14.5) I came across this > issue: > postgres=# CREATE UNLOGGED TABLE stats ( > postgres(# pg_hash BIGINT NOT NULL, > postgres(# category TEXT NOT NULL, > postgres(# PRIMARY KEY (pg_hash, category) >

Re: logical decoding and replication of sequences, take 2

2022-11-17 Thread Robert Haas
On Wed, Nov 16, 2022 at 8:41 PM Tomas Vondra wrote: > There's a couple of caveats, though: > > 1) Maybe we should focus more on "actually observed" state instead of > "observable". Who cares if the sequence moved forward in a transaction > that was ultimately rolled back? No committed transaction

Re: HOT chain validation in verify_heapam()

2022-11-17 Thread Himanshu Upadhyaya
On Tue, Nov 15, 2022 at 3:32 AM Andres Freund wrote: > > > Furthermore, it is > > possible that successor[x] = successor[x'] since the page might be > corrupted > > and we haven't checked otherwise. > > > > predecessor[y] = x means that successor[x] = y but in addition we've > > checked that y

Re: Moving forward with TDE

2022-11-17 Thread David Christensen
Hi Jacob, Thanks, I've added this patch in my tree [1]. (For now, just adding fixes and the like atop the original separate patches, but will eventually get things winnowed down into probably the same 12 parts the originals were reviewed in. Best, David [1]

Re: HOT chain validation in verify_heapam()

2022-11-17 Thread Himanshu Upadhyaya
On Wed, Nov 16, 2022 at 12:41 PM Himanshu Upadhyaya < upadhyaya.himan...@gmail.com> wrote: > > >> > + } >> > + >> > + /* Loop over offsets and validate the data in the >> predecessor array. */ >> > + for (OffsetNumber currentoffnum = FirstOffsetNumber; >>

Re: redundant check of msg in does_not_exist_skipping

2022-11-17 Thread Tom Lane
Japin Li writes: > On Thu, 17 Nov 2022 at 23:06, Japin Li wrote: >> I think we cannot remove the check, for example, if objtype is >> OBJECT_OPFAMILY, >> and schema_does_not_exist_skipping() returns true, the so the msg keeps NULL, >> if we remove this check, a sigfault might be occurd in

Re: allow segment size to be set to < 1GiB

2022-11-17 Thread Tom Lane
Andres Freund writes: > On 2022-11-17 09:58:48 -0500, Andrew Dunstan wrote: >> Are we going to impose some sane minimum, or leave it up to developers >> to discover that for themselves? > I don't think we should. It's actually useful to e.g. use 1 page sized > segments for testing, and with one

Re: [PoC] configurable out of disk space elog level

2022-11-17 Thread Andres Freund
On 2022-11-17 14:40:02 +0300, Maxim Orlov wrote: > On Wed, 16 Nov 2022 at 20:41, Andres Freund wrote: > > that aside, we can't do catalog accesses in all kinds of environments that > > this currently is active in - most importantly it's affecting the startup > > process. We don't do catalog

Re: allow segment size to be set to < 1GiB

2022-11-17 Thread Andres Freund
Hi, On 2022-11-17 09:58:48 -0500, Andrew Dunstan wrote: > On 2022-11-09 We 15:25, Andres Freund wrote: > > Hi, > > > > On 2022-11-09 14:44:42 -0500, Tom Lane wrote: > >> Andres Freund writes: > >>> A second question: Both autoconf and meson print the segment size as GB > >>> right > >>> now.

Re: Introduce "log_connection_stages" setting.

2022-11-17 Thread Justin Pryzby
On Tue, Nov 08, 2022 at 07:30:38PM +0100, Sergey Dudoladov wrote: > +Logs reception of a connection. At this point a connection > has been received, but no further work has been done: receipt > +Logs the original identity that an authentication method > employs to identify a

Re: redundant check of msg in does_not_exist_skipping

2022-11-17 Thread Japin Li
On Thu, 17 Nov 2022 at 23:06, Japin Li wrote: > On Thu, 17 Nov 2022 at 20:12, Ted Yu wrote: >> Hi, >> I was looking at commit aca992040951c7665f1701cd25d48808eda7a809 >> >> I think the check of msg after the switch statement is not necessary. The >> variable msg is used afterward. >> If there

Re: libpq compression (part 2)

2022-11-17 Thread Peter Eisentraut
On 17.11.22 01:27, Andrey Borodin wrote: Also I've found one more TODO item for the patch. Currently in fe-connect.c patch is doing buffer reset: conn->inStart = conn->inCursor = conn->inEnd = 0; This effectively consumes butes up tu current cursor. However, packet inspection is continued. The

Re: redundant check of msg in does_not_exist_skipping

2022-11-17 Thread Japin Li
On Thu, 17 Nov 2022 at 20:12, Ted Yu wrote: > Hi, > I was looking at commit aca992040951c7665f1701cd25d48808eda7a809 > > I think the check of msg after the switch statement is not necessary. The > variable msg is used afterward. > If there is (potential) missing case in switch statement, the

contrib: auth_delay module

2022-11-17 Thread 成之焕
The attached patch is a contrib module to set login restrictions on users with too many authentication failure. The administrator could manage several GUC parameters to control the login restrictions which are listed below. - set the wait time when password authentication fails. - allow the

Re: libpq support for NegotiateProtocolVersion

2022-11-17 Thread Peter Eisentraut
On 16.11.22 19:35, Jacob Champion wrote: On Tue, Nov 15, 2022 at 2:19 AM Peter Eisentraut wrote: I think for the current code, the following would be an appropriate adjustment: diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c index 746e9b4f1efc..d15fb96572d9

Re: allow segment size to be set to < 1GiB

2022-11-17 Thread Andrew Dunstan
On 2022-11-09 We 15:25, Andres Freund wrote: > Hi, > > On 2022-11-09 14:44:42 -0500, Tom Lane wrote: >> Andres Freund writes: >>> A second question: Both autoconf and meson print the segment size as GB >>> right >>> now. Obviously that'll print out a size of 0 for a segsize < 1GB. >>> The

Re: Hash index build performance tweak from sorting

2022-11-17 Thread Tomas Vondra
Hi, I did some simple benchmark with v2 and v3, using the attached script, which essentially just builds hash index on random data, with different data types and maintenance_work_mem values. And what I see is this (median of 10 runs): machine data type m_w_mmasterv2

Re: locale -a missing on Alpine Linux?

2022-11-17 Thread Tom Lane
Andrew Dunstan writes: > malleefowl is a docker instance (mostly docker images is what Alpine is > used for). It would be extremely easy to recreate the image and add in > musl-locales, but maybe we should just leave it as it is to test the > case where the command isn't available. Agreed, leave

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-11-17 Thread Robert Haas
On Wed, Nov 16, 2022 at 2:52 PM Andres Freund wrote: > I don't think we'd want every buffer write or whatnot go through the > changecount mechanism, on some non-x86 platforms that could be noticable. But > if we didn't stage the stats updates locally I think we could make most of the > stats

Re: Supporting TAP tests with MSVC and Windows

2022-11-17 Thread Andrew Dunstan
On 2022-11-17 Th 01:18, Noah Misch wrote: > On Mon, Aug 22, 2022 at 09:44:42AM -0400, Andrew Dunstan wrote: >> On 2022-08-21 Su 20:40, Noah Misch wrote: >>> This (commit 13d856e of 2015-07-29) added the following: >>> >>> --- a/src/test/perl/TestLib.pm >>> +++ b/src/test/perl/TestLib.pm >>> @@

Re: locale -a missing on Alpine Linux?

2022-11-17 Thread Andrew Dunstan
On 2022-11-16 We 14:25, Tom Lane wrote: > Christoph Moench-Tegeder writes: >> ## Peter Eisentraut (peter.eisentr...@enterprisedb.com): >>> First of all, is this a standard installation of this OS, or is perhaps >>> something incomplete, broken, or unusual about the current OS installation? >>

Re: HOT chain validation in verify_heapam()

2022-11-17 Thread Robert Haas
On Wed, Nov 16, 2022 at 10:57 PM Himanshu Upadhyaya wrote: > I think if pred_xmin is aborted and curr_xmin is in progress we should > consider it as a corruption case but vice versa is not true. Yeah, you're right. I'm being stupid about subtransactions again. -- Robert Haas EDB:

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-11-17 Thread Bharath Rupireddy
On Thu, Nov 17, 2022 at 12:49 AM Robert Haas wrote: > > > I also think that a new pg_stat_checkpoint view is needed > > because, right now, the PgStat_CheckpointerStats stats are exposed via > > the pg_stat_bgwriter view, having a separate view for checkpoint stats > > is good here. > > Yep. On

Don't treate IndexStmt like AlterTable when DefineIndex is called from ProcessUtilitySlow.

2022-11-17 Thread 正华吕
Hi, Recently read the code, I find that when calling DefineIndex from ProcessUtilitySlow, is_alter_table will be set true if this statement is came from expandTableLikeClause. I check the code of DefineIndex, there are only two places use is_alter_table: 1. the function

Introduce a new view for checkpointer related stats

2022-11-17 Thread Bharath Rupireddy
Hi, pg_stat_bgwriter view currently reports checkpointer stats as well. It is that way because historically checkpointer was part of bgwriter until the commits 806a2ae and bf405ba, that went into PG 9.2, separated them out. I think it is time for us to separate checkpointer stats to its own view.

Re: old_snapshot: add test for coverage

2022-11-17 Thread Daniel Gustafsson
> On 8 Aug 2022, at 07:37, Tom Lane wrote: > Dong Wook Lee writes: >> I wrote a test of the old_snapshot extension for coverage. > > Hmm, does this really provide any meaningful coverage? The test > sure looks like it's not doing much. Looking at this I agree, this test doesn't provide

CREATE UNLOGGED TABLE seq faults when debug_discard_caches=1

2022-11-17 Thread Spyridon Dimitrios Agathos
Hi Hackers, while testing the developer settings of PSQL (14.5) I came across this issue: postgres=# CREATE UNLOGGED TABLE stats ( postgres(# pg_hash BIGINT NOT NULL, postgres(# category TEXT NOT NULL, postgres(# PRIMARY KEY (pg_hash, category) postgres(# ); server closed the

Re: Typo in SH_LOOKUP and SH_LOOKUP_HASH comments

2022-11-17 Thread Daniel Gustafsson
> On 17 Nov 2022, at 10:56, vignesh C wrote: > When I was reviewing one of the patches, I found a typo in the > comments of SH_LOOKUP and SH_LOOKUP_HASH. I felt "lookup up" should > have been "lookup". > Attached a patch to modify it. I agree with that, applied to HEAD. Thanks! -- Daniel

redundant check of msg in does_not_exist_skipping

2022-11-17 Thread Ted Yu
Hi, I was looking at commit aca992040951c7665f1701cd25d48808eda7a809 I think the check of msg after the switch statement is not necessary. The variable msg is used afterward. If there is (potential) missing case in switch statement, the compiler would warn. How about removing the check ? Thanks

Re: Lockless queue of waiters in LWLock

2022-11-17 Thread Pavel Borisov
Hi, hackers! I've noticed that alignment requirements for using pg_atomic_init_u64() for LWlock state (there's an assertion that it's aligned at 8 bytes) do not correspond to the code in SimpleLruInit() on 32-bit arch when MAXIMUM_ALIGNOF = 4. Fixed this in v4 patch (PFA). Regards, Pavel.

Re: [PoC] Reducing planning time when tables have many partitions

2022-11-17 Thread Thom Brown
On Thu, 17 Nov 2022 at 11:20, Thom Brown wrote: > > On Thu, 17 Nov 2022 at 09:31, Alvaro Herrera wrote: > > > > On 2022-Nov-16, Thom Brown wrote: > > > > > Once the issue Tom identified has been resolved, I'd like to test > > > drive newer patches. > > > > What issue? If you mean the one from

  1   2   >