Re: Avoiding "wrong tuple length" errors at the end of VACUUM on pg_database update (Backpatch of 947789f to v12 and v13)

2023-01-09 Thread Tom Lane
Michael Paquier writes: > Any objections about getting 947789f applied to REL_13_STABLE and > REL_12_STABLE and see this issue completely gone from all the versions > supported? No objections to back-patching the fix, but I wonder if we can find some mechanical way to prevent this sort of error

Re: typos

2023-01-09 Thread Michael Paquier
On Tue, Jan 10, 2023 at 12:24:40PM +0530, Amit Kapila wrote: > Thanks for noticing this. I'll take care of this and some other typo > patches together. Does this include 0010? I was just looking at the whole set and this one looked like a cleanup worth on its own so I was going to handle it,

Avoiding "wrong tuple length" errors at the end of VACUUM on pg_database update (Backpatch of 947789f to v12 and v13)

2023-01-09 Thread Michael Paquier
Hi all, The problem mentioned in $subject has been discussed here: https://www.postgresql.org/message-id/dm5pr0501mb38800d9e4605bca72dd35557cc...@dm5pr0501mb3880.namprd05.prod.outlook.com Ths issue has been fixed by 947789f, without a backpatch to v12 (as per 96cdeae) as the risk seemed rather

Re: split TOAST support out of postgres.h

2023-01-09 Thread Noah Misch
On Tue, Jan 10, 2023 at 06:07:49AM +0100, Peter Eisentraut wrote: > On 30.12.22 17:50, Tom Lane wrote: > >Peter Eisentraut writes: > >>On 28.12.22 16:07, Tom Lane wrote: > >>>I dunno, #3 seems kind of unprincipled. Also, since fmgr.h is included > >>>so widely, I doubt it is buying very much in

Re: allowing for control over SET ROLE

2023-01-09 Thread Jeff Davis
On Fri, 2022-12-30 at 22:16 -0800, Noah Misch wrote: > create user unpriv; > grant pg_maintain to unpriv with set false; > create schema maint authorization pg_maintain >   create table t (c int); > create or replace function maint.f() returns int language sql as > 'select 1'; > alter function

Re: FYI: 2022-10 thorntail failures from coreutils FICLONE

2023-01-09 Thread Noah Misch
On Mon, Jan 09, 2023 at 10:49:26PM -0500, Tom Lane wrote: > Noah Misch writes: > > thorntail failed some recovery tests in 2022-10: > > Speaking of which ... thorntail hasn't reported in for nearly > three weeks. Is it stuck? Its machine has been unresponsive to ssh for those weeks.

Re: typos

2023-01-09 Thread Amit Kapila
On Tue, Jan 10, 2023 at 10:27 AM Justin Pryzby wrote: > > On Tue, Jan 03, 2023 at 03:39:22PM -0600, Justin Pryzby wrote: > > On Tue, Jan 03, 2023 at 04:28:29PM +0900, Michael Paquier wrote: > > > On Fri, Dec 30, 2022 at 05:12:57PM -0600, Justin Pryzby wrote: > > > > > > # Use larger ccache

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread Amit Kapila
On Tue, Jan 10, 2023 at 11:16 AM Kyotaro Horiguchi wrote: > > At Mon, 9 Jan 2023 14:21:03 +0530, Amit Kapila > wrote in > > Pushed the first (0001) patch. > > It added the following error message. > > + seg = dsm_attach(handle); > + if (!seg) > + ereport(ERROR, > +

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 12:58 PM Peter Geoghegan wrote: > I didn't realize that affected visibilitymap_set() calls could > generate useless set-VM WAL records until you pointed it out. That's > far more likely to happen than the race condition that I described -- > it has nothing at all to do with

Re: Add BufFileRead variants with short read and EOF detection

2023-01-09 Thread Amit Kapila
On Fri, Jan 6, 2023 at 6:18 PM Peter Eisentraut wrote: > > On 02.01.23 13:13, Amit Kapila wrote: > > On Wed, Dec 28, 2022 at 4:17 PM Peter Eisentraut > > wrote: > >> > >> Most callers of BufFileRead() want to check whether they read the full > >> specified length. Checking this at every call

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Brar Piening
On 10.01.2023 at 06:28, Brar Piening wrote: I'll repost a rebased version of the styling patch in a minute. After checking that there's no need for rebasing I'm reposting the original patch here, to make cfbot pick it up as the latest one in a somewhat screwed up thread mixing two patches

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread Kyotaro Horiguchi
Hello. At Mon, 9 Jan 2023 14:21:03 +0530, Amit Kapila wrote in > Pushed the first (0001) patch. It added the following error message. + seg = dsm_attach(handle); + if (!seg) + ereport(ERROR, +

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-01-09 Thread Tom Lane
David Rowley writes: > Ideally, our sort costing would just be better, but I think that > raises the bar a little too high to start thinking of making > improvements to that for this patch. It's trickier than it looks, cf f4c7c410e. But if you just want to add a small correction based on number

Re: wake up logical workers after ALTER SUBSCRIPTION

2023-01-09 Thread Amit Kapila
On Sat, Jan 7, 2023 at 6:15 AM Nathan Bossart wrote: > > On Fri, Jan 06, 2023 at 05:31:26PM -0500, Tom Lane wrote: > > > Attached is a rebased 0003, just to keep the cfbot happy. > > I'm kind of wondering whether 0003 is worth the complexity TBH, > > but in any case I ran out of time to look at

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Brar Piening
On 09.01.2023 at 23:28, Karl O. Pinc wrote: On Mon, 09 Jan 2023 15:18:18 -0500 Tom Lane wrote: It's probably going to be necessary to have follow-on patches, because I'm sure there is stuff in the pipeline that adds more ID-less tags. Or do we have a way to create warnings about that? I am

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-01-09 Thread David Rowley
On Tue, 10 Jan 2023 at 06:15, Ankit Kumar Pandey wrote: > Do we have any pending items for this patch now? I'm just wondering if not trying this when the query has a DISTINCT clause is a copout. What I wanted to avoid was doing additional sorting work for WindowAgg just to have it destroyed by

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Tom Lane
Brar Piening writes: > On 09.01.2023 at 21:18, Tom Lane wrote: >> * I got rid of a couple of "-et-al" additions, because it did not >> seem like a good precedent. That would tempt people to modify >> existing ID tags when adding variables to an entry, which'd defeat >> the purpose I think. > I

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Brar Piening
On 09.01.2023 at 21:18, Tom Lane wrote: It's not great to have multiple CF entries pointing at the same email thread --- it confuses both people and bots. Next time please split off a thread for each distinct patch. I agree. I had overestimated the cfbot's ability to handle branched threads.

Re: split TOAST support out of postgres.h

2023-01-09 Thread Peter Eisentraut
On 30.12.22 17:50, Tom Lane wrote: Peter Eisentraut writes: On 28.12.22 16:07, Tom Lane wrote: I dunno, #3 seems kind of unprincipled. Also, since fmgr.h is included so widely, I doubt it is buying very much in terms of reducing header footprint. How bad is it to do #2? See this

Re: typos

2023-01-09 Thread Justin Pryzby
On Tue, Jan 03, 2023 at 03:39:22PM -0600, Justin Pryzby wrote: > On Tue, Jan 03, 2023 at 04:28:29PM +0900, Michael Paquier wrote: > > On Fri, Dec 30, 2022 at 05:12:57PM -0600, Justin Pryzby wrote: > > > > # Use larger ccache cache, as this task compiles with multiple > > compilers / > >

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread houzj.f...@fujitsu.com
On Monday, January 9, 2023 4:51 PM Amit Kapila wrote: > > On Sun, Jan 8, 2023 at 11:32 AM houzj.f...@fujitsu.com > wrote: > > > > On Sunday, January 8, 2023 11:59 AM houzj.f...@fujitsu.com > wrote: > > > Attach the updated patch set. > > > > Sorry, the commit message of 0001 was accidentally

Re: [Proposal] Add foreign-server health checks infrastructure

2023-01-09 Thread Tom Lane
"Hayato Kuroda (Fujitsu)" writes: > Thanks for reporting. PSA rebased version. > These can be applied work well on my HEAD(bd8d45). I think that it's a really bad idea to require postgres_fdw.sql to have two expected-files: that will be a maintenance nightmare. Please put whatever it is that

Re: Strengthen pg_waldump's --save-fullpage tests

2023-01-09 Thread Bharath Rupireddy
On Tue, Jan 10, 2023 at 6:52 AM Michael Paquier wrote: > > On Mon, Jan 09, 2023 at 08:30:00AM +0530, Bharath Rupireddy wrote: > > A recent commit [1] added --save-fullpage option to pg_waldump to > > extract full page images (FPI) from WAL records and save them into > > files (one file per FPI)

Re: Show various offset arrays for heap WAL records

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 1:58 PM Andres Freund wrote: > A couple times when investigating data corruption issues, the last time just > yesterday in [1], I needed to see the offsets affected by PRUNE and VACUUM > records. As that's probably not just me, I think we should make that change > in-tree.

Re: FYI: 2022-10 thorntail failures from coreutils FICLONE

2023-01-09 Thread Tom Lane
Noah Misch writes: > thorntail failed some recovery tests in 2022-10: Speaking of which ... thorntail hasn't reported in for nearly three weeks. Is it stuck? regards, tom lane

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Mark Dilger
> On Jan 9, 2023, at 2:07 PM, Andres Freund wrote: > > The tests encounter the issue today. If you add > Assert(TransactionIdIsValid(ctx->next_xid)); > Assert(FullTransactionIdIsValid(ctx->next_fxid)); > early in FullTransactionIdFromXidAndCtx() it'll be hit in the > amcheck/pg_amcheck tests.

RE: Fix pg_publication_tables to exclude generated columns

2023-01-09 Thread shiy.f...@fujitsu.com
On Mon, Jan 9, 2023 11:06 PM Tom Lane wrote: > > Amit Kapila writes: > > On Mon, Jan 9, 2023 at 5:29 PM shiy.f...@fujitsu.com > > wrote: > >> I think one way to fix it is to modify pg_publication_tables query to > >> exclude > >> generated columns. But in this way, we need to bump catalog

Re: Allow DISTINCT to use Incremental Sort

2023-01-09 Thread Richard Guo
On Tue, Jan 10, 2023 at 10:14 AM David Rowley wrote: > > /* For explicit-sort case, always use the more rigorous clause */ > > if (list_length(root->distinct_pathkeys) < > > list_length(root->sort_pathkeys)) > > { > > needed_pathkeys =

RE: [PATCH] Support % wildcard in extension upgrade filenames

2023-01-09 Thread Regina Obe
> I continue to think that this is a fundamentally bad idea. It creates all sorts of > uncertainties about what is a valid update path and what is not. Restrictions > like > > + Such wildcard update > + scripts will only be used when no explicit path is found from > + old to target

Re: Handle infinite recursion in logical replication setup

2023-01-09 Thread Jonathan S. Katz
On 9/12/22 1:23 AM, vignesh C wrote: On Fri, 9 Sept 2022 at 11:12, Amit Kapila wrote: On Thu, Sep 8, 2022 at 9:32 AM vignesh C wrote: The attached patch has the changes to handle the same. Pushed. I am not completely sure whether we want the remaining documentation patch in this thread

Re: ATTACH PARTITION seems to ignore column generation status

2023-01-09 Thread Tom Lane
Amit Langote writes: > Thanks for the patch. It looks good, though I guess you said that we > should also change the error message that CREATE TABLE ... PARTITION > OF emits to match the other cases while we're here. I've attached a > delta patch. Thanks. I hadn't touched that issue because I

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-09 Thread Andres Freund
Hi, On 2023-01-05 13:44:20 -0500, Reid Thompson wrote: > From 0a6b152e0559a25033bd7d43eb0959432e0d Mon Sep 17 00:00:00 2001 > From: Reid Thompson > Date: Thu, 11 Aug 2022 12:01:25 -0400 > Subject: [PATCH 1/2] Add tracking of backend memory allocated to > pg_stat_activity > > This new field

Re: ATTACH PARTITION seems to ignore column generation status

2023-01-09 Thread Amit Langote
On Tue, Jan 10, 2023 at 6:41 AM Tom Lane wrote: > I wrote: > > After thinking about this awhile, I feel that we ought to disallow > > it in the traditional-inheritance case as well. The reason is that > > there are semantic prohibitions on inserting or updating a generated > > column, eg > > >

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-09 Thread Takamichi Osumi (Fujitsu)
On Tuesday, December 27, 2022 6:29 PM Tuesday, December 27, 2022 6:29 PM wrote: > Thanks for reviewing our patch! PSA new version patch set. Now, the patches fails to apply to the HEAD, because of recent commits (c6e1f62e2c and 216a784829c) as reported in [1]. I'll rebase the patch with other

Re: Schema variables - new implementation for Postgres 15 (typo)

2023-01-09 Thread Julien Rouhaud
Hi, On Sat, Jan 07, 2023 at 08:09:27AM +0100, Pavel Stehule wrote: > > > > Hmm, how safe is it for third-party code to access the stored data directly > > rather than a copy? If it makes extension fragile if they're not careful > > enough with cache invalidation, or even give them a way to mess

Re: Allow DISTINCT to use Incremental Sort

2023-01-09 Thread David Rowley
Thanks for having a look at this. On Tue, 10 Jan 2023 at 02:28, Richard Guo wrote: > +1 for the changes. A minor comment is that previously on HEAD for > SELECT DISTINCT case, if we have to do an explicit full sort atop the > cheapest path, we try to make sure to always use the more rigorous >

Re: refactoring relation extension and BufferAlloc(), faster COPY

2023-01-09 Thread Andres Freund
Hi, On 2023-01-06 11:52:04 +0530, vignesh C wrote: > On Sat, 29 Oct 2022 at 08:24, Andres Freund wrote: > > > > The patches here aren't fully polished (as will be evident). But they should > > be more than good enough to discuss whether this is a sane direction. > > The patch does not apply on

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-09 Thread Andres Freund
Hi, On 2023-01-03 12:05:20 -0800, Andres Freund wrote: > The attached patch adds libpq-be-fe-helpers.h and uses it in libpqwalreceiver, > dblink, postgres_fdw. > As I made libpq-be-fe-helpers.h handle reserving external fds, > libpqwalreceiver now does so. I briefly looked through its users

Re: Add index scan progress to pg_stat_progress_vacuum

2023-01-09 Thread Andres Freund
Hi, On 2023-01-07 01:59:40 +, Imseih (AWS), Sami wrote: > --- a/src/backend/access/nbtree/nbtree.c > +++ b/src/backend/access/nbtree/nbtree.c > @@ -998,6 +998,8 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult > *stats, > if (info->report_progress) >

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 5:22 PM Andres Freund wrote: > I've also seen the inverse, with recent versions of postgres: Autovacuum can > only ever make progress if it's an anti-wraparound vacuum, because it'll > always get cancelled otherwise. I'm worried that substantially increasing the > time

Re: Strengthen pg_waldump's --save-fullpage tests

2023-01-09 Thread Michael Paquier
On Mon, Jan 09, 2023 at 08:30:00AM +0530, Bharath Rupireddy wrote: > A recent commit [1] added --save-fullpage option to pg_waldump to > extract full page images (FPI) from WAL records and save them into > files (one file per FPI) under a specified directory. While it added > tests to check the

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-09 Thread Andres Freund
Hi, On 2023-01-08 17:49:20 -0800, Peter Geoghegan wrote: > Teach autovacuum.c to launch "table age" autovacuums at the same point > that it previously triggered antiwraparound autovacuums. Antiwraparound > autovacuums are retained, but are only used as a true option of last > resort, when

Re: Allow +group in pg_ident.conf

2023-01-09 Thread Michael Paquier
On Mon, Jan 09, 2023 at 05:33:14PM -0500, Andrew Dunstan wrote: > I've adapted a sentence from the pg_hba.conf documentation so we stay > consistent. + + If the database-username begins with a + + character, then the operating system user can login as + any user belonging to that role,

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Andres Freund
Hi, I'm a bit worried that this is optimizing the rare case while hurting the common case. See e.g. my point below about creating additional slots in the happy path. It's also not clear that change is right directionally. If we want to avoid re-fetching the "original" row version, why don't we

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Andres Freund
Hi, On 2023-01-09 13:46:50 +0300, Alexander Korotkov wrote: > I'm going to push v9. Could you hold off for a bit? I'd like to look at this, I'm not sure I like the direction. Greetings, Andres Freund

Re: [PATCH] random_normal function

2023-01-09 Thread Tom Lane
Dean Rasheed writes: > On Mon, 9 Jan 2023 at 18:52, Tom Lane wrote: >> (We could probably go further >> than this, like trying to verify distribution properties. But >> it's been too long since college statistics for me to want to >> write such tests myself, and I'm not real sure we need them.)

Re: Allow +group in pg_ident.conf

2023-01-09 Thread Jelte Fennema
This seems very much related to my patch here: https://commitfest.postgresql.org/41/4081/ (yes, somehow the thread got split. I blame outlook) I'll try to review this one soonish.

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 12:51 PM Robert Haas wrote: > I feel that you should at least have a reproducer for these problems > posted to the thread, and ideally a regression test, before committing > things. I think it's very hard to understand what the problems are > right now. Hard to understand

Re: allow granting CLUSTER, REFRESH MATERIALIZED VIEW, and REINDEX

2023-01-09 Thread Nathan Bossart
On Tue, Jan 03, 2023 at 03:45:49PM -0800, Nathan Bossart wrote: > I'd like to get this fixed, but I have yet to hear thoughts on the > suggested approach. I'll proceed with adjusting the tests and > documentation shortly unless someone objects. As promised, here is a new version of the patch

Re: [PATCH] Support % wildcard in extension upgrade filenames

2023-01-09 Thread Tom Lane
I continue to think that this is a fundamentally bad idea. It creates all sorts of uncertainties about what is a valid update path and what is not. Restrictions like + Such wildcard update + scripts will only be used when no explicit path is found from + old to target version. are

Re: [PATCH] random_normal function

2023-01-09 Thread Dean Rasheed
On Mon, 9 Jan 2023 at 18:52, Tom Lane wrote: > > I pushed Paul's patch with the previously-discussed tests, but > the more I look at random.sql the less I like it. I propose > that we nuke the existing tests from orbit and replace with > something more or less as attached. Looks like a definite

Re: Allow +group in pg_ident.conf

2023-01-09 Thread Andrew Dunstan
On 2023-01-09 Mo 13:24, Nathan Bossart wrote: > On Mon, Jan 09, 2023 at 08:00:26AM -0500, Andrew Dunstan wrote: >> + If the database-username begins with a >> + + character, then the operating system user can login >> as >> + any user belonging to that role, similarly to how user names

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Tom Lane
"Karl O. Pinc" writes: > I think there's more to comment on regards the xslt in the > other patch, but I'll wait for the new thread for that patch. > That might be where there should be warnings raised, etc. We can continue using this thread, now that the other commit is in.

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Karl O. Pinc
On Mon, 09 Jan 2023 15:18:18 -0500 Tom Lane wrote: > Brar Piening writes: > > On 09.01.2023 at 03:38, vignesh C wrote: > >> There are couple of commitfest entries for this: > >> https://commitfest.postgresql.org/41/4041/ > >> https://commitfest.postgresql.org/41/4042/ Can one of them be > >>

Re: WAL Insertion Lock Improvements (was: Re: Avoid LWLockWaitForVar() for currently held WAL insertion lock in WaitXLogInsertionsToFinish())

2023-01-09 Thread Andres Freund
Hi, On 2022-12-08 12:29:54 +0530, Bharath Rupireddy wrote: > On Tue, Dec 6, 2022 at 12:00 AM Andres Freund wrote: > > I think it'd be safe to optimize LWLockConflictsWithVar(), due to some > > pre-existing, quite crufty, code. LWLockConflictsWithVar() says: > > > > * Test first to see

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 1:43 PM Andres Freund wrote: > ISTM that some of the page level freezing functions are misnamed. In heapam.c > the heap_xlog* routines are for replay, afaict. However > heap_xlog_freeze_plan() is used to WAL log the freeze > plan. heap_xlog_freeze_page() is used to replay

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Andres Freund
Hi, On 2023-01-09 13:55:02 -0800, Mark Dilger wrote: > > On Jan 9, 2023, at 11:34 AM, Andres Freund wrote: > > > > 1) Because ctx->next_xid is set after the XidFromFullTransactionId() call in > > update_cached_xid_range(), we end up using the xid 0 (or an outdated value > > in > > subsequent

Show various offset arrays for heap WAL records

2023-01-09 Thread Andres Freund
Hi, A couple times when investigating data corruption issues, the last time just yesterday in [1], I needed to see the offsets affected by PRUNE and VACUUM records. As that's probably not just me, I think we should make that change in-tree. The attached patch adds details to XLOG_HEAP2_PRUNE,

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Mark Dilger
> On Jan 9, 2023, at 11:34 AM, Andres Freund wrote: > > 1) Because ctx->next_xid is set after the XidFromFullTransactionId() call in > update_cached_xid_range(), we end up using the xid 0 (or an outdated value in > subsequent calls) to determine whether epoch needs to be reduced. Can you say

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2023-01-09 Thread Andres Freund
Hi, On 2022-11-15 10:26:05 -0800, Peter Geoghegan wrote: > Pushed something like this earlier today, though without any changes > to VISIBLE records. While updating a patch to log various offsets in pg_waldump, I noticed a few minor issues in this patch: ISTM that some of the page level

Re: ATTACH PARTITION seems to ignore column generation status

2023-01-09 Thread Tom Lane
I wrote: > After thinking about this awhile, I feel that we ought to disallow > it in the traditional-inheritance case as well. The reason is that > there are semantic prohibitions on inserting or updating a generated > column, eg > regression=# create table t (f1 int, f2 int generated always as

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2023-01-09 Thread Melanie Plageman
Attached is v45 of the patchset. I've done some additional code cleanup and changes. The most significant change, however, is the docs. I've separated the docs into its own patch for ease of review. The docs patch here was edited and co-authored by Samay Sharma. I'm not sure if the order of

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 11:57 AM Andres Freund wrote: > Afaict we'll need to backpatch this all the way? I thought that we probably wouldn't need to, at first. But I now think that we really have to. I didn't realize that affected visibilitymap_set() calls could generate useless set-VM WAL

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Robert Haas
On Mon, Jan 2, 2023 at 1:32 PM Peter Geoghegan wrote: > On Sat, Dec 31, 2022 at 4:53 PM Peter Geoghegan wrote: > > The first patch makes sure that the snapshotConflictHorizon cutoff > > (XID cutoff for recovery conflicts) is never a special XID, unless > > that XID is InvalidTransactionId, which

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Karl O. Pinc
On Mon, 09 Jan 2023 15:18:18 -0500 Tom Lane wrote: > I pushed the ID-addition patch, with a few fixes: > > * AFAIK our practice is to use "-" never "_" in XML ID attributes. > You weren't very consistent about that even within this patch, and > the overall effect would have been to have no

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-01-09 Thread Tom Lane
Andres Freund writes: > On 2023-01-07 13:50:04 -0500, Tom Lane wrote: >> Also ... maybe I am missing something, but is REPLICA IDENTITY FULL >> sanely defined in the first place? It looks to me that >> RelationFindReplTupleSeq assumes without proof that there is a unique >> full-tuple match, but

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Thomas Munro
On Tue, Jan 10, 2023 at 8:34 AM Andres Freund wrote: > A different approach would be to represent fxids as *signed* 64bit > integers. That'd of course loose more range, but could represent everything > accurately, and would have a compatible on-disk representation on two's > complement platforms

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Tom Lane
Brar Piening writes: > On 09.01.2023 at 03:38, vignesh C wrote: >> There are couple of commitfest entries for this: >> https://commitfest.postgresql.org/41/4041/ >> https://commitfest.postgresql.org/41/4042/ Can one of them be closed? > I've split the initial patch into two parts upon Álvaro's

Re: Split index and table statistics into different types of stats

2023-01-09 Thread Andres Freund
Hi, On 2023-01-09 17:08:42 +0530, Nitin Jadhav wrote: > +1 to keep common functions/code between table and index stats. Only > the data structure should be different as the goal is to shrink the > current memory usage. I don't think the goal is solely to shrink memory usage - it's also to make

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 11:44 AM Andres Freund wrote: > I think that's just an imprecise formulation though - the problem is that we > can call visibilitymap_set() with just VISIBILITYMAP_ALL_FROZEN, even though > VISIBILITYMAP_ALL_VISIBLE was concurrently unset. That's correct. You're right

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-01-09 Thread Andres Freund
Hi, On 2023-01-07 13:50:04 -0500, Tom Lane wrote: > I think we should either live within the constraints set by this > overarching design, or else nuke execReplication.c from orbit and > start using the real planner and executor. Perhaps the foreign > key enforcement mechanisms could be a model

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Andres Freund
Hi, On 2023-01-09 10:16:03 -0800, Peter Geoghegan wrote: > Attached is v3, which explicitly checks the need to set the PD_ALL_VISIBLE > flag at the relevant visibilitymap_set() call site. It also has improved > comments. Afaict we'll need to backpatch this all the way? > From

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Andres Freund
Hi, On 2023-01-08 16:27:59 -0800, Peter Geoghegan wrote: > On Sun, Jan 8, 2023 at 3:53 PM Andres Freund wrote: > > How? > > See the commit message for the scenario I have in mind, which involves > a concurrent HOT update that aborts. I looked at it. I specifically was wondering about this part

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Andres Freund
Hi, Robert, Mark, CCing you because of the amcheck aspect (see below). On 2023-01-09 17:50:10 +0100, Matthias van de Meent wrote: > On Sun, 8 Jan 2023 at 04:09, Andres Freund wrote: > > > For a bit I was thinking that we should introduce the notion that a > > > FullTransactionId can be from the

Re: [PATCH] random_normal function

2023-01-09 Thread Tom Lane
I wrote: > Hmm ... it occurred to me to try the same check on the existing > random() tests (attached), and darn if they don't fail even more > often, usually within 50K iterations. So maybe we should rethink > that whole thing. I pushed Paul's patch with the previously-discussed tests, but the

Re: Add SHELL_EXIT_CODE to psql

2023-01-09 Thread Corey Huinker
On Mon, Jan 9, 2023 at 10:01 AM Maxim Orlov wrote: > Hi! > > In overall, I think we move in the right direction. But we could make code > better, should we? > > + /* Capture exit code for SHELL_EXIT_CODE */ > + close_exit_code = pclose(fd); > + if

Re: Allow +group in pg_ident.conf

2023-01-09 Thread Nathan Bossart
On Mon, Jan 09, 2023 at 08:00:26AM -0500, Andrew Dunstan wrote: > + If the database-username begins with a > + + character, then the operating system user can login > as > + any user belonging to that role, similarly to how user names beginning > with > + + are treated in pg_hba.conf. I

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-01-09 Thread Önder Kalacı
Hi, Thank you for the useful comments! > I took a very brief look through this. I'm not too pleased with > this whole line of development TBH. It seems to me that the core > design of execReplication.c and related code is "let's build our > own half-baked executor and

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Peter Geoghegan
On Sun, Jan 8, 2023 at 6:43 PM Peter Geoghegan wrote: > On further reflection even v2 won't repair the page-level > PD_ALL_VISIBLE flag in passing in this scenario. ISTM that on HEAD we > might actually leave the all-frozen bit set in the VM, while both the > all-visible bit and the page-level

Re: Common function for percent placeholder replacement

2023-01-09 Thread Nathan Bossart
On Mon, Jan 09, 2023 at 09:36:12AM +0100, Peter Eisentraut wrote: > On 04.01.23 01:37, Nathan Bossart wrote: >> On Tue, Dec 20, 2022 at 06:30:40AM +0100, Peter Eisentraut wrote: >> > + * A value may be NULL. If the corresponding placeholder is found in the >> > + * input string, the whole

Re: add \dpS to psql

2023-01-09 Thread Nathan Bossart
On Sat, Jan 07, 2023 at 11:18:59AM +, Dean Rasheed wrote: > It might be true that temp tables aren't usually interesting from a > permissions point of view, but it's not hard to imagine situations > where interesting things do happen. It's also probably the case that > most users won't have

Re: MERGE ... RETURNING

2023-01-09 Thread Dean Rasheed
On Mon, 9 Jan 2023 at 16:23, Vik Fearing wrote: > > Bikeshedding here. Instead of Yet Another WITH Clause, could we perhaps > make a MERGING() function analogous to the GROUPING() function that goes > with grouping sets? > > MERGE ... > RETURNING *, MERGING('clause'), MERGING('action'); > Hmm,

Re: suppressing useless wakeups in logical/worker.c

2023-01-09 Thread Nathan Bossart
rebased for cfbot -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 2466001a3ae6f94aac8eff45b488765e619bea1b Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Thu, 1 Dec 2022 20:50:21 -0800 Subject: [PATCH v2 1/1] suppress unnecessary wakeups in logical/worker.c ---

Re: pub/sub - specifying optional parameters without values.

2023-01-09 Thread Zheng Li
Hi, This documentation change looks good to me. I verified in testing and in code that the value for boolean parameters in PUB/SUB commands can be omitted. which is equivalent to specifying TRUE. For example, CREATE PUBLICATIOIN mypub for ALL TABLES with (publish_via_partition_root); is

Re: wake up logical workers after ALTER SUBSCRIPTION

2023-01-09 Thread Nathan Bossart
rebased for cfbot -- Nathan Bossart Amazon Web Services: https://aws.amazon.com diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index cf220c3bcb..7661c0c86e 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -1996,6 +1996,16 @@ postgres

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-01-09 Thread Ankit Kumar Pandey
On 09/01/23 17:53, David Rowley wrote: We need to keep looping backwards until we find the first WindowClause which does not contain the pathkeys of the ORDER BY. I found the cause of confusion, *first* WindowClause means first from forward direction. Since we are looping from backward, I

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Matthias van de Meent
On Sun, 8 Jan 2023 at 04:09, Andres Freund wrote: > > Hi, > > On 2023-01-07 16:29:23 -0800, Andres Freund wrote: > > It's probably not too hard to fix specifically in this one place - we could > > just clamp vacuum_defer_cleanup_age to be smaller than latest_completed, but > > it strikes me as as

Re: Make EXPLAIN generate a generic plan for a parameterized query

2023-01-09 Thread Laurenz Albe
On Tue, 2022-12-27 at 14:37 -0800, Michel Pelletier wrote: > I built and tested this patch for review and it works well, although I got > the following warning when building: > > analyze.c: In function 'transformStmt': > analyze.c:2919:35: warning: 'generic_plan' may be used uninitialized in

Re: MERGE ... RETURNING

2023-01-09 Thread Vik Fearing
On 1/9/23 13:29, Dean Rasheed wrote: On Sun, 8 Jan 2023 at 20:09, Isaac Morland wrote: Would it be useful to have just the action? Perhaps "WITH ACTION"? My idea is that this would return an enum of INSERT, UPDATE, DELETE (so is "action" the right word?). It seems to me in many situations I

Re: [PATCH] random_normal function

2023-01-09 Thread Dean Rasheed
On Mon, 9 Jan 2023 at 15:26, Tom Lane wrote: > > Dean Rasheed writes: > > So IMO all pseudorandom functions should share the same PRNG state and > > seed-setting functions. That would mean they should all be in the same > > (new) C file, so that the PRNG state can be kept private to that file. >

psql: Add role's membership options to the \du+ command

2023-01-09 Thread Pavel Luzanov
When you include one role in another, you can specify three options: ADMIN, INHERIT (added in e3ce2de0) and SET (3d14e171). For example. CREATE ROLE alice LOGIN; GRANT pg_read_all_settings TO alice WITH ADMIN TRUE, INHERIT TRUE, SET TRUE; GRANT pg_stat_scan_tables TO alice WITH ADMIN FALSE,

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Karl O. Pinc
On Mon, 9 Jan 2023 08:09:02 +0100 Brar Piening wrote: > On 09.01.2023 at 03:31, vignesh C wrote: > > The patch does not apply on top of HEAD as in [1], please post a > > rebased patch: > This one applies on top of 3c569049b7b502bb4952483d19ce622ff0af5fd6 > and the documentation build

Re: PL/pgSQL cursors should get generated portal names by default

2023-01-09 Thread Pavel Stehule
Hi I wrote a new check in plpgsql_check, that tries to identify explicit work with the name of the referenced portal. create or replace function foo01() returns refcursor as $$#option dump declare c cursor for select 1; r refcursor; begin open c; r := 'c'; return r; end; $$ language

Re: [PATCH] Add `verify-system` sslmode to use system CA pool for server cert

2023-01-09 Thread Andrew Dunstan
On 2023-01-09 Mo 10:07, Jelte Fennema wrote: > Thanks for clarifying your reasoning. I now agree that ssrootcert=system > is now the best option. Cool, that looks like a consensus. > >>> 2. Should we allow the same approach with ssl_ca_file on the server side, >>> for client cert

Re: [PATCH] psql: Add tab-complete for optional view parameters

2023-01-09 Thread Jim Jones
Hi Christoph, Thanks for the patch! I just tested it and I could reproduce the expected behaviour in these cases: postgres=# CREATE VIEW w AS  WITH ( postgres=# CREATE OR REPLACE VIEW w AS  WITH ( postgres=# CREATE VIEW w WITH ( CHECK_OPTION  SECURITY_BARRIER  SECURITY_INVOKER

Re: [PATCH] random_normal function

2023-01-09 Thread Tom Lane
Dean Rasheed writes: > So IMO all pseudorandom functions should share the same PRNG state and > seed-setting functions. That would mean they should all be in the same > (new) C file, so that the PRNG state can be kept private to that file. I agree with this in principle, but I feel no need to

Re: [PATCH] Add `verify-system` sslmode to use system CA pool for server cert

2023-01-09 Thread Jelte Fennema
Thanks for clarifying your reasoning. I now agree that ssrootcert=system is now the best option. > > 2. Should we allow the same approach with ssl_ca_file on the server side, > > for client cert validation? > > I don't know enough about this use case to implement it safely. We'd > have to make

Re: Fix pg_publication_tables to exclude generated columns

2023-01-09 Thread Tom Lane
Amit Kapila writes: > On Mon, Jan 9, 2023 at 5:29 PM shiy.f...@fujitsu.com > wrote: >> I think one way to fix it is to modify pg_publication_tables query to exclude >> generated columns. But in this way, we need to bump catalog version when >> fixing >> it in back-branch. Another way is to

Re: Add SHELL_EXIT_CODE to psql

2023-01-09 Thread Maxim Orlov
Hi! In overall, I think we move in the right direction. But we could make code better, should we? + /* Capture exit code for SHELL_EXIT_CODE */ + close_exit_code = pclose(fd); + if (close_exit_code == -1) + { +

Re: Timeout when changes are filtered out by the core during logical replication

2023-01-09 Thread Ashutosh Bapat
On Fri, Dec 23, 2022 at 2:45 PM Amit Kapila wrote: > On Thu, Dec 22, 2022 at 6:58 PM Ashutosh Bapat > wrote: > > > > Hi All, > > A customer ran a script dropping a few dozens of users in a transaction. > Before dropping a user they change the ownership of the tables owned by > that user to

  1   2   >