Re: Synchronizing slots from primary to standby

2024-01-12 Thread Amit Kapila
On Fri, Jan 12, 2024 at 5:50 PM shveta malik wrote: > > There are multiple approaches discussed and tried when it comes to > starting a slot-sync worker. I am summarizing all here: > > 1) Make slotsync worker as an Auxiliary Process (like checkpointer, > walwriter, walreceiver etc). The benefit t

Re: plpgsql memory leaks

2024-01-12 Thread Pavel Stehule
Hi pá 12. 1. 2024 v 22:25 odesílatel Tom Lane napsal: > Pavel Stehule writes: > > default master branch - res 190MB ram > > jit_inline_above_cost = -1 doesn't helps > > disabling JIT doesn't helps too, > > > so it looks like the wrong hypothesis , and the problem is maybe > somewhere > > else :

Re: Recovering from detoast-related catcache invalidations

2024-01-12 Thread Xiaoran Wang
Great! That's what exactly we need. The patch LGTM, +1 Tom Lane 于2024年1月13日周六 04:47写道: > I wrote: > > This is uncomfortably much in bed with the tuple table slot code, > > perhaps, but I don't see a way to do it more cleanly unless we want > > to add some new provisions to that API. Andres,

Re: Synchronizing slots from primary to standby

2024-01-12 Thread Amit Kapila
On Fri, Jan 12, 2024 at 12:07 PM Bertrand Drouvot wrote: > > On Fri, Jan 12, 2024 at 08:42:39AM +0530, Amit Kapila wrote: > > On Thu, Jan 11, 2024 at 9:11 PM Bertrand Drouvot > > wrote: > > > > > > I'm not sure to follow here. If the remote slot is re-created then it > > > would > > > be also dr

Re: Fix minor memory leak in connection string validation

2024-01-12 Thread Tom Lane
Nathan Bossart writes: > On Fri, Jan 12, 2024 at 03:06:26PM -0800, Jeff Davis wrote: >> It makes me wonder if we should use the resowner mechanism to track >> pointers to malloc'd memory. Then we could use a standard pattern for >> these kinds of cases, and it would also catch more remote issues,

Re: Fix minor memory leak in connection string validation

2024-01-12 Thread Nathan Bossart
On Fri, Jan 12, 2024 at 03:06:26PM -0800, Jeff Davis wrote: > It makes me wonder if we should use the resowner mechanism to track > pointers to malloc'd memory. Then we could use a standard pattern for > these kinds of cases, and it would also catch more remote issues, like > if a pstrdup() fails i

Re: cfbot is failing all tests on FreeBSD/Meson builds

2024-01-12 Thread Thomas Munro
On Sat, Jan 13, 2024 at 1:51 PM Tom Lane wrote: > Time for a bug report to IO::Tty's authors, I guess. Ahh, there is one: https://github.com/cpan-authors/IO-Tty/issues/38 In the meantime, will look into whether I can pin that package to 1.17 somewhere in the pipeline, hopefully later today...

Re: cfbot is failing all tests on FreeBSD/Meson builds

2024-01-12 Thread Tom Lane
Thomas Munro writes: > Looking at the relevant packages > p5-IPC-Run and p5-IO-Tty I see that they recently moved to 20231003.0 > and 1.18, respectively. Downgrading p5-IPC-Run to > p5-IPC-Run-20220807.0.pkg was not enough. Downgrading p5-IO-Tty to > p5-IO-Tty-1.17.pkg allowed psql/010_tab_comp

Re: cfbot is failing all tests on FreeBSD/Meson builds

2024-01-12 Thread Thomas Munro
On Sat, Jan 13, 2024 at 9:32 AM Tom Lane wrote: > It looks like every recent cfbot run has failed in the > FreeBSD-13-Meson build, even if it worked in other ones. > The symptoms are failures in the TAP tests that try to > use interactive_psql: > > Can't call method "slave" on an undefined value a

Fix minor memory leak in connection string validation

2024-01-12 Thread Jeff Davis
Introduced in commit c3afe8cf5a. Someone issuing repeated "CREATE SUBSCRIPTION" commands where the connection has no password and must_have_password is true will leak malloc'd memory in the error path. Minor issue in practice, because I suspect that a user privileged enough to create a subscriptio

Re: pg_upgrade failing for 200+ million Large Objects

2024-01-12 Thread Tom Lane
Nathan Bossart writes: > On Wed, Dec 20, 2023 at 06:47:44PM -0500, Tom Lane wrote: >> +char *cmdEnd = psprintf(" OWNER TO %s", >> fmtId(te->owner)); >> + >> +IssueCommandPerBlob(AH, te, "ALTER LARGE OBJECT ", >> cmdEnd); > This is just a nitpick, bu

Re: pg_upgrade failing for 200+ million Large Objects

2024-01-12 Thread Nathan Bossart
On Fri, Jan 12, 2024 at 04:42:27PM -0600, Nathan Bossart wrote: > On Wed, Dec 20, 2023 at 06:47:44PM -0500, Tom Lane wrote: >> +char *cmdEnd = psprintf(" OWNER TO %s", >> fmtId(te->owner)); >> + >> +IssueCommandPerBlob(AH, te, "ALTER LARGE OBJECT ", >

Re: pg_upgrade failing for 200+ million Large Objects

2024-01-12 Thread Nathan Bossart
On Fri, Jan 05, 2024 at 03:02:34PM -0500, Tom Lane wrote: > On further reflection, there is a very good reason why it's done like > that. Because pg_upgrade is doing schema-only dump and restore, > there's next to no opportunity for parallelism within either pg_dump > or pg_restore. There's no da

Re: pg_upgrade failing for 200+ million Large Objects

2024-01-12 Thread Nathan Bossart
On Wed, Dec 20, 2023 at 06:47:44PM -0500, Tom Lane wrote: > * I did not invent a switch to control the batching of blobs; it's > just hard-wired at 1000 blobs per group here. Probably we need some > user knob for that, but I'm unsure if we want to expose a count or > just a boolean for one vs more

Re: [DOC] Add detail regarding resource consumption wrt max_connections

2024-01-12 Thread Cary Huang
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed I think it is good to warn the user about the increased alloc

Re: Make NUM_XLOGINSERT_LOCKS configurable

2024-01-12 Thread Jim Nasby
On 1/12/24 12:32 AM, Bharath Rupireddy wrote: Test case: ./pgbench --initialize --scale=100 --username=ubuntu postgres ./pgbench --progress=10 --client=64 --time=300 --builtin=tpcb-like --username=ubuntu postgres Setup: ./configure --prefix=$PWD/inst/ CFLAGS="-ggdb3 -O3" > install.log && make -

Re: plpgsql memory leaks

2024-01-12 Thread Tom Lane
Pavel Stehule writes: > default master branch - res 190MB ram > jit_inline_above_cost = -1 doesn't helps > disabling JIT doesn't helps too, > so it looks like the wrong hypothesis , and the problem is maybe somewhere > else :-/ I see no leak with these examples on HEAD, either with or without --

Re: libpq compression (part 3)

2024-01-12 Thread Robert Haas
On Fri, Jan 12, 2024 at 4:02 PM Jacob Burroughs wrote: > > I wonder if we could use "upstream" and "downstream" to be clearer? Or > > some other terminology? > > What about `send` and `receive`? I think that would definitely be better than "compress" and "decompress," but I was worried that it mi

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Melanie Plageman
On Fri, Jan 12, 2024 at 3:22 PM Robert Haas wrote: > > On Fri, Jan 12, 2024 at 3:04 PM Melanie Plageman > wrote: > > So what's the best way to solve the problem that Peter pointed out? > Should we pass in the prunestate? Maybe just replace bool > *recordfreespace with bool *has_lpdead_items? Yea

Re: libpq compression (part 3)

2024-01-12 Thread Jacob Burroughs
> I wonder if we could use "upstream" and "downstream" to be clearer? Or > some other terminology? What about `send` and `receive`?

Re: Stack overflow issue

2024-01-12 Thread Robert Haas
On Fri, Jan 12, 2024 at 10:12 AM Heikki Linnakangas wrote: > Here's one goto-free attempt. It adds a local loop to where the > recursion was, so that if you have a chain of subtransactions that need > to be aborted in CommitTransactionCommand, they are aborted iteratively. > The TBLOCK_SUBCOMMIT c

Re: Recovering from detoast-related catcache invalidations

2024-01-12 Thread Tom Lane
I wrote: > This is uncomfortably much in bed with the tuple table slot code, > perhaps, but I don't see a way to do it more cleanly unless we want > to add some new provisions to that API. Andres, do you have any > thoughts about that? Oh! After nosing around a bit more I remembered systable_rec

Re: libpq compression (part 3)

2024-01-12 Thread Robert Haas
On Sun, Dec 31, 2023 at 2:32 AM Jacob Burroughs wrote: > I ended up reworking this to use a version of this option in place of > the `none` hackery, but naming the parameters `compress` and > `decompress, so to disable compression but allow decompression you > would specify `libpq_compression=gzip

cfbot is failing all tests on FreeBSD/Meson builds

2024-01-12 Thread Tom Lane
It looks like every recent cfbot run has failed in the FreeBSD-13-Meson build, even if it worked in other ones. The symptoms are failures in the TAP tests that try to use interactive_psql: Can't call method "slave" on an undefined value at /usr/local/lib/perl5/site_perl/IPC/Run.pm line 2889. I s

Re: Invalidate the subscription worker in cases where a user loses their superuser status

2024-01-12 Thread Jeff Davis
On Tue, 2023-10-17 at 14:17 +0530, Amit Kapila wrote: > > Fair enough. I'll wait till early next week (say till Monday EOD) > > to > > see if anyone thinks otherwise and push this patch to HEAD after > > some > > more testing and review. > > > > Pushed. There was a brief discussion on backportin

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Robert Haas
On Fri, Jan 12, 2024 at 3:04 PM Melanie Plageman wrote: > Also, I think you should combine these in lazy_scan_noprune() now > > /* Save any LP_DEAD items found on the page in dead_items array */ > if (vacrel->nindexes == 0) > { > /* Using one-pass strategy (since table has no i

Re: Recovering from detoast-related catcache invalidations

2024-01-12 Thread Tom Lane
Xiaoran Wang writes: >> Maybe, but that undocumented hack in SetHintBits seems completely >> unacceptable. Isn't there a cleaner way to make this check? > Maybe we don't need to call 'HeapTupleSatisfiesVisibility' to check if the > tuple has been deleted. > As the tuple's xmin must been committe

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Melanie Plageman
On Fri, Jan 12, 2024 at 2:47 PM Robert Haas wrote: > > On Fri, Jan 12, 2024 at 2:43 PM Peter Geoghegan wrote: > > You're using "!prunestate.has_lpdead_items" as part of your test that > > sets "recordfreespace". But lazy_scan_noprune doesn't get passed a > > pointer to prunestate, so clearly you'

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Peter Geoghegan
On Fri, Jan 12, 2024 at 1:52 PM Melanie Plageman wrote: > On Fri, Jan 12, 2024 at 1:07 PM Peter Geoghegan wrote: > > What is "space_freed"? Isn't that something from your uncommitted patch? > > Yes, I was mixing the two together. An understandable mistake. > I just want to make sure that we agr

Re: Extensible storage manager API - SMGR hook Redux

2024-01-12 Thread Tristan Partin
Thought I would show off what is possible with this patchset. Heikki, a couple of months ago in our internal Slack, said: [I would like] a debugging tool that checks that we're not missing any fsyncs. I bumped into a few missing fsync bugs with unlogged tables lately and a tool like that would

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Robert Haas
On Fri, Jan 12, 2024 at 2:43 PM Peter Geoghegan wrote: > You're using "!prunestate.has_lpdead_items" as part of your test that > sets "recordfreespace". But lazy_scan_noprune doesn't get passed a > pointer to prunestate, so clearly you'll need to detect the same > condition some other way. OOPS.

Re: introduce dynamic shared memory registry

2024-01-12 Thread Nathan Bossart
On Fri, Jan 12, 2024 at 11:13:46PM +0530, Abhijit Menon-Sen wrote: > At 2024-01-12 11:21:52 -0600, nathandboss...@gmail.com wrote: >> + Each backend sould obtain a pointer to the reserved shared memory by > > sould → should D'oh. Thanks. >> + Add-ins can reserve LWLocks on server star

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Peter Geoghegan
On Fri, Jan 12, 2024 at 2:32 PM Robert Haas wrote: > On Fri, Jan 12, 2024 at 1:52 PM Melanie Plageman > wrote: > This analysis seems correct to me, except that "when > lazy_scan_noprune() is called" should really say "when > lazy_scan_noprune() is called (and returns true)", because when it > ret

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Robert Haas
On Fri, Jan 12, 2024 at 1:52 PM Melanie Plageman wrote: > Yes, I was mixing the two together. > > I just want to make sure that we agree that, on master, when > lazy_scan_prune() is called, the logic for whether or not to update > the FSM after the first pass is: > > indexes == 0 || !has_lpdead_it

Re: Custom explain options

2024-01-12 Thread Konstantin Knizhnik
On 12/01/2024 7:03 pm, Tomas Vondra wrote: On 10/21/23 14:16, Konstantin Knizhnik wrote: Hi hackers, EXPLAIN statement has a list of options (i.e. ANALYZE, BUFFERS, COST,...) which help to provide useful details of query execution. In Neon we have added PREFETCH option which shows information

Re: plpgsql memory leaks

2024-01-12 Thread Pavel Stehule
pá 12. 1. 2024 v 14:53 odesílatel Michael Banck napsal: > Hi, > > On Fri, Jan 12, 2024 at 01:35:24PM +0100, Pavel Stehule wrote: > > pá 12. 1. 2024 v 11:54 odesílatel Michael Banck napsal: > > > Which version of Postgres is this and on which platform/distribution? > > > > It was tested on master

Re: Confine vacuum skip logic to lazy_scan_skip

2024-01-12 Thread Melanie Plageman
On Fri, Jan 12, 2024 at 2:02 PM Jim Nasby wrote: > > On 1/11/24 5:50 PM, Melanie Plageman wrote: > > On Fri, Jan 5, 2024 at 5:51 AM Nazir Bilal Yavuz wrote: > >> > >> On Fri, 5 Jan 2024 at 02:25, Jim Nasby wrote: > >>> > >>> On 1/4/24 2:23 PM, Andres Freund wrote: > >>> > >>> On 2024-01-02 12:36

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Melanie Plageman
On Fri, Jan 12, 2024 at 1:07 PM Peter Geoghegan wrote: > > On Fri, Jan 12, 2024 at 12:33 PM Melanie Plageman > wrote: > > So, I think this is the logic in master: > > > > Prune case, first pass > > > > ... > > - indexes > 0 && (!space_freed || !index_vacuuming) -> update FSM > > What is "space_fr

Re: Confine vacuum skip logic to lazy_scan_skip

2024-01-12 Thread Jim Nasby
On 1/11/24 5:50 PM, Melanie Plageman wrote: On Fri, Jan 5, 2024 at 5:51 AM Nazir Bilal Yavuz wrote: On Fri, 5 Jan 2024 at 02:25, Jim Nasby wrote: On 1/4/24 2:23 PM, Andres Freund wrote: On 2024-01-02 12:36:18 -0500, Melanie Plageman wrote: Subject: [PATCH v2 1/6] lazy_scan_skip remove unn

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Melanie Plageman
On Fri, Jan 12, 2024 at 1:07 PM Peter Geoghegan wrote: > > On Fri, Jan 12, 2024 at 12:33 PM Melanie Plageman > wrote: > > So, I think this is the logic in master: > > > > Prune case, first pass > > > > ... > > - indexes > 0 && (!space_freed || !index_vacuuming) -> update FSM > > What is "space_fr

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Robert Haas
On Fri, Jan 12, 2024 at 11:50 AM Peter Geoghegan wrote: > Why do you think that lazy_scan_prune() and lazy_scan_noprune() have > different ideas about whether to update the FSM or not? When lazy_scan_prune() is called, we call RecordPageWithFreeSpace if vacrel->nindexes == 0 && prunestate.has_lpd

Re: speed up a logical replica setup

2024-01-12 Thread Nazir Bilal Yavuz
Hi, On Fri, 12 Jan 2024 at 09:32, Hayato Kuroda (Fujitsu) wrote: > > Good, all warnings were removed. However, the patch failed to pass tests on > FreeBSD twice. > I'm quite not sure the ERROR, but is it related with us? FreeBSD errors started after FreeBSD's CI image was updated [1]. I do not

Re: Built-in CTYPE provider

2024-01-12 Thread Jeff Davis
On Fri, 2024-01-12 at 19:00 +0100, Daniel Verite wrote: > Another one is that version 12 broke \d in older psql by > removing pg_class.relhasoids. > > ISTM that in general the behavior of old psql vs new server does > not weight much against choosing optimal catalog changes. > > There's also that

Re: Built-in CTYPE provider

2024-01-12 Thread Robert Haas
On Fri, Jan 12, 2024 at 1:00 PM Daniel Verite wrote: > ISTM that in general the behavior of old psql vs new server does > not weight much against choosing optimal catalog changes. +1. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Peter Geoghegan
On Fri, Jan 12, 2024 at 12:33 PM Melanie Plageman wrote: > So, I think this is the logic in master: > > Prune case, first pass > > ... > - indexes > 0 && (!space_freed || !index_vacuuming) -> update FSM What is "space_freed"? Isn't that something from your uncommitted patch? As I said, the aim i

Re: Built-in CTYPE provider

2024-01-12 Thread Daniel Verite
Jeff Davis wrote: > > Jeremy also raised a problem with old versions of psql connecting to > > a > > new server: the \l and \dO won't work. Not sure exactly what to do > > there, but I could work around it by adding a new field rather than > > renaming (though that's not ideal). > > I did

Re: introduce dynamic shared memory registry

2024-01-12 Thread Abhijit Menon-Sen
At 2024-01-12 11:21:52 -0600, nathandboss...@gmail.com wrote: > > From: Nathan Bossart > Date: Thu, 11 Jan 2024 21:55:25 -0600 > Subject: [PATCH v6 1/3] reorganize shared memory and lwlocks documentation > > --- > doc/src/sgml/xfunc.sgml | 182 +--- > 1 file c

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Melanie Plageman
On Fri, Jan 12, 2024 at 11:50 AM Peter Geoghegan wrote: > > On Fri, Jan 12, 2024 at 9:44 AM Robert Haas wrote: > > Looking through the git history, I see that this behavior seems to > > date back to 44fa84881fff4529d68e2437a58ad2c906af5805 which introduced > > lazy_scan_noprune(). The comments do

Re: reorganize "Shared Memory and LWLocks" section of docs

2024-01-12 Thread Nathan Bossart
On Fri, Jan 12, 2024 at 09:46:50AM -0600, Nathan Bossart wrote: > On Fri, Jan 12, 2024 at 05:12:28PM +0300, Aleksander Alekseev wrote: >> """ >> Any registered shmem_startup_hook will be executed shortly after each >> backend attaches to shared memory. >> """ >> >> IMO the word "each" here can giv

Re: Report planning memory in EXPLAIN ANALYZE

2024-01-12 Thread Abhijit Menon-Sen
At 2024-01-12 17:52:27 +0100, alvhe...@alvh.no-ip.org wrote: > > I think this patch is mostly OK (After the last few rounds of changes, it looks fine to me too.) > Planning: >Buffers: shared hit=120 read=30 >Memory: used=67944 bytes allocated=73728 bytes > Planning Time: 0.892 ms > > […

Re: introduce dynamic shared memory registry

2024-01-12 Thread Nathan Bossart
Here's a new version of the patch set in which I've attempted to address the feedback in this thread. Note that 0001 is being tracked in a separate thread [0], but it is basically a prerequisite for adding the documentation for this feature, so that's why I've also added it here. [0] https://post

Re: psql not responding to SIGINT upon db reconnection

2024-01-12 Thread Tristan Partin
On Fri Jan 12, 2024 at 10:45 AM CST, Robert Haas wrote: On Mon, Jan 8, 2024 at 1:03 AM Tristan Partin wrote: > I think the way to go is to expose some variation of libpq's > pqSocketPoll(), which I would be happy to put together a patch for. > Making frontends, psql in this case, have to reimple

Re: Custom explain options

2024-01-12 Thread Tomas Vondra
On 10/21/23 14:16, Konstantin Knizhnik wrote: > Hi hackers, > > EXPLAIN statement has a list of options (i.e. ANALYZE, BUFFERS, > COST,...) which help to provide useful details of query execution. > In Neon we have added PREFETCH option which shows information about page > prefetching during query

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Melanie Plageman
On Fri, Jan 12, 2024 at 9:44 AM Robert Haas wrote: > > On Thu, Jan 11, 2024 at 9:05 PM Melanie Plageman > wrote: > > Ah! I think you are right. Good catch. I could fix this with logic like > > this: > > > > bool space_freed = vacrel->tuples_deleted > tuples_already_deleted; > > if ((vacrel->nind

Re: Built-in CTYPE provider

2024-01-12 Thread Jeff Davis
On Thu, 2024-01-11 at 18:02 -0800, Jeff Davis wrote: > Jeremy also raised a problem with old versions of psql connecting to > a > new server: the \l and \dO won't work. Not sure exactly what to do > there, but I could work around it by adding a new field rather than > renaming (though that's not id

Re: index prefetching

2024-01-12 Thread Robert Haas
Not a full response, but just to address a few points: On Fri, Jan 12, 2024 at 11:42 AM Tomas Vondra wrote: > Thinking about this, I think it should be possible to make prefetching > work even for plans with execute_once=false. In particular, when the > plan changes direction it should be possibl

Re: Report planning memory in EXPLAIN ANALYZE

2024-01-12 Thread Alvaro Herrera
I think this patch is mostly OK and decided to use it to test something else. In doing so I noticed one small thing that bothers me: if planning uses buffers, and those were requested to be reported, we get something like this at the end of the explain Planning: Buffers: shared hit=120 read=3

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Peter Geoghegan
On Fri, Jan 12, 2024 at 9:44 AM Robert Haas wrote: > Looking through the git history, I see that this behavior seems to > date back to 44fa84881fff4529d68e2437a58ad2c906af5805 which introduced > lazy_scan_noprune(). The comments don't explain the reasoning, but my > guess is that it was just an ac

Re: psql not responding to SIGINT upon db reconnection

2024-01-12 Thread Robert Haas
On Mon, Jan 8, 2024 at 1:03 AM Tristan Partin wrote: > I think the way to go is to expose some variation of libpq's > pqSocketPoll(), which I would be happy to put together a patch for. > Making frontends, psql in this case, have to reimplement the polling > logic doesn't strike me as fruitful, wh

Re: index prefetching

2024-01-12 Thread Tomas Vondra
Hi, Here's an improved version of this patch, finishing a lot of the stuff that I alluded to earlier - moving the code from indexam.c, renaming a bunch of stuff, etc. I've also squashed it into a single patch, to make it easier to review. I'll briefly go through the main changes in the patch, and

Re: pgsql: Add new pg_walsummary tool.

2024-01-12 Thread Robert Haas
On Thu, Jan 11, 2024 at 1:49 PM Robert Haas wrote: > On Thu, Jan 11, 2024 at 12:56 PM Robert Haas wrote: > > Add new pg_walsummary tool. > > culicidae is unhappy with this, but I don't yet understand why. The output is: > > # Failed test 'stdout shows block 0 modified' > # at t/002_blocks.pl

Re: pgsql: Add new pg_walsummary tool.

2024-01-12 Thread Robert Haas
On Thu, Jan 11, 2024 at 12:56 PM Robert Haas wrote: > Add new pg_walsummary tool. culicidae is unhappy with this, but I don't yet understand why. The output is: # Failed test 'stdout shows block 0 modified' # at t/002_blocks.pl line 85. # 'TS 1663, DB 5, REL 16384, FORK mai

Re: Error "initial slot snapshot too large" in create replication slot

2024-01-12 Thread Robert Haas
On Thu, Jan 11, 2024 at 9:47 PM Kyotaro Horiguchi wrote: > I understand. So, summirizing the current state briefly, I believe it > as follows: > > a. snapbuild lacks a means to differentiate between top and sub xids > during snapshot building. > > b. Abusing takenDuringRecovery could lead to pot

Re: System username in pg_stat_activity

2024-01-12 Thread Magnus Hagander
On Thu, Jan 11, 2024 at 5:55 PM Bertrand Drouvot wrote: > > Hi, > > On Thu, Jan 11, 2024 at 02:24:58PM +0100, Magnus Hagander wrote: > > On Wed, Jan 10, 2024 at 3:12 PM Bertrand Drouvot > > wrote: > > > > > > If we go the 2 fields way, then what about auth_identity and auth_method > > > then? >

Re: Make all Perl warnings fatal

2024-01-12 Thread Bharath Rupireddy
On Fri, Jan 12, 2024 at 9:03 PM Peter Eisentraut wrote: > > On 11.01.24 12:29, Bharath Rupireddy wrote: > > On Sat, Dec 30, 2023 at 12:57 AM Peter Eisentraut > > wrote: > >> > >> committed > > > > With the commit c5385929 converting perl warnings to FATAL, use of > > psql/safe_psql with timeout

Re: tablecmds.c/MergeAttributes() cleanup

2024-01-12 Thread Robert Haas
On Fri, Jan 12, 2024 at 10:09 AM Robert Haas wrote: > Open-coding stuff like this can easily work out to a loss, and I > personally think we're overly dependent on List. It's not a > particularly good abstraction, IMHO, and if we do a lot of work to > start using it everywhere because a list is re

Re: reorganize "Shared Memory and LWLocks" section of docs

2024-01-12 Thread Nathan Bossart
Thanks for reviewing. On Fri, Jan 12, 2024 at 05:12:28PM +0300, Aleksander Alekseev wrote: > """ > Any registered shmem_startup_hook will be executed shortly after each > backend attaches to shared memory. > """ > > IMO the word "each" here can give the wrong impression as if there are > certain

Re: the s_lock_stuck on perform_spin_delay

2024-01-12 Thread Robert Haas
On Wed, Jan 10, 2024 at 10:17 PM Andy Fan wrote: > fixed in v2. Timing the spinlock wait seems like a separate patch from the new sanity checks. I suspect that the new sanity checks should only be armed in assert-enabled builds. I'm doubtful that this method of tracking the wait time will be ac

Re: tablecmds.c/MergeAttributes() cleanup

2024-01-12 Thread Tom Lane
Robert Haas writes: > On Fri, Jan 12, 2024 at 5:32 AM Alvaro Herrera > wrote: >> In addition, it also occurs to me now that maybe it would make sense to >> change the TupleDesc implementation to use a List of Form_pg_attribute >> instead of an array, and do away with ->natts. This would let us

Re: Make all Perl warnings fatal

2024-01-12 Thread Peter Eisentraut
On 11.01.24 12:29, Bharath Rupireddy wrote: On Sat, Dec 30, 2023 at 12:57 AM Peter Eisentraut wrote: committed With the commit c5385929 converting perl warnings to FATAL, use of psql/safe_psql with timeout parameters [1] fail with the following error: Use of uninitialized value $ret in bitw

Re: [PATCH] New predefined role pg_manage_extensions

2024-01-12 Thread Jelte Fennema-Nio
On Fri, 12 Jan 2024 at 15:53, Michael Banck wrote: > I propose to add a new predefined role to Postgres, > pg_manage_extensions. The idea is that it allows Superusers to delegate > the rights to create, update or delete extensions to other roles, even > if those extensions are not trusted or those

Re: Stack overflow issue

2024-01-12 Thread Heikki Linnakangas
On 11/01/2024 19:37, Robert Haas wrote: On Wed, Jan 10, 2024 at 4:25 PM Heikki Linnakangas wrote: The problem with CommitTransactionCommand (or rather AbortCurrentTransaction() which has the same problem) and ShowTransactionStateRec is that they get called in a state where aborting can lead to

Re: tablecmds.c/MergeAttributes() cleanup

2024-01-12 Thread Robert Haas
On Fri, Jan 12, 2024 at 5:32 AM Alvaro Herrera wrote: > On 2024-Jan-11, Alvaro Herrera wrote: > > If you look closely at InsertPgAttributeTuples and accompanying code, it > > all looks a bit archaic. They seem to be treating TupleDesc as a > > glorified array of Form_pg_attribute elements in a co

[PATCH] New predefined role pg_manage_extensions

2024-01-12 Thread Michael Banck
Hi, I propose to add a new predefined role to Postgres, pg_manage_extensions. The idea is that it allows Superusers to delegate the rights to create, update or delete extensions to other roles, even if those extensions are not trusted or those users are not the database owner. I have attached a W

Re: cleanup patches for incremental backup

2024-01-12 Thread Robert Haas
On Thu, Jan 11, 2024 at 8:00 PM Shinoda, Noriyoshi (HPE Services Japan - FSIP) wrote: > Thank you for developing the new tool. I have attached a patch that corrects > the spelling of the --individual option in the SGML file. Thanks, committed. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Emit fewer vacuum records by reaping removable tuples during pruning

2024-01-12 Thread Robert Haas
On Thu, Jan 11, 2024 at 9:05 PM Melanie Plageman wrote: > Yes, this is a good point. Seems like writing maintainable code is > really about never lying and then figuring out when hiding the truth > is also lying. Darn! I think that's pretty true. In this particular case, this code has a fair numb

Re: reorganize "Shared Memory and LWLocks" section of docs

2024-01-12 Thread Aleksander Alekseev
Hi, > I recently began trying to write documentation for the dynamic shared > memory registry feature [0], and I noticed that the "Shared Memory and > LWLocks" section of the documentation might need some improvement. I know that feeling. > Thoughts? """ Any registered shmem_startup_hook will b

Re: plpgsql memory leaks

2024-01-12 Thread Michael Banck
Hi, On Fri, Jan 12, 2024 at 01:35:24PM +0100, Pavel Stehule wrote: > pá 12. 1. 2024 v 11:54 odesílatel Michael Banck napsal: > > Which version of Postgres is this and on which platform/distribution? > > It was tested on master branch (pg 17) on Fedora 39 > > > Did you try keep jit on but set ji

Re: Test slots invalidations in 035_standby_logical_decoding.pl only if dead rows are removed

2024-01-12 Thread Bertrand Drouvot
Hi, On Fri, Jan 12, 2024 at 02:00:01PM +0300, Alexander Lakhin wrote: > Hi, > > 12.01.2024 10:15, Bertrand Drouvot wrote: > > > > For this one, the "good" news is that it looks like that we don’t see the > > "terminating" message not followed by an "obsolete" message (so the engine > > behaves c

Re: Improve the connection failure error messages

2024-01-12 Thread Aleksander Alekseev
Hi, Thanks for the patch. > Due to this behavior, it is not possible to add a test to show the > error message as it is done for CREATE SUBSCRIPTION. > Let me know if you think there is another way to add this test. I believe it can be done with TAP tests, see for instance: contrib/auto_explain

Re: Compile warnings in dbcommands.c building with meson

2024-01-12 Thread jian he
On Fri, Jan 12, 2024 at 8:03 PM Alvaro Herrera wrote: > > On 2024-Jan-12, jian he wrote: > > > I saw it sometimes, sometimes not. > > Now I think the reason is: > > it will appear when you do `-Dbuildtype=release`. > > > > but it will not occur when I do: > > `-Dbuildtype=debug` > > > > my current

Re: Show WAL write and fsync stats in pg_stat_io

2024-01-12 Thread Nazir Bilal Yavuz
Hi, On Thu, 11 Jan 2024 at 17:28, Melanie Plageman wrote: > > On Thu, Jan 11, 2024 at 6:19 AM Nazir Bilal Yavuz wrote: > > > > On Thu, 11 Jan 2024 at 08:01, Michael Paquier wrote: > > > > > > On Wed, Jan 10, 2024 at 07:24:50PM -0500, Melanie Plageman wrote: > > > > On Wed, Jan 3, 2024 at 8:11 A

Re: plpgsql memory leaks

2024-01-12 Thread Pavel Stehule
pá 12. 1. 2024 v 11:54 odesílatel Michael Banck napsal: > Hi, > > On Fri, Jan 12, 2024 at 11:02:14AM +0100, Pavel Stehule wrote: > > pá 12. 1. 2024 v 10:27 odesílatel Pavel Stehule > > > napsal: > > > > > Hi > > > > > > I have reported very memory expensive pattern: > > [...] > > > > takes lot o

Re: Synchronizing slots from primary to standby

2024-01-12 Thread shveta malik
On Fri, Jan 12, 2024 at 5:30 PM Masahiko Sawada wrote: > > On Thu, Jan 11, 2024 at 7:53 PM Amit Kapila wrote: > > > > On Tue, Jan 9, 2024 at 6:39 PM Amit Kapila wrote: > > > > > > +static bool > > > +synchronize_one_slot(WalReceiverConn *wrconn, RemoteSlot *remote_slot) > > > { > > > ... > > > +

Re: Improve the connection failure error messages

2024-01-12 Thread Nisha Moond
Thanks for the review. Attached v2 patch with suggested changes. Please find my response inline. On Fri, Jan 12, 2024 at 8:20 AM Peter Smith wrote: > > Thanks for the patch! Here are a couple of review comments for it. > > == > src/backend/commands/subscriptioncmds.c > > 1. > @@ -742,7 +742,7

Re: Make mesage at end-of-recovery less scary.

2024-01-12 Thread Aleksander Alekseev
Hi, > The errors occurred in a part of the tests for end-of-WAL detection > added in the master branch. These failures were primarily due to > changes in the message contents introduced by this patch. During the > revision, I discovered an issue with the handling of empty pages that > appear in th

Re: Compile warnings in dbcommands.c building with meson

2024-01-12 Thread Alvaro Herrera
On 2024-Jan-12, jian he wrote: > I saw it sometimes, sometimes not. > Now I think the reason is: > it will appear when you do `-Dbuildtype=release`. > > but it will not occur when I do: > `-Dbuildtype=debug` > > my current meson version is 1.3.1, my ninja version is 1.10.1. Hmm, but why doesn't

Re: Synchronizing slots from primary to standby

2024-01-12 Thread Masahiko Sawada
On Thu, Jan 11, 2024 at 7:53 PM Amit Kapila wrote: > > On Tue, Jan 9, 2024 at 6:39 PM Amit Kapila wrote: > > > > +static bool > > +synchronize_one_slot(WalReceiverConn *wrconn, RemoteSlot *remote_slot) > > { > > ... > > + /* Slot ready for sync, so sync it. */ > > + else > > + { > > + /* > > + *

Re: doc: add LITERAL tag to RETURNING

2024-01-12 Thread Alvaro Herrera
On 2024-Jan-12, Ashutosh Bapat wrote: > On Fri, Jan 12, 2024 at 11:27 AM torikoshia > wrote: > > > > RETURNING is usually tagged with appropriate tags, such as , > > but not in the 'query' section of COPY. > The patch looks good. Good catch, pushed. It has user-visible effect, so I backpatche

Re: Make attstattarget nullable

2024-01-12 Thread Alvaro Herrera
BTW I wanted to but didn't say so explicitly, so here goes: 0001 looks ready to go in. -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ "Find a bug in a program, and fix it, and the program will work today. Show the program how to find and fix a bug, and the progra

Re: Make attstattarget nullable

2024-01-12 Thread Alvaro Herrera
On 2024-Jan-11, Peter Eisentraut wrote: > On 10.01.24 14:16, Alvaro Herrera wrote: > > Seems reasonable. Do we really need a catversion bump for this? > > Yes, this changes the order of the fields in pg_attribute. Ah, right. > > In get_attstattarget() I think we should return 0 for dropped co

Re: plpgsql memory leaks

2024-01-12 Thread Michael Banck
Hi, On Fri, Jan 12, 2024 at 11:02:14AM +0100, Pavel Stehule wrote: > pá 12. 1. 2024 v 10:27 odesílatel Pavel Stehule > napsal: > > > Hi > > > > I have reported very memory expensive pattern: [...] > > takes lot of megabytes of memory too. > > The megabytes leaks are related to JIT. With JIT o

Re: tablecmds.c/MergeAttributes() cleanup

2024-01-12 Thread Alvaro Herrera
On 2024-Jan-11, Alvaro Herrera wrote: > If you look closely at InsertPgAttributeTuples and accompanying code, it > all looks a bit archaic. They seem to be treating TupleDesc as a > glorified array of Form_pg_attribute elements in a convenient packaging. > It's probably cleaner to change these AP

RE: speed up a logical replica setup

2024-01-12 Thread Hayato Kuroda (Fujitsu)
Dear Euler, Here are comments for your v5 patch. 01. In the document, two words target/standby are used as almost the same meaning. Can you unify them? 02. ``` pg_subscriber option ``` There are some mandatory options like -D/-S/-P. It must be listed in Synopsis chapter. 03. `

plperl and perl 5.38

2024-01-12 Thread Christoph Berg
Perl 5.38 has landed in Debian unstable, and plperl doesn't like it: diff -U3 /home/myon/projects/postgresql/pg/postgresql/src/pl/plperl/expected/plperl_elog_1.out /home/myon/projects/postgresql/pg/postgresql/build/testrun/plperl/regress/results/plperl_elog.out --- /home/myon/projects/postgres

Re: A failure in t/038_save_logical_slots_shutdown.pl

2024-01-12 Thread Bharath Rupireddy
On Fri, Jan 12, 2024 at 9:28 AM Amit Kapila wrote: > > On Thu, Jan 11, 2024 at 10:03 PM Bharath Rupireddy > wrote: > > > > On Thu, Jan 11, 2024 at 4:35 PM vignesh C wrote: > > > > > > On further analysis, it was found that in the failing test, > > > CHECKPOINT_SHUTDOWN was started in a new page,

Re: plpgsql memory leaks

2024-01-12 Thread Pavel Stehule
pá 12. 1. 2024 v 10:27 odesílatel Pavel Stehule napsal: > Hi > > I have reported very memory expensive pattern: > > CREATE OR REPLACE FUNCTION public.fx(iter integer) > RETURNS void > LANGUAGE plpgsql > AS $function$ > declare > c cursor(m bigint) for select distinct i from generate_series(1,

Re: alter table add x wrong error position

2024-01-12 Thread Alvaro Herrera
On 2024-Jan-08, jian he wrote: > hi. > Maybe this is a small printout err_position bug. > > create table atacc2 ( test int, a int, b int) ; > success tests: > alter table atacc2 add CONSTRAINT x PRIMARY KEY (id, b ); > alter table atacc2 add CONSTRAINT x PRIMARY KEY (id, b a); > alter table atacc

plpgsql memory leaks

2024-01-12 Thread Pavel Stehule
Hi I have reported very memory expensive pattern: CREATE OR REPLACE FUNCTION public.fx(iter integer) RETURNS void LANGUAGE plpgsql AS $function$ declare c cursor(m bigint) for select distinct i from generate_series(1, m) g(i); t bigint; s bigint; begin for i in 1..iter loop open c

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-01-12 Thread Masahiko Sawada
On Thu, Jan 11, 2024 at 9:28 AM Masahiko Sawada wrote: > > On Mon, Jan 8, 2024 at 8:35 PM John Naylor wrote: > > > > On Wed, Jan 3, 2024 at 9:10 PM John Naylor wrote: > > > > > > On Tue, Jan 2, 2024 at 8:01 PM Masahiko Sawada > > > wrote: > > > > > > > I agree that we expose RT_LOCK_* function

  1   2   >