Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 08:17:46PM -0800, Nathan Bossart wrote: > On Fri, Jan 27, 2023 at 05:55:42PM -0800, Andres Freund wrote: >> On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: >>> I think it would be weird for the archive module and >>> recovery module interfaces to look so different, but

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-27 Thread Amit Kapila
On Sat, Jan 28, 2023 at 9:36 AM houzj.f...@fujitsu.com wrote: > > On Friday, January 27, 2023 8:16 PM Amit Kapila > > > > On Fri, Jan 27, 2023 at 3:45 PM vignesh C wrote: > > > > > > On Mon, 23 Jan 2023 at 10:52, Amit Kapila > > wrote: > > > > > > > > IIRC, this is done to prevent concurrent

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Thomas Munro
On Sat, Jan 28, 2023 at 4:42 PM Andres Freund wrote: > Did you use the same compiler / compilation flags as when elver hit it? > Clearly Tomas' case was with at least some optimizations enabled. I did use the same compiler version and optimisation level, clang llvmorg-13.0.0-0-gd7b669b3a303 at

Re: Something is wrong with wal_compression

2023-01-27 Thread Tom Lane
Thomas Munro writes: > Reading Andres's comments and realising how relatively young > txid_status() is compared to txid_current(), I'm now wondering if we > shouldn't just disclaim the whole thing in back branches. My thoughts were trending in that direction too. It's starting to sound like we

Re: Something is wrong with wal_compression

2023-01-27 Thread Thomas Munro
On Sat, Jan 28, 2023 at 4:57 PM Andrey Borodin wrote: > It's not trustworthy anyway. Xid wraparound might happen during > reconnect. I suspect we can design a test that will show that it does > not always show correct results during xid->2pc conversion (there is a > point in time when xid is not

Re: suppressing useless wakeups in logical/worker.c

2023-01-27 Thread Amit Kapila
On Fri, Jan 27, 2023 at 4:07 AM Tom Lane wrote: > > Thanks, pushed. > > Returning to the prior patch ... I don't much care for this: > > +/* Maybe there will be a free slot in a second... */ > +retry_time = TimestampTzPlusSeconds(now, 1); > +

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 23:18:39 -0500, Tom Lane wrote: > I also saw it on florican, which is/was an i386 machine using clang and > pretty standard build options other than > 'CFLAGS' => '-msse2 -O2', > so I think this isn't too much about machine architecture or compiler > flags. Ah. Florican

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-27 Thread Takamichi Osumi (Fujitsu)
Hi, On Friday, January 27, 2023 8:00 PM Amit Kapila wrote: > On Fri, Jan 27, 2023 at 1:39 PM Takamichi Osumi (Fujitsu) > wrote: > > > > On Wednesday, January 25, 2023 11:24 PM I wrote: > > > Attached the updated v22. > > Hi, > > > > During self-review, I noticed some changes are required for

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 19:57:35 -0800, Andrey Borodin wrote: > On Fri, Jan 27, 2023 at 7:40 PM Tom Lane wrote: > > > > What are you using it for, that you don't care whether the answer > > is trustworthy? > > > > It's not trustworthy anyway. Xid wraparound might happen during > reconnect. I think

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Tom Lane
Andres Freund writes: > Except that you're saying that you hit this on elver (amd64), I think it'd be > interesting that we see the failure on an arm host, which has a less strict > memory order model than x86. I also saw it on florican, which is/was an i386 machine using clang and pretty

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 05:55:42PM -0800, Andres Freund wrote: > On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: >> I think it would be weird for the archive module and >> recovery module interfaces to look so different, but if that's okay, I can >> change it. > > I'm a bit sad about the

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 19:49:17 -0800, Andres Freund wrote: > It's quite commonly used as part of trigger based replication tools (IIRC > that's its origin), monitoring, as part of client side logging, as part of > snapshot management. Forgot one: Queues. The way it's used for trigger based

RE: Deadlock between logrep apply worker and tablesync worker

2023-01-27 Thread houzj.f...@fujitsu.com
On Friday, January 27, 2023 8:16 PM Amit Kapila > > On Fri, Jan 27, 2023 at 3:45 PM vignesh C wrote: > > > > On Mon, 23 Jan 2023 at 10:52, Amit Kapila > wrote: > > > > > > IIRC, this is done to prevent concurrent drops of origin drop say by > > > exposed API pg_replication_origin_drop(). See

Re: Something is wrong with wal_compression

2023-01-27 Thread Andrey Borodin
On Fri, Jan 27, 2023 at 7:40 PM Tom Lane wrote: > > What are you using it for, that you don't care whether the answer > is trustworthy? > It's not trustworthy anyway. Xid wraparound might happen during reconnect. I suspect we can design a test that will show that it does not always show correct

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 22:39:56 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-01-28 11:38:50 +0900, Michael Paquier wrote: > >> FWIW, my vote goes for a more expensive but reliable function even in > >> stable branches. > > > I very strenuously object. If we make txid_current() (by way

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 22:23:58 +1300, Thomas Munro wrote: > After 1000 make check loops, and 1000 make -C src/test/modules/test_shm_mq > check loops, on the same FBSD 13.1 machine as elver which has failed > like this once before, I haven't been able to reproduce this on > REL_12_STABLE. Did you use

Re: Something is wrong with wal_compression

2023-01-27 Thread Tom Lane
Andres Freund writes: > On 2023-01-28 11:38:50 +0900, Michael Paquier wrote: >> FWIW, my vote goes for a more expensive but reliable function even in >> stable branches. > I very strenuously object. If we make txid_current() (by way of > pg_current_xact_id()) flush WAL, we'll cause outages.

Re: heapgettup refactoring

2023-01-27 Thread David Rowley
"On Wed, 25 Jan 2023 at 10:17, Melanie Plageman wrote: > I've added a comment but I didn't include the function name in it -- I > find it repetitive when the comments above functions do that -- however, > I'm not strongly attached to that. I think the general format for header comments is: /*

Re: Generating code for query jumbling through gen_node_support.pl

2023-01-27 Thread Michael Paquier
On Tue, Jan 24, 2023 at 03:57:56PM +0900, Michael Paquier wrote: > Still, my plan here is to enforce the loading of > pg_stat_statements with compute_query_id = regress and > utility_query_id = jumble (if needed) in a new buildfarm machine, Actually, about this specific point, I have been able

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-28 11:38:50 +0900, Michael Paquier wrote: > On Fri, Jan 27, 2023 at 06:06:05AM +0100, Laurenz Albe wrote: > > On Fri, 2023-01-27 at 16:15 +1300, Thomas Munro wrote: > >> There is no > >> doubt that the current situation is unacceptable, though, so maybe we > >> really should just

Re: Something is wrong with wal_compression

2023-01-27 Thread Maciek Sakrejda
On Fri, Jan 27, 2023, 18:58 Andres Freund wrote: > Hi, > > On 2023-01-27 16:15:08 +1300, Thomas Munro wrote: > > It would be pg_current_xact_id() that would have to pay the cost of > > the WAL flush, not pg_xact_status() itself, but yeah that's what the > > patch does (with some optimisations).

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 16:15:08 +1300, Thomas Munro wrote: > It would be pg_current_xact_id() that would have to pay the cost of > the WAL flush, not pg_xact_status() itself, but yeah that's what the > patch does (with some optimisations). I guess one question is whether > there are any other

Re: bug: copy progress reporting of backends which run multiple COPYs

2023-01-27 Thread Michael Paquier
On Sat, Jan 21, 2023 at 02:45:40AM +0100, Matthias van de Meent wrote: > Let me think about it. I think it would work, but I'm not sure that's > a great option - it adds backend state that we would need to add > cleanup handles for. But then again, COPY ... TO could use TRIGGER to > trigger actual

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Peter Geoghegan
On Fri, Jan 27, 2023 at 6:44 PM Andres Freund wrote: > I don't think that'd make it particularly easy to figure out how often > out-of-space causes non-HOT updates to go out of page, and how often it causes > potential HOT updates to go out of page. If you just have a single index, > it's not

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 17:59:32 -0800, Peter Geoghegan wrote: > > I think this might cause some trouble for existing monitoring setups after > > an > > upgrade. Suddenly the number of updates will appear way lower than > > before... And if we end up eventually distinguishing between different > >

Re: Something is wrong with wal_compression

2023-01-27 Thread Michael Paquier
On Fri, Jan 27, 2023 at 06:06:05AM +0100, Laurenz Albe wrote: > On Fri, 2023-01-27 at 16:15 +1300, Thomas Munro wrote: >> There is no >> doubt that the current situation is unacceptable, though, so maybe we >> really should just do it and make a faster one later.  Anyone else >> want to vote on

Re: Small omission in type_sanity.sql

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 20:39:04 -0500, Tom Lane wrote: > Andres Freund writes: > > Tom, is there a reason we run the various sanity tests early-ish in the > > schedule? It does seem to reduce their effectiveness a bit... > > Originally, those tests were mainly needed to sanity-check the >

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Peter Geoghegan
On Fri, Jan 27, 2023 at 3:55 PM Andres Freund wrote: > I wonder if it's quite detailed enough - we can be forced to do out-of-page > updates because we actually are out of space, or because we reach the max > number of line pointers we allow in a page. HOT pruning can't remove dead line >

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Andres Freund
Hi, On 2023-01-28 14:25:38 +1300, Thomas Munro wrote: > The nearby thread about searching for uses of volatile reminded me: we > can now drop a bunch of these in postmaster.c. The patch I originally > wrote to do that as part of this series somehow morphed into an > experimental patch to nuke

Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: > On Fri, Jan 27, 2023 at 04:23:19PM -0800, Andres Freund wrote: > >> +typedef bool (*RecoveryRestoreCB) (const char *file, const char *path, > >> + const char > >>

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Tom Lane
Thomas Munro writes: > The nearby thread about searching for uses of volatile reminded me: we > can now drop a bunch of these in postmaster.c. The patch I originally > wrote to do that as part of this series somehow morphed into an > experimental patch to nuke all global variables[1], but of

Re: Small omission in type_sanity.sql

2023-01-27 Thread Tom Lane
Andres Freund writes: > Tom, is there a reason we run the various sanity tests early-ish in the > schedule? It does seem to reduce their effectiveness a bit... Originally, those tests were mainly needed to sanity-check the hand-maintained initial catalog data, so it made sense to run them early.

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Thomas Munro
The nearby thread about searching for uses of volatile reminded me: we can now drop a bunch of these in postmaster.c. The patch I originally wrote to do that as part of this series somehow morphed into an experimental patch to nuke all global variables[1], but of course we should at least drop

Re: Small omission in type_sanity.sql

2023-01-27 Thread Andres Freund
Hi, On 2023-01-18 14:51:32 -0500, Melanie Plageman wrote: > I only changed these few lines in type_sanity to be more correct; I > didn't change anything else in regress to actually exercise them (e.g. > ensuring a partitioned table is around when running type_sanity). It > might be worth moving

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 04:23:19PM -0800, Andres Freund wrote: >> +typedef bool (*RecoveryRestoreCB) (const char *file, const char *path, >> + const char >> *lastRestartPointFileName); >> +typedef void (*RecoveryArchiveCleanupCB)

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 18:28:16 -0500, Tom Lane wrote: > Andres Freund writes: > > I'm sure we have a few places that aren't that careful, but I would hope > > it's > > not a large number. Are you thinking of specific "patterns" we've repeated > > all > > over, or just a few cases you recall? > >

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Tomas Vondra
On 1/27/23 22:19, Andres Freund wrote: > Hi, > > On 2023-01-27 12:06:49 +0100, Jakub Wartak wrote: >> On Thu, Jan 26, 2023 at 4:49 PM Andres Freund wrote: >> >>> Huh? Why did you remove the GUC? >> >> After reading previous threads, my optimism level of getting it ever >> in shape of being

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Tomas Vondra
On 1/27/23 22:33, Andres Freund wrote: > Hi, > > On 2023-01-27 21:45:16 +0100, Tomas Vondra wrote: >> On 1/27/23 08:18, Bharath Rupireddy wrote: I think my idea of only forcing to flush/wait an LSN some distance in the past would automatically achieve that? >>> >>> I'm sorry, I

Re: recovery modules

2023-01-27 Thread Michael Paquier
On Fri, Jan 27, 2023 at 04:09:39PM -0800, Andres Freund wrote: > I think it would be hard to write a good module that isn't just implementing > the existing commands without it. Needing to clean up archives and reacting to > the end of recovery seems a pretty core task. FWIW, recovery_end_command

Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-25 16:34:21 +0900, Michael Paquier wrote: > diff --git a/src/include/access/xlogarchive.h > b/src/include/access/xlogarchive.h > index 299304703e..71c9b88165 100644 > --- a/src/include/access/xlogarchive.h > +++ b/src/include/access/xlogarchive.h > @@ -30,9 +30,45 @@ extern bool

Re: How to find the number of cached pages for a relation?

2023-01-27 Thread Amin
Thank you Andres. If I want to do "a" ( Do one probe of the buffer mapping table for each block of the relation. Thus O(#relation blocks)) what function calls can I use, assuming I only have access to the relation id? How can I access and scan the buffer mapping table? On Fri, Jan 13, 2023 at

Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 15:28:21 -0800, Nathan Bossart wrote: > The more I think about this, the more I wonder whether we really need to > include archive_cleanup_command and recovery_end_command in recovery > modules. I think it would be hard to write a good module that isn't just implementing the

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 18:23:39 -0500, Corey Huinker wrote: > This patch adds the n_tup_newpage_upd to all the table stat views. > > Just as we currently track HOT updates, it should be beneficial to track > updates where the new tuple cannot fit on the existing page and must go to > a different one.

Re: recovery modules

2023-01-27 Thread Michael Paquier
On Thu, Jan 26, 2023 at 09:40:58PM -0800, Nathan Bossart wrote: > Yeah, there are some problems here. If we ERROR, we'll just bounce back to > the sigsetjmp() block once a second, and we'll never pick up configuration > reloads, shutdown signals, etc. If we FATAL, we'll just rapidly restart >

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Tom Lane
David Rowley writes: > My personal preference would have been to call it > ScanDirectionCombine, so the naming is more aligned to the 4 other > macro names that start with ScanDirection in sdir.h, but I'm not going > to fuss over it. No objection to that. regards, tom

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread David Rowley
On Sat, 28 Jan 2023 at 12:15, Tom Lane wrote: > /* > * Determine the net effect of two direction specifications. > * This relies on having ForwardScanDirection = +1, BackwardScanDirection = > -1, > * and will probably not do what you want if applied to any other values. > */ > #define

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 09:40:58PM -0800, Nathan Bossart wrote: > On Wed, Jan 25, 2023 at 04:34:21PM +0900, Michael Paquier wrote: >> The loop part is annoying.. I've never been a fan of adding this >> cross-value checks for the archiver modules in the first place, and it >> would make things

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Tom Lane
Andres Freund writes: > On 2023-01-27 15:06:37 -0500, Tom Lane wrote: >> There are a lot of places where we're implicitly relying on >> cross-compilation-unit optimizations NOT happening, because the code isn't >> adequately decorated with memory barriers and the like. > We have a fallback

Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Corey Huinker
This patch adds the n_tup_newpage_upd to all the table stat views. Just as we currently track HOT updates, it should be beneficial to track updates where the new tuple cannot fit on the existing page and must go to a different one. Hopefully this can give users some insight as to whether their

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Tom Lane
Melanie Plageman writes: > On Fri, Jan 27, 2023 at 05:05:16PM -0500, Tom Lane wrote: >> AFAIR, there is noplace today that depends on the exact encoding >> of ForwardScanDirection and BackwardScanDirection, and I'm not >> sure that we want to introduce such a dependency. > I think you mean the

Re: improving user.c error messages

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 07:31:19PM +0100, Alvaro Herrera wrote: > On 2023-Jan-26, Nathan Bossart wrote: >> ereport(ERROR, >> >> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), >> - errmsg("permission denied:

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 15:06:37 -0500, Tom Lane wrote: > There are a lot of places where we're implicitly relying on > cross-compilation-unit optimizations NOT happening, because the code isn't > adequately decorated with memory barriers and the like. We have a fallback compiler barrier

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 10:05:09 -0700, João Paulo Labegalini de Carvalho wrote: > I am investigating the benefits of different profile-guided optimizations > (PGO) and link-time optimizations (LTO) versus binary optimizers (e.g. > BOLT) for applications such as PostgreSQL. > > I am facing issues when

Re: Non-superuser subscription owners

2023-01-27 Thread Mark Dilger
> On Jan 27, 2023, at 1:35 PM, Robert Haas wrote: > >> I started out asking what benefits it provides to own a subscription one >> cannot modify. But I think it is a good capability to have, to restrict the >> set of relations that replication could target. Although perhaps it'd be >> better

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Melanie Plageman
On Fri, Jan 27, 2023 at 05:05:16PM -0500, Tom Lane wrote: > Melanie Plageman writes: > > I have written the patch to remove the unreachable code in > > heapgettup_pagemode](). > > A few thoughts ... > > 1. Do we really need quite so many Asserts? I'd kind of lean > to just having one, at some

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Tom Lane
Melanie Plageman writes: > I have written the patch to remove the unreachable code in > heapgettup_pagemode](). A few thoughts ... 1. Do we really need quite so many Asserts? I'd kind of lean to just having one, at some higher level of the executor. 2. I'm not sure if we want to do this: -

Re: Non-superuser subscription owners

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 16:35:11 -0500, Robert Haas wrote: > > Maybe a daft question: > > > > Have we considered using a "normal grant", e.g. on the database, instead of > > a > > role? Could it e.g. be useful to grant a user the permission to create a > > subscription in one database, but not in

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-27 Thread Melanie Plageman
Hi, I have written the patch to remove the unreachable code in heapgettup_pagemode](). On Wed, Jan 25, 2023 at 10:02 AM Tom Lane wrote: > > I wonder if we couldn't also get rid of this confusingly-inconsistent > alternative usage in the planner: > > * 'indexscandir' is one of: > *

Re: Non-superuser subscription owners

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 4:09 PM Andres Freund wrote: > Hm, compared to postgres_fdw, the user has far less control over what's > happening using that connection. Is there a way a subscription owner can > trigger evaluation of near-arbitrary SQL on the publisher side? I'm not aware of one, but

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 21:45:16 +0100, Tomas Vondra wrote: > On 1/27/23 08:18, Bharath Rupireddy wrote: > >> I think my idea of only forcing to flush/wait an LSN some distance in the > >> past > >> would automatically achieve that? > > > > I'm sorry, I couldn't get your point, can you please explain

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:48:43 +0530, Bharath Rupireddy wrote: > Looking at the patch, the feature, in its current shape, focuses on > improving replication lag (by throttling WAL on the primary) only when > synchronous replication is enabled. Why is that? Why can't we design > it for replication in

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:06:49 +0100, Jakub Wartak wrote: > On Thu, Jan 26, 2023 at 4:49 PM Andres Freund wrote: > > > Huh? Why did you remove the GUC? > > After reading previous threads, my optimism level of getting it ever > in shape of being widely accepted degraded significantly (mainly due >

Re: Non-superuser subscription owners

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 14:42:01 -0500, Robert Haas wrote: > At first, I found it a bit tempting to see this as a further > indication that the force-a-password approach is not the right idea, > because the test case clearly memorializes a desire *not* to require a > password in this situation.

Re: postgres_fdw, dblink, and CREATE SUBSCRIPTION security

2023-01-27 Thread Robert Haas
On Wed, Jan 25, 2023 at 6:22 PM Jacob Champion wrote: > Sure: Ambient authority [1] means that something is granted access based > on some aspect of its existence that it can't remove (or even > necessarily enumerate). Up above, when you said "I cannot choose not to > be myself," that's a clear

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Tomas Vondra
On 1/27/23 08:18, Bharath Rupireddy wrote: > On Thu, Jan 26, 2023 at 9:21 PM Andres Freund wrote: >> >>> 7. I think we need to not let backends throttle too frequently even >>> though they have crossed wal_throttle_threshold bytes. The best way is >>> to rely on replication lag, after all the

Re: GUCs to control abbreviated sort keys

2023-01-27 Thread Jeff Davis
On Fri, 2023-01-27 at 11:41 -0800, Peter Geoghegan wrote: > I cannot recreate the issue you describe. Interesting. For my test: glibc 2.35 ICU 70.1 gcc11.3.0LLVM 14.0.0 > It's not impossible that the perl program you wrote produces > non-deterministic output It is

Re: Authentication fails for md5 connections if ~/.postgresql/postgresql.{crt and key} exist

2023-01-27 Thread Cary Huang
> I think the sslcertmode=disable option that I introduced in [1] solves > this issue too; would it work for your case? That whole patchset is > meant to tackle the general case of the problem you've described. > > (Eventually I'd like to teach the server not to ask for a client >  certificate if

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Tom Lane
=?UTF-8?Q?Jo=C3=A3o_Paulo_Labegalini_de_Carvalho?= writes: > I am facing issues when applying LTO to PostgreSQL as the produced binary > seems broken (the server dies quickly after it has started). This is > definitely a compiler bug, but I was wondering if anyone here have > experimented with

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 3:02 PM Tom Lane wrote: > The string-hacking was fully as tedious as I expected. However, the > output looks pretty nice, and this does have the advantage that the > pre-programmed substitutions become a lot more robust: they are no > longer dependent on the initdb code

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Tom Lane
I wrote: >>> Anyway, it seems like I gotta work harder. I'll produce a >>> new patch. The string-hacking was fully as tedious as I expected. However, the output looks pretty nice, and this does have the advantage that the pre-programmed substitutions become a lot more robust: they are no longer

Re: Show various offset arrays for heap WAL records

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 12:24 PM Melanie Plageman wrote: > I believe I have addressed this in the attached patch. I'm not sure what's best in terms of formatting details but I definitely like the idea of making pg_waldump show more details. I'd even like to have a way to extract the tuple data,

Re: Non-superuser subscription owners

2023-01-27 Thread Robert Haas
On Thu, Jan 19, 2023 at 8:46 PM Andres Freund wrote: > > If we already had (or have) that logic someplace else, it would > > probably make sense to reuse it > > We hve. See at least postgres_fdw's check_conn_params(), dblink's > dblink_connstr_check() and dblink_security_check(). In the patch I

Re: GUCs to control abbreviated sort keys

2023-01-27 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 3:29 PM Jeff Davis wrote: > On Thu, 2023-01-26 at 22:39 +0100, Peter Eisentraut wrote: > > Maybe an easier way to enable or disable it in the source code with a > > #define would serve this. Making it a GUC right away seems a bit > > heavy-handed. Further exploration and

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Komяpa
Hi, We have implemented LTO in PostGIS's build system a couple releases ago. It definitely gives +10% on heavy maths. Unfortunately we did not manage to get it running under FreeBSD because of default system linker issues so we had to hide it under --with-lto switch which we recommend to

Re: improving user.c error messages

2023-01-27 Thread Alvaro Herrera
While we're here, On 2023-Jan-26, Nathan Bossart wrote: > @@ -838,7 +867,8 @@ AlterRole(ParseState *pstate, AlterRoleStmt *stmt) > if (!should_be_super && roleid == BOOTSTRAP_SUPERUSERID) > ereport(ERROR, > >

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Peter Geoghegan
On Fri, Jan 27, 2023 at 12:52 AM Andres Freund wrote: > I agree with bringing high-level context into the decision about whether to > freeze agressively - my problem with the eager freezing strategy patch isn't > that it did that too much, it's that it didn't do it enough. > > > But I also don't

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:53:58 -0500, Robert Haas wrote: > On Fri, Jan 27, 2023 at 12:58 AM Andres Freund wrote: > > Essentially the "any fpi" logic is a very coarse grained way of using the > > page > > LSN as a measurement. As I said, I don't think "has a checkpoint occurred > > since the last

Re: improving user.c error messages

2023-01-27 Thread Alvaro Herrera
On 2023-Jan-27, Tom Lane wrote: > Good point. My vote is for standardizing on *not* mentioning it. > Error messages should say "you need privilege X". That is not > the place to go into all the ways you could hold privilege X > (one of which is being superuser). +1 -- Álvaro Herrera

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 12:58 AM Andres Freund wrote: > Essentially the "any fpi" logic is a very coarse grained way of using the page > LSN as a measurement. As I said, I don't think "has a checkpoint occurred > since the last write" is a good metric to avoid unnecessary freezing - it's > too

Re: Show various offset arrays for heap WAL records

2023-01-27 Thread Melanie Plageman
Hi, I have taken a stab at doing some of the tasks listed in this email. I have made the new files rmgr_utils.c/h. I have come up with a standard format that I like for the output and used it in all the heap record types. Examples below: snapshotConflictHorizon: 2184, nplans: 2, plans [ {

Re: Add LZ4 compression in pg_dump

2023-01-27 Thread Justin Pryzby
On Thu, Jan 26, 2023 at 12:22:45PM -0600, Justin Pryzby wrote: > That commit also added this to pg-dump.c: > > + case PG_COMPRESSION_ZSTD: > + pg_fatal("compression with %s is not yet supported", > "ZSTD"); > + break; > +

Re: Improve GetConfigOptionValues function

2023-01-27 Thread Tom Lane
Nitin Jadhav writes: >> Both of you are arguing as though GUC_NO_SHOW_ALL is a security >> property. It is not, or at least it's so trivially bypassable >> that it's useless to consider it one. All it is is a de-clutter >> mechanism. > Understood. If that is the case, then I am ok with the

Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread João Paulo Labegalini de Carvalho
Hi all, I am investigating the benefits of different profile-guided optimizations (PGO) and link-time optimizations (LTO) versus binary optimizers (e.g. BOLT) for applications such as PostgreSQL. I am facing issues when applying LTO to PostgreSQL as the produced binary seems broken (the server

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-27 Thread Masahiko Sawada
On Wed, Jan 25, 2023 at 3:27 PM Amit Kapila wrote: > > On Wed, Jan 25, 2023 at 10:05 AM Amit Kapila wrote: > > > > On Wed, Jan 25, 2023 at 3:15 AM Peter Smith wrote: > > > > > > 1. > > > @@ -210,7 +210,7 @@ int logical_decoding_work_mem; > > > static const Size max_changes_in_memory = 4096; /*

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Tom Lane
"David G. Johnston" writes: > On Fri, Jan 27, 2023 at 8:53 AM Tom Lane wrote: >> Anyway, it seems like I gotta work harder. I'll produce a >> new patch. > How about just adding a "section" to the end of the file as needed: > # AdHoc Settings Specified During InitDB > max_connections=75 > ...

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Peter Geoghegan
On Fri, Jan 27, 2023 at 6:48 AM Robert Haas wrote: > > One of the key strengths of systems like Postgres is the ability to > > inexpensively store a relatively large amount of data that has just > > about zero chance of being read, let alone modified. While at the same > > time having decent OLTP

Re: Timeline ID hexadecimal format

2023-01-27 Thread Sébastien Lardière
On 27/01/2023 15:55, Peter Eisentraut wrote: On 27.01.23 14:52, Sébastien Lardière wrote: The attached patch proposes to change the format of timelineid from %u to %X. I think your complaint has merit.  But note that if we did a change like this, then log files or reports from different

Re: improving user.c error messages

2023-01-27 Thread Tom Lane
Robert Haas writes: > I almost hate to bring this up since I'm not sure how far we want to > go down this rat hole, but what should be our policy about mentioning > superuser? I don't think we're entirely consistent right now, and I'm > not sure whether every error message needs to mention that

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread David G. Johnston
On Fri, Jan 27, 2023 at 8:53 AM Tom Lane wrote: > Robert Haas writes: > > The idea is that instead of: > > > replace_token(conflines, "#max_connections = 100", repltok); > > > You'd write something like: > > > replace_guc_value(conflines, "max_connections", repltok); > > > Which would look for

Re: improving user.c error messages

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 10:52 AM Nathan Bossart wrote: > IMHO superuser should typically only be mentioned when it is the only way > to do something. Since superusers have all privileges, I think logs like > "superuser or privileges of X" are kind of redundant. If Robert has > privileges of X,

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Tom Lane
Robert Haas writes: > The idea is that instead of: > replace_token(conflines, "#max_connections = 100", repltok); > You'd write something like: > replace_guc_value(conflines, "max_connections", repltok); > Which would look for a line matching /^#max_connections\s+=\s/, and > then identify

Re: improving user.c error messages

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 08:31:32AM -0500, Robert Haas wrote: > I almost hate to bring this up since I'm not sure how far we want to > go down this rat hole, but what should be our policy about mentioning > superuser? I don't think we're entirely consistent right now, and I'm > not sure whether

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 10:34 AM Tom Lane wrote: > One idea if we want to make it work like that could be to stop > trying to edit out the default value, and instead make the file > contents look like, say, > > #huge_pages = try # on, off, or try > huge_pages = off

Re: Named Operators

2023-01-27 Thread Matthias van de Meent
On Fri, 27 Jan 2023 at 16:26, Peter Eisentraut wrote: > > On 12.01.23 14:55, Matthias van de Meent wrote: > >> Matter of taste, I guess. But more importantly, defining an operator > >> gives you many additional features that the planner can use to > >> optimize your query differently, which it

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Tom Lane
Peter Eisentraut writes: > On 25.01.23 22:25, Tom Lane wrote: >> The specified settings >> are applied on the command line of the initial probe calls >> (which happen before we've made any config files), and then they >> are added to postgresql.auto.conf, which causes them to take >> effect for

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Robert Haas
On Wed, Jan 25, 2023 at 4:26 PM Tom Lane wrote: > So this invents an initdb switch "-c NAME=VALUE" just like the > one that the server itself has long had. HUGE +1 from me. This will, I think, be extremely convenient in many situations. > The specified settings > are applied on the command line

Re: Named Operators

2023-01-27 Thread Peter Eisentraut
On 12.01.23 14:55, Matthias van de Meent wrote: Matter of taste, I guess. But more importantly, defining an operator gives you many additional features that the planner can use to optimize your query differently, which it can't do with functions. See the COMMUTATOR, HASHES, etc. clause in the

Re: Set arbitrary GUC options during initdb

2023-01-27 Thread Isaac Morland
On Fri, 27 Jan 2023 at 09:49, Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > On 25.01.23 22:25, Tom Lane wrote: > > So this invents an initdb switch "-c NAME=VALUE" just like the > > one that the server itself has long had. > > This seems useful. > > > The specified settings > >

Re: Lazy JIT IR code generation to increase JIT speed with partitions

2023-01-27 Thread Dmitry Dolgov
> On Fri, Jan 27, 2023 at 10:02:32AM +0100, David Geier wrote: > It's very curious as to why we didn't really see that when dumping the > bitcode. It seems like the bitcode is always different enough to not spot > that. As I've noted off-list, there was noticeable difference in the dumped

Re: old_snapshot_threshold bottleneck on replica

2023-01-27 Thread Robert Haas
On Fri, Jan 27, 2023 at 9:30 AM Maxim Orlov wrote: > I thank you for your advices. I've dived deeper into the problem and I think > v2 patch is wrong. Cool! > Accessing threshold_timestamp and threshold_xid in > TransactionIdLimitedForOldSnapshots > without lock would lead to an improper

  1   2   >