Re: Minimal logical decoding on standbys

2023-01-26 Thread Drouvot, Bertrand
Hi, On 1/26/23 9:13 PM, Andres Freund wrote: Hi, On 2023-01-26 18:56:10 +0100, Drouvot, Bertrand wrote: - I'm struggling to create a test for btree killtuples as there is a need for rows removal on the table (that could produce a conflict too): Do you've a scenario in mind for this one? (and

Re: drop postmaster symlink

2023-01-26 Thread Peter Eisentraut
On 26.01.23 19:36, Karl O. Pinc wrote: I see a possible problem at line 1,412 of runtime.sgml This says: in the postmaster's startup script just before invoking the postmaster. Depending on how this is read, it could be interpreted to mean that a "postmaster" binary is invoked. It might be

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Bharath Rupireddy
On Thu, Jan 26, 2023 at 9:21 PM Andres Freund wrote: > > > 7. I think we need to not let backends throttle too frequently even > > though they have crossed wal_throttle_threshold bytes. The best way is > > to rely on replication lag, after all the goal of this feature is to > > keep replication

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 9:58 PM Andres Freund wrote: > It doesn't seem like a great proxy to me. ISTM that this means that how > aggressive vacuum is about opportunistically freezing pages depends on config > variables like checkpoint_timeout & max_wal_size (less common opportunistic > freezing),

Re: Reducing power consumption on idle servers

2023-01-26 Thread Thomas Munro
On Fri, Jan 27, 2023 at 7:37 PM Bharath Rupireddy wrote: > On Wed, Jan 25, 2023 at 2:10 AM Tom Lane wrote: > > It's kind of moot until we've reached the point where we can > > credibly claim to have explicit wakeups for every event of > > interest. I don't think we're very close to that today,

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-26 Thread Hayato Kuroda (Fujitsu)
I found cfbot failure, PSA fixed version. Sorry for noise. Best Regards, Hayato Kuroda FUJITSU LIMITED v29-0003-add-test.patch Description: v29-0003-add-test.patch v29-0004-add-kqueue-support-for-PQconnCheck-and-PQcanConn.patch Description:

Re: Considering additional sort specialisation functions for PG16

2023-01-26 Thread John Naylor
On Thu, Jan 26, 2023 at 7:15 PM David Rowley wrote: > > On Thu, 26 Jan 2023 at 23:29, John Naylor wrote: > > Coming back to this, I wanted to sketch out this idea in a bit more detail. > > > > Have two memtuple arrays, one for first sortkey null and one for first sortkey non-null: > > - Qsort

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-01-26 Thread Masahiko Sawada
On Fri, Jan 27, 2023 at 3:17 PM Andres Freund wrote: > > Hi, > > On 2023-01-27 14:24:51 +0900, Masahiko Sawada wrote: > > If I'm understanding this result correctly, it seems to me that your > > patch works well with the WAL DIO patch (WALDIO vs. WAL DIO & WAL > > BUFFERS READ), but there seems

Re: Reducing power consumption on idle servers

2023-01-26 Thread Bharath Rupireddy
On Wed, Jan 25, 2023 at 2:10 AM Tom Lane wrote: > > Thomas Munro writes: > > Yeah, I definitely want to fix it. I just worry that 60s is so long > > that it also needs that analysis work to be done to explain that it's > > OK that we're a bit sloppy on noticing when to wake up, at which point >

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-01-26 Thread Andres Freund
Hi, On 2023-01-27 14:24:51 +0900, Masahiko Sawada wrote: > If I'm understanding this result correctly, it seems to me that your > patch works well with the WAL DIO patch (WALDIO vs. WAL DIO & WAL > BUFFERS READ), but there seems no visible performance gain with only > your patch (HEAD vs. WAL

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 19:01:03 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 6:37 PM Andres Freund wrote: > > I also don't really see how that is responsive to anything else in my > > email. That's just as true for the current gating condition (the issuance of > > an FPI during

Re: recovery modules

2023-01-26 Thread Nathan Bossart
On Wed, Jan 25, 2023 at 04:34:21PM +0900, Michael Paquier wrote: > The loop part is annoying.. I've never been a fan of adding this > cross-value checks for the archiver modules in the first place, and it > would make things much simpler in the checkpointer if we need to think > about that as we

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-01-26 Thread Masahiko Sawada
On Thu, Jan 26, 2023 at 2:33 PM Bharath Rupireddy wrote: > > On Thu, Jan 26, 2023 at 2:45 AM Andres Freund wrote: > > > > Hi, > > > > On 2023-01-14 12:34:03 -0800, Andres Freund wrote: > > > On 2023-01-14 00:48:52 -0800, Jeff Davis wrote: > > > > On Mon, 2022-12-26 at 14:20 +0530, Bharath

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-26 Thread Amit Kapila
On Thu, Dec 8, 2022 at 8:17 AM Masahiko Sawada wrote: > > The same assertion failure has been reported on another thread[1]. > Since I could reproduce this issue several times in my environment > I've investigated the root cause. > > I think there is a race condition of updating >

Re: Inconsistency in reporting checkpointer stats

2023-01-26 Thread Bharath Rupireddy
On Wed, Dec 21, 2022 at 5:15 PM Nitin Jadhav wrote: > > I have modified the code accordingly and attached the new version of > patches. patch 0001 fixes the inconsistency in checkpointer stats and > patch 0002 separates main buffer and SLRU buffer count from checkpoint > complete log message.

Re: Something is wrong with wal_compression

2023-01-26 Thread Laurenz Albe
On Fri, 2023-01-27 at 16:15 +1300, Thomas Munro wrote: > On Fri, Jan 27, 2023 at 3:04 PM Tom Lane wrote: > > Thomas Munro writes: > > > On Fri, Jan 27, 2023 at 1:30 PM Michael Paquier > > > wrote: > > > > My opinion would be to make this function more reliable, FWIW, even if > > > > that

RE: Exit walsender before confirming remote flush in logical replication

2023-01-26 Thread Hayato Kuroda (Fujitsu)
Dear Dilip, hackers, Thanks for giving your opinion. I analyzed the relation with the given commit, and I thought I could keep my patch. How do you think? # Abstract * Some modifications should be needed. * We cannot rollback the shutdown if walsenders are stuck * We don't have a good way to

Re: Improve GetConfigOptionValues function

2023-01-26 Thread Bharath Rupireddy
On Tue, Jan 24, 2023 at 8:43 PM Tom Lane wrote: > > Bharath Rupireddy writes: > > On Mon, Jan 23, 2023 at 9:51 PM Tom Lane wrote: > >> Also, I intentionally dropped the GUC_NO_SHOW_ALL check in > >> get_explain_guc_options, because it seems redundant given > >> the preceding GUC_EXPLAIN check.

Re: Add LZ4 compression in pg_dump

2023-01-26 Thread Justin Pryzby
On Mon, Jan 16, 2023 at 11:54:46AM +0900, Michael Paquier wrote: > On Sun, Jan 15, 2023 at 07:56:25PM -0600, Justin Pryzby wrote: > > On Mon, Jan 16, 2023 at 10:28:50AM +0900, Michael Paquier wrote: > >> The functions changed by 0001 are cfopen[_write](), > >> AllocateCompressor() and

RE: [Proposal] Add foreign-server health checks infrastructure

2023-01-26 Thread Hayato Kuroda (Fujitsu)
Dear hackers, I have updated my patch for error handling and kqueue() support. Actually I do not have BSD-like machine, but I developed by using github CICD. I think at first we should focus on 0001-0003, and then work for 0004. Followings are change notes and my analysis. 0001 * Fix missed

Re: Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
Thomas Munro writes: > On Fri, Jan 27, 2023 at 3:04 PM Tom Lane wrote: >> I think we need to get the thing correct first and worry about >> performance later. What's wrong with simply making pg_xact_status >> write and flush a record of the XID's existence before returning it? >> Yeah, it will

Re: Monotonic WindowFunc support for ntile(), percent_rank() and cume_dist()

2023-01-26 Thread David Rowley
On Wed, 25 Jan 2023 at 02:39, Melanie Plageman wrote: > > On Tue, Jan 24, 2023 at 02:00:33PM +1300, David Rowley wrote: > > If you feel strongly about that, then feel free to show me what you > > have in mind in more detail so I can think harder about it. > > Nah, I don't feel strongly. I think

Re: Something is wrong with wal_compression

2023-01-26 Thread Thomas Munro
On Fri, Jan 27, 2023 at 3:04 PM Tom Lane wrote: > Thomas Munro writes: > > On Fri, Jan 27, 2023 at 1:30 PM Michael Paquier wrote: > >> My opinion would be to make this function more reliable, FWIW, even if > >> that involves a performance impact when called in a close loop by > >> forcing more

Re: Generating code for query jumbling through gen_node_support.pl

2023-01-26 Thread Michael Paquier
On Thu, Jan 26, 2023 at 09:39:05AM +0100, Peter Eisentraut wrote: > There are a couple of repetitive comments, like "typmod and collation > information are irrelevant for the query jumbling". This applies to all > nodes, so we don't need to repeat it for a number of nodes (and then not > mention

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 6:37 PM Andres Freund wrote: > I also don't really see how that is responsive to anything else in my > email. That's just as true for the current gating condition (the issuance of > an FPI during heap_page_prune() / HTSV()). > > What I was wondering about is whether we

Re: Generating code for query jumbling through gen_node_support.pl

2023-01-26 Thread Michael Paquier
On Thu, Jan 26, 2023 at 09:37:13AM +0100, Peter Eisentraut wrote: > Ok, the documentation make sense now. I wonder what the performance impact > is. Probably, nobody cares about microoptimizing CREATE TABLE statements. > But BEGIN/COMMIT could matter. However, whatever you do in between the >

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 15:36:52 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 12:45 PM Andres Freund wrote: > > > Most of the overhead of FREEZE WAL records (with freeze plan > > > deduplication and page-level freezing in) is generic WAL record header > > > overhead. Your recent adversarial

Re: Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
Thomas Munro writes: > On Fri, Jan 27, 2023 at 1:30 PM Michael Paquier wrote: >> My opinion would be to make this function more reliable, FWIW, even if >> that involves a performance impact when called in a close loop by >> forcing more WAL flushes to ensure its report durability and >>

Re: wrong Append/MergeAppend elision?

2023-01-26 Thread Amit Langote
On Fri, Jan 27, 2023 at 5:43 AM Tom Lane wrote: > David Rowley writes: > > On Fri, 27 Jan 2023 at 01:30, Amit Langote wrote: > >> It seems that the planner currently elides an Append/MergeAppend that > >> has run-time pruning info (part_prune_index) set, but which I think is > >> a bug. > > >

Re: Add LZ4 compression in pg_dump

2023-01-26 Thread Michael Paquier
On Thu, Jan 26, 2023 at 12:22:45PM -0600, Justin Pryzby wrote: > Yeah. But the original log_warning text was better, and should be > restored: > > - if (AH->compression != 0) > - pg_log_warning("archive is compressed, but this installation > does not support compression --

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 14:27:53 -0500, Robert Haas wrote: > One idea that I've had about how to solve this problem is to try to > make vacuum try to aggressively freeze some portion of the table on > each pass, and to behave less aggressively on the rest of the table so > that, hopefully, no single

Re: Something is wrong with wal_compression

2023-01-26 Thread Thomas Munro
On Fri, Jan 27, 2023 at 1:30 PM Michael Paquier wrote: > On Thu, Jan 26, 2023 at 04:14:57PM -0800, Andrey Borodin wrote: > > If we agree that xid allocation is not something persistent, let's fix > > the test? We can replace a check with select * from pg_class or, > > maybe, add an amcheck run. >

Re: Record queryid when auto_explain.log_verbose is on

2023-01-26 Thread Michael Paquier
On Thu, Jan 26, 2023 at 10:00:04PM +0900, torikoshia wrote: > I'll work on this next. Cool, thanks! -- Michael signature.asc Description: PGP signature

Re: Something is wrong with wal_compression

2023-01-26 Thread Michael Paquier
On Thu, Jan 26, 2023 at 04:14:57PM -0800, Andrey Borodin wrote: > If we agree that xid allocation is not something persistent, let's fix > the test? We can replace a check with select * from pg_class or, > maybe, add an amcheck run. > As far as I recollect, this test was introduced to test this

Re: Partition key causes problem for volatile target list query

2023-01-26 Thread Bruce Momjian
On Thu, Jan 26, 2023 at 07:21:16PM -0500, Tom Lane wrote: > Well, if you looked further than the first few rows, it wouldn't be > "always zero". But the select from the partitioned table will read > the first partition first, and that partition will have the rows > with d1=0, by definition. > >

Re: Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
Andrey Borodin writes: > On Thu, Jan 26, 2023 at 3:04 PM Tom Lane wrote: >> Indeed, it seems like this behavior makes pg_xact_status() basically >> useless as things stand. > If we agree that xid allocation is not something persistent, let's fix > the test? If we're not going to fix this

Re: Partition key causes problem for volatile target list query

2023-01-26 Thread Tom Lane
Bruce Momjian writes: > I have found an odd behavior --- a query in the target list that assigns > to a partitioned column causes queries that would normally be volatile > to return always zero. Well, if you looked further than the first few rows, it wouldn't be "always zero". But the select

Re: Something is wrong with wal_compression

2023-01-26 Thread Andrey Borodin
On Thu, Jan 26, 2023 at 3:04 PM Tom Lane wrote: > > Indeed, it seems like this behavior makes pg_xact_status() basically > useless as things stand. > If we agree that xid allocation is not something persistent, let's fix the test? We can replace a check with select * from pg_class or, maybe, add

Re: improving user.c error messages

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 05:41:32PM -0500, Tom Lane wrote: > Well, it's not a hint. I think the above is fine for non-password > cases, but for passwords maybe > > ERROR: permission denied to alter role password > DETAIL: To change another role's password, you must have CREATEROLE

Partition key causes problem for volatile target list query

2023-01-26 Thread Bruce Momjian
I have found an odd behavior --- a query in the target list that assigns to a partitioned column causes queries that would normally be volatile to return always zero. In the first query, no partitioning is used: d1 | d2 + 1 | 0 2 | 0 2 |

Re: Rework of collation code, extensibility

2023-01-26 Thread Jeff Davis
Attached v9 and added perf numbers below. I'm hoping to commit 0002 and 0003 soon-ish, maybe a week or two, please let me know if you want me to hold off. (I won't commit the GUCs unless others find them generally useful; they are included here to more easily reproduce my performance tests.) My

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 12:45 PM Andres Freund wrote: > > Most of the overhead of FREEZE WAL records (with freeze plan > > deduplication and page-level freezing in) is generic WAL record header > > overhead. Your recent adversarial test case is going to choke on that, > > too. At least if you set

Re: GUCs to control abbreviated sort keys

2023-01-26 Thread Jeff Davis
On Thu, 2023-01-26 at 22:39 +0100, Peter Eisentraut wrote: > Maybe an easier way to enable or disable it in the source code with a > #define would serve this.  Making it a GUC right away seems a bit > heavy-handed.  Further exploration and tweaking might well require > further source code

Re: Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
Thomas Munro writes: > On Fri, Jan 27, 2023 at 11:14 AM Tom Lane wrote: >> If any tuples made by that transaction had reached disk, >> we'd have a problem. > The problem is that the WAL wasn't flushed, allowing the same xid to > be allocated again after crash recovery. But for any data pages

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Jelte Fennema
On Thu, 26 Jan 2023 at 22:46, Andrew Dunstan wrote: > Hmm, but I usually run with -a, I even have a git alias for it. I guess > what this discussion illustrates is that there are various patters for > using git, and we shouldn't assume that everyone else is using the same > patterns we are. I

Re: Something is wrong with wal_compression

2023-01-26 Thread Thomas Munro
On Fri, Jan 27, 2023 at 11:14 AM Tom Lane wrote: > Andrey Borodin writes: > > On Thu, Jan 26, 2023 at 12:12 PM Tom Lane wrote: > >> That test case is demonstrating fundamental > >> database corruption after a crash. > > > Not exactly corruption. XID was not persisted and buffer data did not > >

Re: improving user.c error messages

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 03:07:43PM -0500, Tom Lane wrote: >> I think the password case needs to be kept separate, because the >> conditions for it are different (specifically the exception that >> you can alter your own password). Lumping the rest together >> seems OK to

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 04:09:51PM -0500, Tom Lane wrote: >> Right, so more like this. > LGTM Thanks, pushed. Returning to the prior patch ... I don't much care for this: +/* Maybe there will be a free slot in a second... */ +

Re: Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
Andrey Borodin writes: > On Thu, Jan 26, 2023 at 12:12 PM Tom Lane wrote: >> That test case is demonstrating fundamental >> database corruption after a crash. > Not exactly corruption. XID was not persisted and buffer data did not > hit a disk. Database is in the correct state. Really? I

Re: improving user.c error messages

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 03:07:43PM -0500, Tom Lane wrote: > Nathan Bossart writes: >> On Thu, Jan 26, 2023 at 02:42:05PM -0500, Robert Haas wrote: >>> Basically my question is whether having one error message for all of >>> those cases is good enough, or whether we should be trying harder. > > I

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 1:22 PM Robert Haas wrote: > On Thu, Jan 26, 2023 at 4:06 PM Peter Geoghegan wrote: > > There is very good reason to believe that the large majority of all > > data that people store in a system like Postgres is extremely cold > > data: > > The systems where I end up

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Andrew Dunstan
On 2023-01-26 Th 12:05, Jelte Fennema wrote: >> At this stage the files are now indented, so if it failed and you run >> `git commit` again it will commit with the indention changes. > No, because at no point a "git add" is happening, so the changes > made by pgindent are not staged. As long as

Re: GUCs to control abbreviated sort keys

2023-01-26 Thread Peter Eisentraut
On 25.01.23 22:16, Jeff Davis wrote: I am highlighting this case because the existence of a single non- contrived case or regression suggests that we may want to explore further and tweak heuristics. That's quite natural when the heuristics are based on a complex dependency like a collation

Re: Something is wrong with wal_compression

2023-01-26 Thread Andrey Borodin
On Thu, Jan 26, 2023 at 12:12 PM Tom Lane wrote: > > That test case is demonstrating fundamental > database corruption after a crash. > Not exactly corruption. XID was not persisted and buffer data did not hit a disk. Database is in the correct state. It was discussed long before WAL

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 04:09:51PM -0500, Tom Lane wrote: > Right, so more like this. LGTM -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 4:06 PM Peter Geoghegan wrote: > There is very good reason to believe that the large majority of all > data that people store in a system like Postgres is extremely cold > data: The systems where I end up troubleshooting problems seem to be, most typically, busy OLTP

Re: Cygwin cleanup

2023-01-26 Thread Justin Pryzby
Note that cirrus failed like this: https://api.cirrus-ci.com/v1/artifact/task/4881596411543552/testrun/build/testrun/subscription/010_truncate/log/010_truncate_publisher.log 2023-01-25 23:17:10.417 GMT [29821][walsender] [sub1][3/0:0] ERROR: could not open file

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 03:04:30PM -0500, Tom Lane wrote: >> Hmm. I'm disinclined to add an assumption that the epoch is in the past, >> but I take your point that the subtraction would overflow with >> TIMESTAMP_INFINITY and a negative finite timestamp. Maybe we should

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-26 Thread Thomas Munro
On Fri, Jan 27, 2023 at 9:57 AM Thomas Munro wrote: > On Fri, Jan 27, 2023 at 9:49 AM Tom Lane wrote: > > Tomas Vondra writes: > > > I received an alert dikkop (my rpi4 buildfarm animal running freebsd 14) > > > did not report any results for a couple days, and it seems it got into > > > an

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 12:54 PM Robert Haas wrote: > > The overwhelming cost is usually FPIs in any case. If you're not > > mostly focussing on that, you're focussing on the wrong thing. At > > least with larger tables. You just have to focus on the picture over > > time, across multiple VACUUM

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-26 Thread Thomas Munro
On Fri, Jan 27, 2023 at 9:49 AM Tom Lane wrote: > Tomas Vondra writes: > > I received an alert dikkop (my rpi4 buildfarm animal running freebsd 14) > > did not report any results for a couple days, and it seems it got into > > an infinite loop in REL_11_STABLE when building hash table in a

Re: Non-superuser subscription owners

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 12:36 PM Jeff Davis wrote: > On Thu, 2023-01-26 at 09:43 -0500, Robert Haas wrote: > > I have no issue with that as a long-term plan. However, I think that > > for right now we should just introduce pg_create_subscription. It > > would make sense to add

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 20:26:00 +0100, Matthias van de Meent wrote: > Could someone explain to me why we don't currently (optionally) > include the functionality of page freezing in the PRUNE records? I think we definitely should (and have argued for it a couple times). It's not just about reducing

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 2:57 PM Peter Geoghegan wrote: > Relatively difficult for Andres, or for somebody else? What are the > real parameters here? Obviously there are no clear answers available. Andres is certainly smarter than the average guy, but practically any scenario that someone can

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-26 Thread Tom Lane
Tomas Vondra writes: > I received an alert dikkop (my rpi4 buildfarm animal running freebsd 14) > did not report any results for a couple days, and it seems it got into > an infinite loop in REL_11_STABLE when building hash table in a parallel > hashjoin, or something like that. > It seems to be

Re: doc: add missing "id" attributes to extension packaging page

2023-01-26 Thread Brar Piening
On 18.01.2023 at 06:50, Brar Piening wrote: I'll give it a proper look this weekend. It turns out I didn't get a round tuit. ... and I'm afraid I probably will not be able to work on this until mid/end February so we'll have to move this to the next commitfest until somebody wants to take it

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 10:44:45 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 9:53 AM Andres Freund wrote: > > > That's going to be very significantly more aggressive. For example > > > it'll impact small tables very differently. > > > > Maybe it would be too aggressive, not sure. The cost

Re: wrong Append/MergeAppend elision?

2023-01-26 Thread Tom Lane
David Rowley writes: > On Fri, 27 Jan 2023 at 01:30, Amit Langote wrote: >> It seems that the planner currently elides an Append/MergeAppend that >> has run-time pruning info (part_prune_index) set, but which I think is >> a bug. > There is still the trade-off of having to pull tuples through

lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-26 Thread Tomas Vondra
Hi, I received an alert dikkop (my rpi4 buildfarm animal running freebsd 14) did not report any results for a couple days, and it seems it got into an infinite loop in REL_11_STABLE when building hash table in a parallel hashjoin, or something like that. It seems to be progressing now, probably

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 11:26 AM Matthias van de Meent wrote: > Could someone explain to me why we don't currently (optionally) > include the functionality of page freezing in the PRUNE records? I > think they're quite closely related (in that they both execute in > VACUUM and are required for

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-26 Thread Reid Thompson
On Mon, 2023-01-23 at 12:31 -0800, Andres Freund wrote: > Hi, > > I think it's basically still waiting on author, until the O(N) cost is gone > from the overflow limit check. > > Greetings, > > Andres Freund Yes, just a rebase. There is still work to be done per earlier in the thread. I do

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 03:04:30PM -0500, Tom Lane wrote: > Nathan Bossart writes: >> I wonder if we should explicitly reject negative timestamps to eliminate >> any chance of int64 overflow, too. > > Hmm. I'm disinclined to add an assumption that the epoch is in the past, > but I take your

Re: Something is wrong with wal_compression

2023-01-26 Thread Justin Pryzby
On Thu, Jan 26, 2023 at 02:08:27PM -0600, Justin Pryzby wrote: > On Thu, Jan 26, 2023 at 02:43:29PM -0500, Tom Lane wrote: > > The symptom being exhibited by Michael's new BF animal tanager > > is perfectly reproducible elsewhere. > > I think these tests have always failed with wal_compression ?

Re: Minimal logical decoding on standbys

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 18:56:10 +0100, Drouvot, Bertrand wrote: > - I'm struggling to create a test for btree killtuples as there is a need for > rows removal on the table (that could produce a conflict too): > Do you've a scenario in mind for this one? (and btw in what kind of WAL > record should

Re: wrong Append/MergeAppend elision?

2023-01-26 Thread David Rowley
On Fri, 27 Jan 2023 at 01:30, Amit Langote wrote: > It seems that the planner currently elides an Append/MergeAppend that > has run-time pruning info (part_prune_index) set, but which I think is > a bug. This is actually how I intended it to work. Whether it was a good idea or not, I'm currently

Re: Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
Justin Pryzby writes: > On Thu, Jan 26, 2023 at 02:43:29PM -0500, Tom Lane wrote: >> The symptom being exhibited by Michael's new BF animal tanager >> is perfectly reproducible elsewhere. > I think these tests have always failed with wal_compression ? If that's a known problem, and we've done

Re: Something is wrong with wal_compression

2023-01-26 Thread Justin Pryzby
On Thu, Jan 26, 2023 at 02:43:29PM -0500, Tom Lane wrote: > The symptom being exhibited by Michael's new BF animal tanager > is perfectly reproducible elsewhere. I think these tests have always failed with wal_compression ?

Re: improving user.c error messages

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 02:42:05PM -0500, Robert Haas wrote: >> Basically my question is whether having one error message for all of >> those cases is good enough, or whether we should be trying harder. I think the password case needs to be kept separate, because the

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Tom Lane
Nathan Bossart writes: > On Thu, Jan 26, 2023 at 01:54:08PM -0500, Tom Lane wrote: >> - * Both inputs must be ordinary finite timestamps (in current usage, >> - * they'll be results from GetCurrentTimestamp()). >> + * At least one input must be an ordinary finite timestamp, else the "diff" >> + *

Re: improving user.c error messages

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 02:42:05PM -0500, Robert Haas wrote: > @@ -758,16 +776,13 @@ AlterRole(ParseState *pstate, AlterRoleStmt *stmt) > { > /* things an unprivileged user certainly can't do */ > if (dinherit || dcreaterole || dcreatedb || dcanlogin || dconnlimit || > - dvalidUntil ||

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 11:28 AM Robert Haas wrote: > I think it's pretty much impossible to freeze more aggressively > without losing in some scenario or other. If waiting longer to freeze > would have resulted in the data getting updated again or deleted > before we froze it, then waiting

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Nathan Bossart
On Thu, Jan 26, 2023 at 01:54:08PM -0500, Tom Lane wrote: > After looking closer, I see that TimestampDifferenceMilliseconds > already explicitly states that its output is intended for WaitLatch > and friends, which makes it perfectly sane for it to clamp the result > to [0, INT_MAX] rather than

Something is wrong with wal_compression

2023-01-26 Thread Tom Lane
The symptom being exhibited by Michael's new BF animal tanager is perfectly reproducible elsewhere. $ cat /home/postgres/tmp/temp_config #default_toast_compression = lz4 wal_compression = lz4 $ export TEMP_CONFIG=/home/postgres/tmp/temp_config $ cd ~/pgsql/src/test/recovery $ make check

Re: Helper functions for wait_for_catchup() in Cluster.pm

2023-01-26 Thread Drouvot, Bertrand
Hi, On 1/26/23 10:42 AM, Alvaro Herrera wrote: On 2023-Jan-26, Drouvot, Bertrand wrote: On 1/24/23 7:27 PM, Alvaro Herrera wrote: 1. I don't think wait_for_write_catchup is necessary, because calling wait_for_catchup() and omitting the 'mode' and 'lsn' arguments would already do the same

Re: improving user.c error messages

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 2:14 PM Nathan Bossart wrote: > Yeah, it's probably better to say "to alter roles with %s" to refer to > roles that presently have the attribute and "to change the %s attribute" > when referring to privileges for the attribute. I did this in v2, too. > > I've also

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Robert Haas
On Thu, Jan 26, 2023 at 11:35 AM Peter Geoghegan wrote: > You complained about the descriptions being theoretical. But there's > nothing theoretical about the fact that we more or less do *all* > freezing in an eventual aggressive VACUUM in many important cases, > including very simple cases like

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Matthias van de Meent
On Thu, 26 Jan 2023 at 19:45, Peter Geoghegan wrote: > > On Thu, Jan 26, 2023 at 9:53 AM Andres Freund wrote: > > I assume the case you're thinking of is that pruning did *not* do any > > changes, > > but in the process of figuring out that nothing needed to be pruned, we did > > a > >

Re: improving user.c error messages

2023-01-26 Thread Nathan Bossart
Thanks for taking a look. On Thu, Jan 26, 2023 at 10:07:39AM +0100, Alvaro Herrera wrote: > Please use > errdetail("You must have %s privilege to create roles with %s.", > "SUPERUSER", "SUPERUSER"))); > > in this kind of message where multiple copies appear

Re: suppressing useless wakeups in logical/worker.c

2023-01-26 Thread Tom Lane
I wrote: >>> It'd probably be reasonable to file down that sharp edge by instead >>> specifying that TimestampDifferenceMilliseconds will clamp overflowing >>> differences to LONG_MAX. Maybe there should be a clamp on the underflow >>> side too ... but should it be to LONG_MIN or to zero? After

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Tomas Vondra
On 1/26/23 16:40, Andres Freund wrote: > Hi, > > On 2023-01-26 12:08:16 +0100, Tomas Vondra wrote: >> It's not clear to me how could it cause deadlocks, as we're not waiting >> for a lock/resource locked by someone else, but it's certainly an issue >> for uninterruptible hangs. > > Maybe not.

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 9:53 AM Andres Freund wrote: > I assume the case you're thinking of is that pruning did *not* do any changes, > but in the process of figuring out that nothing needed to be pruned, we did a > MarkBufferDirtyHint(), and as part of that emitted an FPI? Yes. > > That's

Re: drop postmaster symlink

2023-01-26 Thread Karl O. Pinc
On Wed, 25 Jan 2023 18:03:25 -0600 "Karl O. Pinc" Buried in > https://www.postgresql.org/message-id/20230107165942.748ccf4e%40slate.karlpinc.com > is the one change I see that should be made. > > > In doc/src/sgml/ref/allfiles.sgml at line 222 there is an ENTITY > > defined which references the

Re: Add LZ4 compression in pg_dump

2023-01-26 Thread Justin Pryzby
On Wed, Jan 25, 2023 at 07:57:18PM +, gkokola...@pm.me wrote: > On Wednesday, January 25th, 2023 at 7:00 PM, Justin Pryzby > wrote: > > While looking at this, I realized that commit 5e73a6048 introduced a > > regression: > > > > @@ -3740,19 +3762,24 @@ ReadHead(ArchiveHandle *AH) > > > > -

Re: meson oddities

2023-01-26 Thread Andres Freund
On 2023-01-26 10:20:58 +0100, Peter Eisentraut wrote: > On 19.01.23 21:45, Andres Freund wrote: > > Hi, > > > > On 2023-01-19 21:37:15 +0100, Peter Eisentraut wrote: > > > On 11.01.23 12:05, Peter Eisentraut wrote: > > > > I think there is also an adjacent issue:  The subdir options may be > > >

Re: fix and document CLUSTER privileges

2023-01-26 Thread Nathan Bossart
On Wed, Jan 25, 2023 at 08:27:57PM -0800, Jeff Davis wrote: > Committed these extra clarifications. Thank you. Thanks! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 08:54:55 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 8:35 AM Andres Freund wrote: > > I think it's probably ok, but perhaps deserves a bit more thought about when > > to "opportunistically" freeze. Perhaps to make it *more* aggressive than > > it's > > now. > > > >

Re: Non-superuser subscription owners

2023-01-26 Thread Jeff Davis
On Thu, 2023-01-26 at 09:43 -0500, Robert Haas wrote: > I have no issue with that as a long-term plan. However, I think that > for right now we should just introduce pg_create_subscription. It > would make sense to add pg_create_connection in the same patch that > adds a CREATE CONNECTION command

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Jelte Fennema
> At this stage the files are now indented, so if it failed and you run > `git commit` again it will commit with the indention changes. No, because at no point a "git add" is happening, so the changes made by pgindent are not staged. As long as you don't run the second "git commit" with the -a

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Peter Geoghegan
On Thu, Jan 26, 2023 at 8:35 AM Andres Freund wrote: > I think it's probably ok, but perhaps deserves a bit more thought about when > to "opportunistically" freeze. Perhaps to make it *more* aggressive than it's > now. > > With "opportunistic freezing" I mean freezing the page, even though we

Re: run pgindent on a regular basis / scripted manner

2023-01-26 Thread Andrew Dunstan
On 2023-01-26 Th 11:16, Jelte Fennema wrote: > On Thu, 26 Jan 2023 at 15:40, Andrew Dunstan wrote: >> I didn't really like your hook, as it forces a reindent, and many people >> won't want that (for reasons given elsewhere in this thread). > I'm not sure what you mean by "forces a reindent".

  1   2   >