Re: cutting down the TODO list thread

2023-05-12 Thread Tom Lane
John Naylor writes: > I've made another pass at this. Previously, I went one or two sections at a > time, but this time I tried just skimming the whole thing and noting what > jumps out at me. Also, I've separated things into three categories: Remove, > move to "not wanted list", and revise. Comme

Re: cutting down the TODO list thread

2023-05-12 Thread John Naylor
On Mon, Feb 6, 2023 at 11:04 AM John Naylor wrote: > I'll try to get back to culling the list items at the end of April. I've made another pass at this. Previously, I went one or two sections at a time, but this time I tried just skimming the whole thing and noting what jumps out at me. Also, I'v

Re: [PATCH] Clarify the behavior of the system when approaching XID wraparound

2023-05-12 Thread John Naylor
Attached is v9, which is mostly editing the steps for restoring normal operation, which are in 0003 now but will be squashed into 0002. Comments to polish the wording welcome. -- John Naylor EDB: http://www.enterprisedb.com From 0e9d6ea72216b196d37de4629736c0cf34e7cd5c Mon Sep 17 00:00:00 2001 Fr

Re: [PATCH] Use RelationClose rather than table_close in heap_create_with_catalog

2023-05-12 Thread Xiaoran Wang
Thanks for all your responses. It seems better to change the comments on the code rather than call RelationClose here. table_close(new_rel_desc, NoLock); /* do not unlock till end of xact */ Do I need to create another patch to fix the comments? Best regards, xiaoran

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-12 Thread Peter Geoghegan
On Fri, May 12, 2023 at 10:36 AM Ryan Booz wrote: > Just to say on the outset, as has been said earlier in the tread by others, > that this is herculean work. Thank you for putting the effort you have thus > far. Thanks! > > It would be nice if it was possible to add an animation/diagram a > >

Re: Large files for relations

2023-05-12 Thread Thomas Munro
On Sat, May 13, 2023 at 11:01 AM Thomas Munro wrote: > On Sat, May 13, 2023 at 4:41 AM MARK CALLAGHAN wrote: > > use XFS and O_DIRECT As for direct I/O, we're only just getting started on that. We currently can't produce more than one concurrent WAL write, and then for relation data, we just go

Re: [PATCHES] Post-special page storage TDE support

2023-05-12 Thread Stephen Frost
Greetings, * David Christensen (david.christen...@crunchydata.com) wrote: > Refreshing this with HEAD as of today, v4. Thanks for updating this! > Subject: [PATCH v4 1/3] Add reserved_page_space to Page structure > > This space is reserved for extended data on the Page structure which will be

Re: smgrzeroextend clarification

2023-05-12 Thread Andres Freund
Hi, On May 12, 2023 11:36:23 AM PDT, Thomas Munro wrote: >Just a thought: should RelationCopyStorageUsingBuffer(), the new code >used by CREATE DATABASE with the default strategy WAL_LOG, use the >newer interface so that it creates fully allocated files instead of >sparse ones? I played with th

Re: Large files for relations

2023-05-12 Thread Thomas Munro
On Sat, May 13, 2023 at 4:41 AM MARK CALLAGHAN wrote: > Repeating what was mentioned on Twitter, because I had some experience with > the topic. With fewer files per table there will be more contention on the > per-inode mutex (which might now be the per-inode rwsem). I haven't read > filesyste

Re: WAL Insertion Lock Improvements

2023-05-12 Thread Michael Paquier
On Fri, May 12, 2023 at 07:35:20AM +0530, Bharath Rupireddy wrote: > --enable-atomics=no, -T60: > --enable-spinlocks=no, -T60: > --enable-atomics=no --enable-spinlocks=no, -T60: Thanks for these extra tests, I have not done these specific cases but the profiles look similar to what I've seen mysel

Re: smgrzeroextend clarification

2023-05-12 Thread Andres Freund
Hi, On May 11, 2023 2:37:00 AM PDT, Peter Eisentraut wrote: >On 10.05.23 20:10, Andres Freund wrote: >>> Moreover, the text "except the relation can be extended by multiple blocks >>> at once and the added blocks will be filled with zeroes" doesn't make much >>> sense as a differentiation, beca

Re: Memory leak from ExecutorState context?

2023-05-12 Thread Melanie Plageman
Thanks for continuing to work on this. Are you planning to modify what is displayed for memory usage in EXPLAIN ANALYZE? Also, since that won't help a user who OOMs, I wondered if the spillCxt is helpful on its own or if we need some kind of logging message for users to discover that this is what

Re: improve more permissions-related error messages

2023-05-12 Thread Nathan Bossart
On Fri, May 12, 2023 at 04:43:08PM -0400, Tom Lane wrote: > Nathan Bossart writes: >> While looking around for other such error messages, I found a few dozen >> "must be superuser" errors that might be improved with the new style. If >> folks feel this is worthwhile, I'll put together a patch. >

Re: improve more permissions-related error messages

2023-05-12 Thread Tom Lane
Nathan Bossart writes: > This is intended as a follow-up to de4d456 [0]. I noticed that c3afe8c > introduced another "must have privileges" error message that I think should > be adjusted to use the new style introduced in de4d456. І've attached a > small patch for this. +1 > While looking aro

improve more permissions-related error messages

2023-05-12 Thread Nathan Bossart
This is intended as a follow-up to de4d456 [0]. I noticed that c3afe8c introduced another "must have privileges" error message that I think should be adjusted to use the new style introduced in de4d456. І've attached a small patch for this. While looking around for other such error messages, I f

Re: Should CSV parsing be stricter about mid-field quotes?

2023-05-12 Thread Andrew Dunstan
On 2023-05-11 Th 10:03, Joel Jacobson wrote: Hi hackers, I've come across an unexpected behavior in our CSV parser that I'd like to bring up for discussion. % cat example.csv id,rating,review 1,5,"Great product, will buy again." 2,3,"I bought this for my 6" laptop but it didn't fit my 8" table

Re: psql tests hangs

2023-05-12 Thread Pavel Stehule
pá 12. 5. 2023 v 21:08 odesílatel Tom Lane napsal: > I wrote: > > Pavel Stehule writes: > >> Except for testing, using pager in non-interactive mode makes no sense. > > > Agreed. Let's solve this by inserting isatty tests in psql, rather > > than hacking the test environment. > > Here's a propo

Re: psql tests hangs

2023-05-12 Thread Tom Lane
I wrote: > Pavel Stehule writes: >> Except for testing, using pager in non-interactive mode makes no sense. > Agreed. Let's solve this by inserting isatty tests in psql, rather > than hacking the test environment. Here's a proposed patch for this. I noticed that another memo the PSQL_WATCH_PAG

Re: Should CSV parsing be stricter about mid-field quotes?

2023-05-12 Thread Greg Stark
On Thu, 11 May 2023 at 10:04, Joel Jacobson wrote: > > The parser currently accepts quoting within an unquoted field. This can lead > to > data misinterpretation when the quote is part of the field data (e.g., > for inches, like in the example). I think you're thinking about it differently than

Re: smgrzeroextend clarification

2023-05-12 Thread Thomas Munro
On Sat, May 13, 2023 at 6:07 AM Greg Stark wrote: > On Thu, 11 May 2023 at 05:37, Peter Eisentraut > wrote: > > Maybe it was never meant that way and only works accidentally? Maybe > > hash indexes are broken? > > It's explicitly documented to be this way. And I think it has to work > this way f

Re: smgrzeroextend clarification

2023-05-12 Thread Greg Stark
On Thu, 11 May 2023 at 05:37, Peter Eisentraut wrote: > > Maybe it was never meant that way and only works accidentally? Maybe > hash indexes are broken? It's explicitly documented to be this way. And I think it has to work this way for recovery to work. I think the reason you and Bharath and A

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-12 Thread Ryan Booz
And, of course, I forgot that I switch to text-mode after writing most of this, so the carriage returns were unnecessary. (facepalm... sigh) -- Ryan On Fri, May 12, 2023 at 1:36 PM Ryan Booz wrote: > > Thanks for the continued work, Peter. I hate to be the guy that starts this > way, > but this

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

2023-05-12 Thread Ryan Booz
Thanks for the continued work, Peter. I hate to be the guy that starts this way, but this is my first ever response on pgsql-hackers. (insert awkward smile face). Hopefully I've followed etiquette well, but please forgive any missteps, and I'm happy for any help in making better contributions in th

Re: psql tests hangs

2023-05-12 Thread Tom Lane
Pavel Stehule writes: > pá 12. 5. 2023 v 17:50 odesílatel Tom Lane napsal: >> OK. So after looking at this a bit, the reason PAGER and PSQL_PAGER >> don't cause us any problems in the test environment is that they are >> not honored unless isatty(fileno(stdin)) && isatty(fileno(stdout)). >> It s

Re: Large files for relations

2023-05-12 Thread MARK CALLAGHAN
Repeating what was mentioned on Twitter, because I had some experience with the topic. With fewer files per table there will be more contention on the per-inode mutex (which might now be the per-inode rwsem). I haven't read filesystem source in a long time. Back in the day, and perhaps today, it wa

Re: psql tests hangs

2023-05-12 Thread Pavel Stehule
pá 12. 5. 2023 v 17:50 odesílatel Tom Lane napsal: > Pavel Stehule writes: > > pá 12. 5. 2023 v 15:26 odesílatel Tom Lane napsal: > >> Oh! AFAICS, we don't do any of those things, but I agree it seems like > >> a good idea. Can you confirm that if you unset PSQL_WATCH_PAGER then > >> the test

Re: psql tests hangs

2023-05-12 Thread Tom Lane
Pavel Stehule writes: > pá 12. 5. 2023 v 15:26 odesílatel Tom Lane napsal: >> Oh! AFAICS, we don't do any of those things, but I agree it seems like >> a good idea. Can you confirm that if you unset PSQL_WATCH_PAGER then >> the test passes for you? > yes, I tested it now, and unset PSQL_WATCH

[PATCH] Slight improvement of worker_spi.c example

2023-05-12 Thread Aleksander Alekseev
Hi, Currently the example uses the following order of calls: StartTransactionCommand(); SPI_connect(); PushActiveSnapshot(...); ... SPI_finish(); PopActiveSnapshot(); CommitTransactionCommand(); This could be somewhat misleading. Typically one expects something to b

Re: psql tests hangs

2023-05-12 Thread Pavel Stehule
pá 12. 5. 2023 v 15:26 odesílatel Tom Lane napsal: > Pavel Stehule writes: > > And generally, the root perl should to reset PSQL_WATCH_PAGER env > variable > > before executing psql. Probably it does with PSQL_PAGER, and maybe with > > PAGER. > > Oh! AFAICS, we don't do any of those things, but

Re: createuser --memeber and PG 16

2023-05-12 Thread Peter Eisentraut
On 11.05.23 16:07, Robert Haas wrote: On Wed, May 10, 2023 at 1:33 PM Bruce Momjian wrote: This seems like it will be forever confusing to people. I frankly don't know why --role matching CREATE ROLE ... ROLE IN was not already confusing in pre-PG 16. Any new ideas for improvement? Yeah, it

Re: Large files for relations

2023-05-12 Thread Stephen Frost
Greetings, * Dagfinn Ilmari Mannsåker (ilm...@ilmari.org) wrote: > Thomas Munro writes: > > On Fri, May 12, 2023 at 8:16 AM Jim Mlodgenski wrote: > >> On Mon, May 1, 2023 at 9:29 PM Thomas Munro wrote: > >>> I am not aware of any modern/non-historic filesystem[2] that can't do > >>> large files

Re: Large files for relations

2023-05-12 Thread Jim Mlodgenski
On Thu, May 11, 2023 at 7:38 PM Thomas Munro wrote: > On Fri, May 12, 2023 at 8:16 AM Jim Mlodgenski wrote: > > On Mon, May 1, 2023 at 9:29 PM Thomas Munro > wrote: > >> I am not aware of any modern/non-historic filesystem[2] that can't do > >> large files with ease. Anyone know of anything to

Re: psql tests hangs

2023-05-12 Thread Tom Lane
Pavel Stehule writes: > And generally, the root perl should to reset PSQL_WATCH_PAGER env variable > before executing psql. Probably it does with PSQL_PAGER, and maybe with > PAGER. Oh! AFAICS, we don't do any of those things, but I agree it seems like a good idea. Can you confirm that if you u

Re: psql tests hangs

2023-05-12 Thread Pavel Stehule
pá 12. 5. 2023 v 10:31 odesílatel Pavel Stehule napsal: > > > pá 12. 5. 2023 v 9:46 odesílatel Alvaro Herrera > napsal: > >> On 2023-May-12, Pavel Stehule wrote: >> >> > It hangs in perl >> >> I wonder if "hanging" actually means that it interpreted the sleep time >> as a very large integer, so

Re: walsender performance regression due to logical decoding on standby changes

2023-05-12 Thread Bharath Rupireddy
On Wed, May 10, 2023 at 3:23 PM Drouvot, Bertrand wrote: > > >> My current guess is that mis-using a condition variable is the best bet. I > >> think it should work to use ConditionVariablePrepareToSleep() before a > >> WalSndWait(), and then ConditionVariableCancelSleep(). I.e. to never use > >>

Re: running logical replication as the subscription owner

2023-05-12 Thread Ajin Cherian
On Fri, May 12, 2023 at 1:49 PM Amit Kapila wrote: > > On Fri, May 12, 2023 at 9:10 AM Masahiko Sawada wrote: > > > > On Fri, May 12, 2023 at 1:12 AM Robert Haas wrote: > > > > > > On Thu, May 11, 2023 at 7:38 AM Amit Kapila > > > wrote: > > > > Do we want the initial sync to also respect 'run

Re: v16 regression - wrong query results with LEFT JOINs + join removal

2023-05-12 Thread Robert Haas
On Thu, May 11, 2023 at 4:16 PM Kirk Wolak wrote: > Forgive the noob question... But does this trigger a regression test to be > created? > And who tracks/pushes that? Tom included one in the commit. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Large files for relations

2023-05-12 Thread Dagfinn Ilmari Mannsåker
Thomas Munro writes: > On Fri, May 12, 2023 at 8:16 AM Jim Mlodgenski wrote: >> On Mon, May 1, 2023 at 9:29 PM Thomas Munro wrote: >>> I am not aware of any modern/non-historic filesystem[2] that can't do >>> large files with ease. Anyone know of anything to worry about on that >>> front? >> >

Re: Orphaned files in base/[oid]

2023-05-12 Thread Alexey Gordeev
On 14.08.2017 23:56, Andres Freund wrote: Alternatively we could do something without marker files, with some added complexity: Keep track of all "uncommitted new files" in memory, and log them every checkpoint. Commit/abort records clear elements of that list. Since we always start replay at the

RE: Non-superuser subscription owners

2023-05-12 Thread Zhijie Hou (Fujitsu)
On Tuesday, April 4, 2023 1:57 AM Robert Haas wrote: > > On Sat, Apr 1, 2023 at 12:00 PM Alexander Lakhin > wrote: > > I've managed to reproduce it using the following script: > > for ((i=1;i<=10;i++)); do > > echo "iteration $i" > > echo " > > CREATE ROLE sub_user; > > CREATE SUBSCRIPTION tests

Re: Orphaned files in base/[oid]

2023-05-12 Thread Alexey Gordeev
 12.05.2023, 14:17, "Andres Freund" :Alternatively we could do something without marker files, with someadded complexity: Keep track of all "uncommitted new files" in memory,and log them every checkpoint. Commit/abort records clear elements ofthat list. Since we always start replay at the beginning

Re: Allow pg_archivecleanup to remove backup history files

2023-05-12 Thread torikoshia
On 2023-05-10 17:52, Bharath Rupireddy wrote: Thanks for your comments! Just curious to know the driving point behind this proposal - is pg_archivecleanup deployed in production that was unable to clean up the history files and there were many such history files left? It will help us know how pg

Re: psql tests hangs

2023-05-12 Thread Pavel Stehule
pá 12. 5. 2023 v 9:46 odesílatel Alvaro Herrera napsal: > On 2023-May-12, Pavel Stehule wrote: > > > It hangs in perl > > I wonder if "hanging" actually means that it interpreted the sleep time > as a very large integer, so it's just sleeping for a long time. > There is some interaction with psp

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-05-12 Thread Peter Eisentraut
On 10.05.23 20:04, Andres Freund wrote: This commit adds a test is(scalar @lp_off, $ROWCOUNT, "acquired row offsets"); *before* that skip_all call. This appears to be invalid. If the skip_all happens, you get a complaint like t/004_verify_heapam.pl (Wstat: 0 Tests: 1 Failed: 0) Parse erro

Re: Add standard collation UNICODE

2023-05-12 Thread Peter Eisentraut
On 08.05.23 17:48, Peter Eisentraut wrote: On 27.04.23 13:44, Daniel Verite wrote: This collation has an empty pg_collation.collversion column, instead of being set to the same value as "und-x-icu" to track its version. The original patch implements this as an INSERT in which it would be easy

Re: psql tests hangs

2023-05-12 Thread Alvaro Herrera
On 2023-May-12, Pavel Stehule wrote: > It hangs in perl I wonder if "hanging" actually means that it interpreted the sleep time as a very large integer, so it's just sleeping for a long time. About the server locale, note that the ->new() call explicitly requests the C locale -- it's only psql t

Re: Assert failure of the cross-check for nullingrels

2023-05-12 Thread Richard Guo
On Fri, Mar 17, 2023 at 11:05 AM Richard Guo wrote: > Here is an updated patch with comments and test case. I also change the > code to store 'group_clause_relids' directly in RestrictInfo. > BTW, I've added an open item for this issue. Thanks Richard

Re: psql tests hangs

2023-05-12 Thread Kirk Wolak
On Fri, May 12, 2023 at 2:40 AM Pavel Stehule wrote: > pá 12. 5. 2023 v 8:20 odesílatel Kirk Wolak napsal: > >> On Fri, May 12, 2023 at 1:46 AM Pavel Stehule >> wrote: >> >>> pá 12. 5. 2023 v 6:50 odesílatel Kirk Wolak napsal: >>> On Fri, May 12, 2023 at 12:14 AM Tom Lane wrote: >>>