Re: pg_walfile_name_offset can return inconsistent values

2023-11-10 Thread Andres Freund
Hi, On 2023-11-09 16:14:07 -0500, Bruce Momjian wrote: > On Thu, Nov 9, 2023 at 09:49:48PM +0100, Matthias van de Meent wrote: > > Either way, I think fix #1 is most correct (as was attached in > > offset2.diff, and quoted verbatim here), because that has no chance of > > having surprising

Re: remaining sql/json patches

2023-11-10 Thread Amit Langote
Hi Erik, On Sat, Nov 11, 2023 at 11:52 Erik Rijkers wrote: > Hi, > > At the moment, what is the patchset to be tested? The latest SQL/JSON > server I have is from September, and it's become unclear to me what > belongs to the SQL/JSON patchset. It seems to me cfbot erroneously > shows green

Re: locked reads for atomics

2023-11-10 Thread Nathan Bossart
On Fri, Nov 10, 2023 at 06:48:39PM -0800, Andres Freund wrote: > Yes. We should optimize pg_atomic_exchange_u32() one of these days - it can be > done *far* faster than a cmpxchg. When I was adding the atomic abstraction > there was concern with utilizing too many different atomic instructions. I

Re: remaining sql/json patches

2023-11-10 Thread Erik Rijkers
Hi, At the moment, what is the patchset to be tested? The latest SQL/JSON server I have is from September, and it's become unclear to me what belongs to the SQL/JSON patchset. It seems to me cfbot erroneously shows green because it successfully compiles later detail-patches (i.e., not the

Re: locked reads for atomics

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 20:38:13 -0600, Nathan Bossart wrote: > On Fri, Nov 10, 2023 at 03:11:50PM -0800, Andres Freund wrote: > > On 2023-11-10 14:51:28 -0600, Nathan Bossart wrote: > >> + * This read is guaranteed to read the current value, > > > > It doesn't guarantee that *at all*. What it

Re: locked reads for atomics

2023-11-10 Thread Nathan Bossart
On Fri, Nov 10, 2023 at 03:11:50PM -0800, Andres Freund wrote: > On 2023-11-10 14:51:28 -0600, Nathan Bossart wrote: >> + * This read is guaranteed to read the current value, > > It doesn't guarantee that *at all*. What it guarantees is solely that the > current CPU won't be doing something that

Re: Adding facility for injection points (or probe points?) for more advanced tests

2023-11-10 Thread Andres Freund
Hi, On 2023-10-25 13:13:38 +0900, Michael Paquier wrote: > So, please find attached a patch set that introduces an in-core > facility to be able to set what I'm calling here an "injection point", > that consists of being able to register in shared memory a callback > that can be run within a

Re: Why do indexes and sorts use the database collation?

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 16:03:16 -0800, Jeff Davis wrote: > An "en_US" user doing: > >CREATE TABLE foo(t TEXT PRIMARY KEY); > > is providing no indication that they want an index tailored to their > locale. Yet we are creating the index with the "en_US" collation and > therefore imposing huge

Re: maybe a type_sanity. sql bug

2023-11-10 Thread Michael Paquier
On Sat, Nov 11, 2023 at 08:00:00AM +0800, jian he wrote: > I am not sure the pg_class "relam" description part is correct. since > partitioned indexes (relkind "I") also have the access method, but no > storage. > " > If this is a table or an index, the access method used (heap, B-tree, > hash,

Re: A recent message added to pg_upgade

2023-11-10 Thread Michael Paquier
On Fri, Nov 10, 2023 at 03:27:25PM +0530, Amit Kapila wrote: > I don't think this comment is correct because there won't be any apply > activity on the new cluster as after restoration subscriptions should > be disabled. On the old cluster, I think one problem is that the > origins may move

Why do indexes and sorts use the database collation?

2023-11-10 Thread Jeff Davis
An "en_US" user doing: CREATE TABLE foo(t TEXT PRIMARY KEY); is providing no indication that they want an index tailored to their locale. Yet we are creating the index with the "en_US" collation and therefore imposing huge performance costs (something like 2X slower index build time than the

Re: maybe a type_sanity. sql bug

2023-11-10 Thread jian he
looking around. I found other three minor issues. attached. I am not sure the pg_class "relam" description part is correct. since partitioned indexes (relkind "I") also have the access method, but no storage. " If this is a table or an index, the access method used (heap, B-tree, hash, etc.);

Re: Force the old transactions logs cleanup even if checkpoint is skipped

2023-11-10 Thread Andres Freund
Hi, On 2023-11-09 11:50:10 +, Zakhlystov, Daniil (Nebius) wrote: > > On 9 Nov 2023, at 01:30, Michael Paquier wrote: > > > > I am not really convinced that this is worth complicating the skipped > > path for this goal. In my experience, I've seen complaints where WAL > > archiving bloat was

Re: ResourceOwner refactoring

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 16:26:51 +0200, Heikki Linnakangas wrote: > The quick, straightforward fix is to move the "CurrentResourceOwner = NULL" > line earlier in CommitTransaction, per attached > 0003-Clear-CurrentResourceOwner-earlier-in-CommitTransact.patch. You're not > allowed to use the resource

Re: locked reads for atomics

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 21:49:06 +, John Morris wrote: > >I wonder if it's worth providing a set of "locked read" functions. > > Most out-of-order machines include “read acquire” and “write release” which > are pretty close to what you’re suggesting. Is that really true? It's IA64 lingo. X86

Re: locked reads for atomics

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 14:51:28 -0600, Nathan Bossart wrote: > Moving this to a new thread and adding it to the January commitfest. > > On Thu, Nov 09, 2023 at 03:27:33PM -0600, Nathan Bossart wrote: > > On Tue, Nov 07, 2023 at 04:58:16PM -0800, Andres Freund wrote: > >> However, even if there's

Re: locked reads for atomics

2023-11-10 Thread Nathan Bossart
On Fri, Nov 10, 2023 at 09:49:06PM +, John Morris wrote: > Most out-of-order machines include “read acquire” and “write release” > which are pretty close to what you’re suggesting. With the current > routines, we only have “read relaxed” and “write relaxed”. I think > implementing

Re: Failure during Building Postgres in Windows with Meson

2023-11-10 Thread Andres Freund
Hi, On November 10, 2023 10:53:12 AM PST, Tristan Partin wrote: >On Thu Nov 9, 2023 at 9:31 AM CST, Nazir Bilal Yavuz wrote: >> Hi, >> >> On Thu, 9 Nov 2023 at 18:27, Tristan Partin wrote: >> > >> > Can you try with Meson v1.2.3? >> >> I tried with Meson v1.2.3 and upstream, both failed with

Re: pgsql: Don't trust unvalidated xl_tot_len.

2023-11-10 Thread Thomas Munro
On Sat, Nov 11, 2023 at 10:42 AM Christoph Berg wrote: > > I haven't investigated the details yet, and it's not affecting the > > builds on apt.postgresql.org, but the Debian amd64 and i386 regression > > tests just failed this test on PG13 (11 and 15 are ok): > > 12 and 14 are also failing, now

Re: locked reads for atomics

2023-11-10 Thread John Morris
>I wonder if it's worth providing a set of "locked read" functions. Most out-of-order machines include “read acquire” and “write release” which are pretty close to what you’re suggesting. With the current routines, we only have “read relaxed” and “write relaxed”. I think implementing

Re: pgsql: Don't trust unvalidated xl_tot_len.

2023-11-10 Thread Christoph Berg
Re: To Thomas Munro > > src/test/recovery/t/039_end_of_wal.pl | 460 > > > > I haven't investigated the details yet, and it's not affecting the > builds on apt.postgresql.org, but the Debian amd64 and i386 regression > tests just failed this test on PG13 (11 and

Re: Atomic ops for unlogged LSN

2023-11-10 Thread Nathan Bossart
On Thu, Nov 09, 2023 at 03:27:33PM -0600, Nathan Bossart wrote: > I wonder if it's worth providing a set of "locked read" functions. Those > could just do a compare/exchange with 0 in the generic implementation. For > patches like this one where the overhead really shouldn't matter, I'd >

locked reads for atomics

2023-11-10 Thread Nathan Bossart
Moving this to a new thread and adding it to the January commitfest. On Thu, Nov 09, 2023 at 03:27:33PM -0600, Nathan Bossart wrote: > On Tue, Nov 07, 2023 at 04:58:16PM -0800, Andres Freund wrote: >> However, even if there's likely some other implied memory barrier that we >> could piggyback on,

Re: pg_dump needs SELECT privileges on irrelevant extension table

2023-11-10 Thread Jacob Champion
On Thu, Nov 9, 2023 at 11:02 AM Tom Lane wrote: > I'm hearing nothing but crickets :-( Yeah :/ Based on your arguments above, it sounds like your patch may improve several other corner cases when backported, so that sounds good overall to me. My best guess is that Timescale will be happy with

Re: Adding facility for injection points (or probe points?) for more advanced tests

2023-11-10 Thread Nathan Bossart
On Tue, Nov 07, 2023 at 05:01:16PM +0900, Michael Paquier wrote: > On Mon, Nov 06, 2023 at 10:28:14PM +0300, Nazir Bilal Yavuz wrote: >> I liked the idea; thanks for working on this! +1, this seems very useful. > +#ifdef USE_INJECTION_POINTS > +#define INJECTION_POINT_RUN(name)

Re: Add recovery to pg_control and remove backup_label

2023-11-10 Thread David Steele
On 11/10/23 00:37, Michael Paquier wrote: On Tue, Nov 07, 2023 at 05:20:27PM +0900, Michael Paquier wrote: On Mon, Nov 06, 2023 at 05:39:02PM -0400, David Steele wrote: I've retested today, and miss the failure. I'll let you know if I see this again. I've done a few more dozen runs, and

Re: Failure during Building Postgres in Windows with Meson

2023-11-10 Thread Tristan Partin
On Thu Nov 9, 2023 at 9:31 AM CST, Nazir Bilal Yavuz wrote: Hi, On Thu, 9 Nov 2023 at 18:27, Tristan Partin wrote: > > Can you try with Meson v1.2.3? I tried with Meson v1.2.3 and upstream, both failed with the same error. An employee at Collabora produced a fix[0]. It might still be

Re: Move bki file pre-processing from initdb to bootstrap

2023-11-10 Thread Krishnakumar R
Thank you for review, Peter. Makes sense on the split part. Was starting to think in same lines, at the end of last iteration. Will come back shortly. On Fri, Nov 10, 2023 at 12:48 AM Peter Eisentraut wrote: > On 17.10.23 03:32, Krishnakumar R wrote: > >> The version comparison has been moved

Re: pgsql: Don't trust unvalidated xl_tot_len.

2023-11-10 Thread Christoph Berg
Re: To Thomas Munro > I haven't investigated the details yet, and it's not affecting the > builds on apt.postgresql.org, but the Debian amd64 and i386 regression > tests just failed this test on PG13 (11 and 15 are ok): That's on Debian bullseye, fwiw. (But the 13 build on apt.pg.o/bullseye

Re: pgsql: Don't trust unvalidated xl_tot_len.

2023-11-10 Thread Christoph Berg
Re: Thomas Munro > Don't trust unvalidated xl_tot_len. > src/test/recovery/t/039_end_of_wal.pl | 460 I haven't investigated the details yet, and it's not affecting the builds on apt.postgresql.org, but the Debian amd64 and i386 regression tests just failed this

Re: Improvements in pg_dump/pg_restore toc format and performances

2023-11-10 Thread Nathan Bossart
On Tue, Oct 03, 2023 at 03:17:57PM +0530, vignesh C wrote: > Few comments: Pierre, do you plan to submit a new revision of this patch set for the November commitfest? If not, the commitfest entry may be marked as returned-with-feedback soon. -- Nathan Bossart Amazon Web Services:

Re: SET ROLE documentation improvement

2023-11-10 Thread Nathan Bossart
On Tue, Sep 26, 2023 at 08:33:25AM -0700, Yurii Rashkovskii wrote: > This is a good start, indeed. I've amended my patch to include it. Thanks for the new patch. Looking again, I'm kind of hesitant to add too much qualification to this note about losing superuser privileges. If we changed it to

Re: AdvanceXLInsertBuffers() vs wal_sync_method=open_datasync

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 17:16:35 +0200, Heikki Linnakangas wrote: > On 10/11/2023 05:54, Andres Freund wrote: > > In this case I had used wal_sync_method=open_datasync - it's often faster > > and > > if we want to scale WAL writes more we'll have to use it more widely (you > > can't have multiple

Re: Buffer Cache Problem

2023-11-10 Thread jacktby jacktby
Hi, I have 3 questions here: 1. I see comments in but_internals.h below: * Also, in places we do one-time reads of the flags without bothering to * lock the buffer header; this is generally for situations where we don't * expect the flag bit being tested

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-10 Thread Nathan Bossart
On Fri, Nov 10, 2023 at 10:17:49AM +0530, Dilip Kumar wrote: > On Thu, Nov 9, 2023 at 4:55 PM Alvaro Herrera wrote: >> The only point on which we do not have full consensus yet is the need to >> have one GUC per SLRU, and a lot of effort seems focused on trying to >> fix the problem without

Re: CRC32C Parallel Computation Optimization on ARM

2023-11-10 Thread Nathan Bossart
On Tue, Nov 07, 2023 at 08:05:45AM +, Xiang Gao wrote: > I think I understand what you mean, this is the latest patch. Thank you! Thanks for the new patch. +# PGAC_ARMV8_VMULL_INTRINSICS +# +# Check if the compiler supports the vmull_p64 +# intrinsic functions.

Re: AdvanceXLInsertBuffers() vs wal_sync_method=open_datasync

2023-11-10 Thread Heikki Linnakangas
On 10/11/2023 05:54, Andres Freund wrote: In this case I had used wal_sync_method=open_datasync - it's often faster and if we want to scale WAL writes more we'll have to use it more widely (you can't have multiple fdatasyncs in progress and reason about which one affects what, but you can have

Re: EXCLUDE COLLATE in CREATE/ALTER TABLE document

2023-11-10 Thread jian he
On Wed, Nov 1, 2023 at 10:30 AM shihao zhong wrote: > > Thank you for your feedback on my previous patch. I have fixed the issue and > attached a new patch for your review. Could you please take a look for it if > you have a sec? Thanks > Your patch works fine. you can see it here:

Re: Buffer Cache Problem

2023-11-10 Thread jacktby jacktby
> 2023年11月10日 22:31,jacktby jacktby 写道: > > In the bus_internal.h,I see > > Note: Buffer header lock (BM_LOCKED flag) must be held to examine or change > tag, state or wait_backend_pgprocno fields. >

Re: Buffer Cache Problem

2023-11-10 Thread jacktby jacktby
In the bus_internal.h,I see Note: Buffer header lock (BM_LOCKED flag) must be held to examine or change tag, state or wait_backend_pgprocno fields. As we all know, this buffer header lock

Re: ResourceOwner refactoring

2023-11-10 Thread Heikki Linnakangas
Thanks for the testing again! On 10/11/2023 11:00, Alexander Lakhin wrote: I could see two failure modes: 2023-11-10 08:42:28.870 UTC [1163274] ERROR:  ResourceOwnerEnlarge called after release started 2023-11-10 08:42:28.870 UTC [1163274] STATEMENT:  drop table t; 2023-11-10 08:42:28.870 UTC

Re: pg_upgrade and logical replication

2023-11-10 Thread vignesh C
On Thu, 9 Nov 2023 at 07:44, Peter Smith wrote: > > Thanks for addressing my previous review comments. > > I re-checked the latest patch v12-0001 and found the following: > > == > Commit message > > 1. > The new SQL binary_upgrade_create_sub_rel_state function has the following > syntax: >

Re: pg_upgrade and logical replication

2023-11-10 Thread vignesh C
On Thu, 9 Nov 2023 at 12:23, Michael Paquier wrote: > > On Thu, Nov 09, 2023 at 01:14:05PM +1100, Peter Smith wrote: > > Looks like v12 accidentally forgot to update this to the modified > > function name 'binary_upgrade_add_sub_rel_state' > > This v12 is overall cleaner than its predecessors.

Re: Bug: RLS policy FOR SELECT is used to check new rows

2023-11-10 Thread Laurenz Albe
On Fri, 2023-11-10 at 09:39 +, Dean Rasheed wrote: > On Thu, 9 Nov 2023 at 18:55, Laurenz Albe wrote: > > I think it can be useful to allow a user an UPDATE where the result > > does not satisfy the USING clause of the FOR SELECT policy. > > > > The idea that an UPDATE should only produce

Re: Parallel aggregates in PG 16.1

2023-11-10 Thread Matthias van de Meent
On Fri, 10 Nov 2023 at 11:47, ZIMANYI Esteban wrote: > > In MobilityDB > https://github.com/MobilityDB/MobilityDB > we have defined a tstzspan type which is a fixed-size equivalent of the > tstzrange type in PostgreSQL. > > We have a span_union aggregate function which is the equivalent of the

Re: Add new option 'all' to pg_stat_reset_shared()

2023-11-10 Thread torikoshia
On 2023-11-10 13:18, Andres Freund wrote: Hi, On November 8, 2023 11:28:08 PM PST, Michael Paquier wrote: On Thu, Nov 09, 2023 at 01:50:34PM +0900, torikoshia wrote: PGSTAT_KIND_SLRU cannot be reset by pg_stat_reset_shared(), so I feel uncomfortable to delete it all together. It might be

Re: trying again to get incremental backup

2023-11-10 Thread Dilip Kumar
On Tue, Nov 7, 2023 at 2:06 AM Robert Haas wrote: > > On Mon, Oct 30, 2023 at 2:46 PM Andres Freund wrote: > > After playing with this for a while, I don't see a reason for > > wal_summarize_mb > > from a memory usage POV at least. > > Here's v8. Changes: Review comments, based on what I

Re: Synchronizing slots from primary to standby

2023-11-10 Thread Drouvot, Bertrand
Hi, On 11/10/23 4:31 AM, shveta malik wrote: On Thu, Nov 9, 2023 at 9:15 PM Drouvot, Bertrand wrote: Yeah I think so, because there is a time window when one could "use" the slot after the promotion and before it is removed. Producing things like: " 2023-11-09 15:16:50.294 UTC [2580462] LOG:

Parallel aggregates in PG 16.1

2023-11-10 Thread ZIMANYI Esteban
In MobilityDB https://github.com/MobilityDB/MobilityDB we have defined a tstzspan type which is a fixed-size equivalent of the tstzrange type in PostgreSQL. We have a span_union aggregate function which is the equivalent of the range_agg function in PostgreSQL defined as follows CREATE

Re: A recent message added to pg_upgade

2023-11-10 Thread Amit Kapila
On Fri, Nov 10, 2023 at 7:50 AM Michael Paquier wrote: > > On Thu, Nov 09, 2023 at 04:52:32PM +0900, Michael Paquier wrote: > > Thanks! > > Also, please see also a patch about switching the logirep launcher to > rely on IsBinaryUpgrade to prevent its startup. Any thoughts about > that? >

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-11-10 Thread jian he
hi. +static void checkAllocations(); should be "static void checkAllocations(void);" ? PgStatShared_Memtrack there is a lock, but seems not initialized, and not used. Can you expand on it? So in view pg_stat_global_memory_tracking, column "total_memory_reserved" is a point of time, total memory

Re: Bug: RLS policy FOR SELECT is used to check new rows

2023-11-10 Thread Dean Rasheed
On Thu, 9 Nov 2023 at 18:55, Laurenz Albe wrote: > > I think it can be useful to allow a user an UPDATE where the result > does not satisfy the USING clause of the FOR SELECT policy. > > The idea that an UPDATE should only produce rows you can SELECT is not > true today: if you run an UPDATE

Re: POC, WIP: OR-clause support for indexes

2023-11-10 Thread Alena Rybakina
On 06.11.2023 16:51, Alena Rybakina wrote: I also support this approach. I have almost finished writing a patch that fixes the first problem related to the quadratic complexity of processing expressions by adding a hash table. I also added a check: if the number of groups is equal to the

Re: ResourceOwner refactoring

2023-11-10 Thread Alexander Lakhin
Hello Heikki, 09.11.2023 02:48, Heikki Linnakangas wrote: Thanks for the testing! Fixed. ... Thank you for the fix! Please look at one more failure caused be the new implementation of ResourceOwners: numdbs=80 for ((i=1;i<=10;i++)); do echo "ITERATION $i" for ((d=1;d<=$numdbs;d++)); do

Re: Move bki file pre-processing from initdb to bootstrap

2023-11-10 Thread Peter Eisentraut
On 17.10.23 03:32, Krishnakumar R wrote: The version comparison has been moved from initdb to bootstrap. This created some compatibility problems with windows tests. For now I kept the version check to not have \n added, which worked fine and serves the purpose. However hoping to have something

Re: Move bki file pre-processing from initdb to bootstrap

2023-11-10 Thread Peter Eisentraut
On 06.10.23 02:24, Krishnakumar R wrote: elog(INFO, "Open bki file %s\n", bki_file); + boot_yyin = fopen(bki_file, "r"); Why is this needed? It already reads the bki file from stdin? We no longer open the bki file in initdb and pass to postgres to parse from stdin, instead we open the bki

Re: Synchronizing slots from primary to standby

2023-11-10 Thread Drouvot, Bertrand
Hi, On 11/10/23 8:55 AM, Amit Kapila wrote: On Fri, Nov 10, 2023 at 12:50 PM Drouvot, Bertrand wrote: But even if we ERROR out instead of emitting a WARNING, the user would still need to be notified/monitor such errors. I agree that then probably they will come to know earlier because the

RE: MinGW compiler warnings in ecpg tests

2023-11-10 Thread Hayato Kuroda (Fujitsu)
Dear Peter, Michael, Sorry for reviving the old thread. While trying to build postgres on msys2 by meson, I faced the same warning. The OS is Windows 10. ``` $ ninja [2378/2402] Compiling C object src/interfaces/ecpg/test/sql/sqlda.exe.p/meson-generated_.._sqlda.c.obj