Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-16 Thread Dilip Kumar
On Thu, Nov 16, 2023 at 3:11 PM Alvaro Herrera wrote: > > I just noticed that 0003 does some changes to > TransactionGroupUpdateXidStatus() that haven't been adequately > explained AFAICS. How do you know that these changes are safe? IMHO this is safe as well as logical to do w.r.t.

Re: remaining sql/json patches

2023-11-16 Thread jian he
hi. minor issues. In transformJsonFuncExpr(ParseState *pstate, JsonFuncExpr *func) func.behavior->on_empty->location and func.behavior->on_error->location are correct. but in ExecInitJsonExpr, jsestate->jsexpr->on_empty->location is -1, jsestate->jsexpr->on_error->location is -1. Maybe we can

Re: SQL:2011 application time

2023-11-16 Thread jian he
based on v17. begin; drop table if exists s1; CREATE TABLE s1 (id numrange, misc int, misc1 text); create role test101 login; grant update, select on s1 to test101; insert into s1 VALUES ('[1,1000]',2); set session authorization test101; update s1 set id = '[1,1000]'; savepoint sp1; update s1

Re: Wrong rows estimations with joins of CTEs slows queries by more than factor 500

2023-11-16 Thread Richard Guo
On Fri, Nov 17, 2023 at 11:38 AM Tom Lane wrote: > That line of argument also leads to the conclusion that it'd be > okay to expose info about the ordering of the CTE result to the > upper planner. This patch doesn't do that, and I'm not sufficiently > excited about the issue to go write some

Re: Use of backup_label not noted in log

2023-11-16 Thread Laurenz Albe
On Thu, 2023-11-16 at 20:18 -0800, Andres Freund wrote: > I've often had to analyze what caused corruption in PG instances, where the > symptoms match not having had backup_label in place when bringing on the > node. However that's surprisingly hard - the only log messages that indicate > use of

Re: Hide exposed impl detail of wchar.c

2023-11-16 Thread Nathan Bossart
On Thu, Nov 16, 2023 at 06:06:30PM -0500, Tom Lane wrote: > I'm generally sympathetic to the idea that simd.h was a rather large > dependency to add to something as widely used as pg_wchar.h. So I'd > favor getting it out of there just on compilation-time grounds, > independently of whether it's

Re: MERGE ... RETURNING

2023-11-16 Thread jian he
> > Attached is a separate patch with those doc updates, intended to be > applied and back-patched independently of the main RETURNING patch. > > Regards, > Dean + You will require the SELECT privilege and any column(s) + of the data_source and + target_table_name referred to + in any

Re: Synchronizing slots from primary to standby

2023-11-16 Thread Ajin Cherian
On Tue, Nov 14, 2023 at 12:57 AM Zhijie Hou (Fujitsu) wrote: > > 2) Raise error if user slot with same name already exists on standby. "ERROR: not synchronizing slot test; it is a user created slot" I just tested this using v35 and to me the error message when this happens is not very good.

Use of backup_label not noted in log

2023-11-16 Thread Andres Freund
Hi, I've often had to analyze what caused corruption in PG instances, where the symptoms match not having had backup_label in place when bringing on the node. However that's surprisingly hard - the only log messages that indicate use of backup_label are at DEBUG1. Given how crucial use of

Re: Wrong rows estimations with joins of CTEs slows queries by more than factor 500

2023-11-16 Thread Laurenz Albe
On Thu, 2023-11-16 at 22:38 -0500, Tom Lane wrote: > That line of argument also leads to the conclusion that it'd be > okay to expose info about the ordering of the CTE result to the > upper planner.  [...]  The fence is sort of one-way > in this line of thinking: information can propagate up to

Re: Wrong rows estimations with joins of CTEs slows queries by more than factor 500

2023-11-16 Thread David G. Johnston
On Thursday, November 16, 2023, Tom Lane wrote: > > That line of argument also leads to the conclusion that it'd be > okay to expose info about the ordering of the CTE result to the > upper planner. This patch doesn't do that, and I'm not sufficiently > excited about the issue to go write some

Re: Wrong rows estimations with joins of CTEs slows queries by more than factor 500

2023-11-16 Thread Tom Lane
Richard Guo writes: > On Fri, Nov 17, 2023 at 2:16 AM Tom Lane wrote: >> So you could argue that there's more to do here, but I'm hesitant >> to go further. Part of the point of MATERIALIZED is to be an >> optimization fence, so breaking down that fence is something to be >> wary of. Maybe we

Re: Wrong results with grouping sets

2023-11-16 Thread Richard Guo
On Thu, Nov 16, 2023 at 11:25 PM Alena Rybakina wrote: > I noticed that this query worked correctly in the main branch with the > inequality operator: > > postgres=# select distinct on (a, b) a, b from (values (3, 1), (2, 2)) as > t (a, b) where a > b group by grouping sets((a, b), (a)); a | b

Re: [HACKERS] Should logtape.c blocks be of type long?

2023-11-16 Thread Michael Paquier
On Thu, Nov 16, 2023 at 03:03:46PM +0100, Heikki Linnakangas wrote: > BufFileTellBlock should be adjusted too. Or removed altogether; it's been > commented out since year 2000. Other than that, looks good to me. Yes, I recall wondering about what to do on this one so I just let it be when

Re: Wrong rows estimations with joins of CTEs slows queries by more than factor 500

2023-11-16 Thread Richard Guo
On Fri, Nov 17, 2023 at 2:16 AM Tom Lane wrote: > So you could argue that there's more to do here, but I'm hesitant > to go further. Part of the point of MATERIALIZED is to be an > optimization fence, so breaking down that fence is something to be > wary of. Maybe we shouldn't even take this

RE: Synchronizing slots from primary to standby

2023-11-16 Thread Zhijie Hou (Fujitsu)
On Tuesday, November 14, 2023 10:27 PM Drouvot, Bertrand wrote: > On 11/13/23 2:57 PM, Zhijie Hou (Fujitsu) wrote: > > On Friday, November 10, 2023 4:16 PM Drouvot, Bertrand > wrote: > >> Yeah good point, agree to just error out in all the case then (if we > >> discard the sync_ reserved

Re: Hide exposed impl detail of wchar.c

2023-11-16 Thread Jubilee Young
On Thu, Nov 16, 2023 at 2:54 PM Nathan Bossart wrote: > That being said, it's not unheard of for Postgres to make adjustments for > third-party code (search for "pljava" in commits 62aba76 and f4aa3a1). I > read the description of the pgrx problem [2], and I'm still trying to > understand the

Re: Faster "SET search_path"

2023-11-16 Thread Jeff Davis
On Tue, 2023-11-14 at 20:13 -0800, Jeff Davis wrote: > On Thu, 2023-10-19 at 19:01 -0700, Jeff Davis wrote: > > 0003: Cache for recomputeNamespacePath. > > Committed with some further simplification around the OOM handling. While I considered OOM during hash key initialization, I missed some

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-16 Thread Melanie Plageman
On Thu, Nov 16, 2023 at 3:29 PM Robert Haas wrote: > > On Wed, Nov 15, 2023 at 6:17 PM Andres Freund wrote: > > Thoughts on whether to backpatch? It'd probably be at least a bit painful, > > there have been a lot of changes in the surrounding code in the last 5 > > years. > > I guess the main

Re: EXCLUDE COLLATE in CREATE/ALTER TABLE document

2023-11-16 Thread shihao zhong
Hi Jian, Thanks for your comments, a new version is attached. Thanks, Shihao On Fri, Nov 10, 2023 at 9:59 AM jian he wrote: > On Wed, Nov 1, 2023 at 10:30 AM shihao zhong > wrote: > > > > Thank you for your feedback on my previous patch. I have fixed the issue > and attached a new patch for

Re: Hide exposed impl detail of wchar.c

2023-11-16 Thread Tom Lane
Nathan Bossart writes: > It looks like is_valid_ascii() was originally added to pg_wchar.h so that > it could easily be used elsewhere [0] [1], but that doesn't seem to have > happened yet. It seems to be new as of v15, so there wouldn't have been a lot of time for external code to adopt it. As

Re: Hide exposed impl detail of wchar.c

2023-11-16 Thread Nathan Bossart
On Thu, Nov 16, 2023 at 12:10:59PM -0800, Jubilee Young wrote: > On the off-chance that everyone agrees with me without reserve, the > attached patch moves the relevant code around (and includes the gory > details). This seems to be unlikely to be the only mildly-exotic build > system failure

Re: Hide exposed impl detail of wchar.c

2023-11-16 Thread Tom Lane
Jubilee Young writes: > I have root-caused the exact problem, but the bug is... social, rather > than technical in nature: people with inadequate options at their > disposal have implemented increasingly clever educated guesses that > are increasingly prone to going wrong, rather than just asking

Re: [PATCH] Tracking statements entry timestamp in pg_stat_statements

2023-11-16 Thread Andrei Zubkov
Hi hackers, Patch rebased to the current master -- regards, Andrei Zubkov Postgres Professional From ff2ff96352a843d32a1213a1e953503fd1525b7b Mon Sep 17 00:00:00 2001 From: Andrei Zubkov Date: Fri, 17 Nov 2023 00:27:24 +0300 Subject: [PATCH 1/2] pg_stat_statements tests: Add NOT NULL checking

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2023-11-16 Thread Daniel Gustafsson
> On 30 Jul 2020, at 14:11, Robert Haas wrote: > > On Mon, Jul 20, 2020 at 3:47 PM Robert Haas wrote: >> But I also agree that what pg_start_backup() was doing before v13 was >> wrong; that's why I committed >> 303640199d0436c5e7acdf50b837a027b5726594. The only reason I didn't >> back-patch it

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-16 Thread Robert Haas
On Thu, Nov 16, 2023 at 3:49 PM Andres Freund wrote: > I think the main reason it's not all that bad, even when hitting this path, is > that one stall every 8GB just isn't that much and that the stalls aren't that > long - the leaf page fsm updates don't use the strategy, so they're still >

Adding deprecation notices to pgcrypto documentation

2023-11-16 Thread Daniel Gustafsson
In the "Allow tests to pass in OpenSSL FIPS mode" thread [0] it was discovered that 3DES is joining the ranks of NIST disallowed algorithms. The attached patch adds a small note to the pgcrypto documentation about deprecated uses of algorithms. I've kept it to "official" notices such as RFC's

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-16 Thread Andres Freund
Hi, On 2023-11-16 15:29:38 -0500, Robert Haas wrote: > On Wed, Nov 15, 2023 at 6:17 PM Andres Freund wrote: > > Thoughts on whether to backpatch? It'd probably be at least a bit painful, > > there have been a lot of changes in the surrounding code in the last 5 > > years. > > I guess the main

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-16 Thread Robert Haas
On Wed, Nov 15, 2023 at 6:17 PM Andres Freund wrote: > Thoughts on whether to backpatch? It'd probably be at least a bit painful, > there have been a lot of changes in the surrounding code in the last 5 years. I guess the main point that I'd make here is that we shouldn't back-patch because it's

Hide exposed impl detail of wchar.c

2023-11-16 Thread Jubilee Young
Hello, I work on pgrx, a Rust crate (really, a set of them) that allows people to use Rust to write extensions against Postgres, exporting what Postgres sees as ordinary "C" dynamic libraries. Unfortunately, the build system for this is a touch complicated, as it cannot simply run pgxs.mk, and

Re: On non-Windows, hard depend on uselocale(3)

2023-11-16 Thread Thomas Munro
On Thu, Nov 16, 2023 at 12:06 PM Tom Lane wrote: > Thomas Munro writes: > > Perhaps we could use snprintf_l() and strtod_l() where available. > > They're not standard, but they are obvious extensions that NetBSD and > > Windows have, and those are the two systems for which we are doing > >

Re: Wrong rows estimations with joins of CTEs slows queries by more than factor 500

2023-11-16 Thread Tom Lane
Richard Guo writes: > I think the second plan (patched) makes more sense. In the first plan > (unpatched), the HashAggregate node actually does not reduce the the > number of rows because it groups by 'unique1', but planner does not know > that because it lacks statistics for Vars referencing

Re: trying again to get incremental backup

2023-11-16 Thread Robert Haas
On Thu, Nov 16, 2023 at 12:34 PM Alvaro Herrera wrote: > Putting those two thoughts together, I think pg_basebackup with > --progress could tell you "still waiting for the summary file up to LSN > %X/%X to appear, and the walsummarizer is currently handling lsn %X/%X" > or something like that.

Re: ALTER TABLE uses a bistate but not for toast tables

2023-11-16 Thread Justin Pryzby
@cfbot: rebased >From a9bb61421c24bd3273a8519362febb4073c297a1 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Tue, 21 Jun 2022 22:28:06 -0500 Subject: [PATCH v4] WIP: use BulkInsertState for toast tuples, too DONE: ALTER, CLUSTER TODO: copyto, copyfrom? slot_getsomeattrs

Re: trying again to get incremental backup

2023-11-16 Thread Alvaro Herrera
On 2023-Nov-16, Alvaro Herrera wrote: > On 2023-Oct-04, Robert Haas wrote: > > - Right now, I have a hard-coded 60 second timeout for WAL > > summarization. If you try to take an incremental backup and the WAL > > summaries you need don't show up within 60 seconds, the backup times > > out. I

Re: trying again to get incremental backup

2023-11-16 Thread Alvaro Herrera
On 2023-Nov-16, Robert Haas wrote: > On Thu, Nov 16, 2023 at 5:21 AM Alvaro Herrera > wrote: > > I meant code like this > > > > memcpy(, rlocator, sizeof(RelFileLocator)); > > key.forknum = forknum; > > entry = blockreftable_lookup(brtab->hash, key); > > Ah, I hadn't

Re: trying again to get incremental backup

2023-11-16 Thread Alvaro Herrera
On 2023-Oct-04, Robert Haas wrote: > - I would like some feedback on the generation of WAL summary files. > Right now, I have it enabled by default, and summaries are kept for a > week. That means that, with no additional setup, you can take an > incremental backup as long as the reference backup

Re: trying again to get incremental backup

2023-11-16 Thread Robert Haas
On Thu, Nov 16, 2023 at 5:21 AM Alvaro Herrera wrote: > I meant code like this > > memcpy(, rlocator, sizeof(RelFileLocator)); > key.forknum = forknum; > entry = blockreftable_lookup(brtab->hash, key); Ah, I hadn't thought about that. Another way of handling that might be

Re: trying again to get incremental backup

2023-11-16 Thread Robert Haas
On Tue, Nov 14, 2023 at 8:12 AM Alvaro Herrera wrote: > 0001 looks OK to push, and since it stands on its own I would get it out > of the way soon rather than waiting for the rest of the series to be > further reviewed. All right, done. > 0003: > AmWalSummarizerProcess() is unused. Remove?

Re: pgsql: doc: fix wording describing the checkpoint_flush_after GUC

2023-11-16 Thread Robert Haas
On Tue, Nov 14, 2023 at 3:01 PM Andres Freund wrote: > Hm, I really can't get excited about this. To me the "you" sounds worse, but > whatever... To me, it seems flat-out incorrect without the "you". It might be better to rephrase the whole thing entirely so that it doesn't need to address the

Re: Do away with a few backwards compatibility macros

2023-11-16 Thread Nathan Bossart
On Thu, Nov 16, 2023 at 07:11:41PM +0530, Bharath Rupireddy wrote: > After a recent commit 6a72c42f (a related discussion [1]) which > removed MemoryContextResetAndDeleteChildren(), I think there are a > couple of other backward compatibility macros out there that can be > removed. These macros

Re: Wrong results with grouping sets

2023-11-16 Thread Alena Rybakina
Hi! Thank you for your work on the subject. On 25.09.2023 10:11, Richard Guo wrote: I think I've come across a wrong result issue with grouping sets, as shown by the query below. -- result is correct with only grouping sets select a, b from (values (1, 1), (2, 2)) as t (a, b) where a = b group

Re: BUG #18097: Immutable expression not allowed in generated at

2023-11-16 Thread Tom Lane
Aleksander Alekseev writes: >> True, but from the perspective of the affected code, the question is >> basically "did you call expression_planner() yet". So I like this >> naming for that connection, whereas something based on "transformation" >> doesn't really connect to anything in existing

Re: psql not responding to SIGINT upon db reconnection

2023-11-16 Thread Heikki Linnakangas
On 06/11/2023 19:16, Tristan Partin wrote: That sounds like a much better solution. Attached you will find a v4 that implements your suggestion. Please let me know if there is something that I missed. I can confirm that the patch works. This patch is missing a select(). It will busy loop until

Re: [HACKERS] Should logtape.c blocks be of type long?

2023-11-16 Thread Heikki Linnakangas
On 26/09/2023 07:15, Michael Paquier wrote: On Sun, Sep 24, 2023 at 10:42:49AM +0900, Michael Paquier wrote: Indeed, or Windows decides that making long 8-byte is wiser, but I doubt that's ever going to happen on backward-compatibility ground. While looking more at that, I've noticed that I

Re: retire MemoryContextResetAndDeleteChildren backwards compatibility macro

2023-11-16 Thread Bharath Rupireddy
On Tue, Nov 14, 2023 at 11:29 PM Nathan Bossart wrote: > > On Tue, Nov 14, 2023 at 10:46:25PM +0530, Bharath Rupireddy wrote: > > FWIW, there are other backward compatibility macros out there like > > tuplestore_donestoring which was introduced by commit dd04e95 21 years > > ago and SPI_push()

Do away with a few backwards compatibility macros

2023-11-16 Thread Bharath Rupireddy
Hi, After a recent commit 6a72c42f (a related discussion [1]) which removed MemoryContextResetAndDeleteChildren(), I think there are a couple of other backward compatibility macros out there that can be removed. These macros are tuplestore_donestoring() which was introduced by commit dd04e95 21

Re: Trigger violates foreign key constraint

2023-11-16 Thread Aleksander Alekseev
Hi, > Attached is a slightly modified version of the patch. The patch was marked as "Needs Review" so I decided to take a look. I believe it's a good patch. The text is well written, has the necessary references, it warns the user but doesn't scare him/her too much. > I couldn't make it

Re: ALTER COLUMN ... SET EXPRESSION to alter stored generated column's expression

2023-11-16 Thread Amul Sul
On Thu, Nov 16, 2023 at 2:50 AM Peter Eisentraut wrote: > On 15.11.23 13:26, Amul Sul wrote: > > Question: Why are you using AT_PASS_ADD_OTHERCONSTR? I don't know if > > it's right or wrong, but if you have a specific reason, it would be > > good > > to know. > > > > I referred

RE: pg_upgrade and logical replication

2023-11-16 Thread Hayato Kuroda (Fujitsu)
Dear Vignesh, Thanks for updating the patch! Here are some comments. They are mainly cosmetic because I have not read yours these days. 01. binary_upgrade_add_sub_rel_state() ``` +/* We must check these things before dereferencing the arguments */ +if (PG_ARGISNULL(0) || PG_ARGISNULL(1)

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-11-16 Thread Bharath Rupireddy
On Thu, Nov 16, 2023 at 4:01 PM Alvaro Herrera wrote: > > On 2023-Nov-16, Peter Smith wrote: > > > I searched HEAD code and did not find any "translator:" comments for > > just ordinary slot name substitutions like these; AFAICT that comment > > is not necessary anymore. > > True. Lose that.

RE: Popcount optimization using AVX512

2023-11-16 Thread Shankaran, Akash
Sorry for the late response here. We spent some time researching and measuring the frequency impact of AVX512 instructions used here. >How does this compare to older CPUs, and more mixed workloads? IIRC, the use of AVX512 (which I believe this instruction to be included in) has significant

Re: serial and partitioned table

2023-11-16 Thread Ashutosh Bapat
On Mon, Nov 13, 2023 at 3:39 PM Peter Eisentraut wrote: > > On 17.10.23 09:25, Ashutosh Bapat wrote: > > #create table tpart (a serial primary key, src varchar) partition by > > range(a); > > CREATE TABLE > > #create table t_p4 (a int primary key, src varchar); > > CREATE TABLE > > To appease

Re: partitioning and identity column

2023-11-16 Thread Ashutosh Bapat
On Mon, Nov 13, 2023 at 3:51 PM Peter Eisentraut wrote: > > On 27.10.23 13:32, Ashutosh Bapat wrote: > > I think we should fix these anomalies as follows > > 1. Allow identity columns to be added to the partitioned table > > irrespective of whether they have partitions of not. > > 2. Propagate

Re: add log messages when replication slots become active and inactive (was Re: Is it worth adding ReplicationSlot active_pid to ReplicationSlotPersistentData?)

2023-11-16 Thread Alvaro Herrera
On 2023-Nov-16, Peter Smith wrote: > I searched HEAD code and did not find any "translator:" comments for > just ordinary slot name substitutions like these; AFAICT that comment > is not necessary anymore. True. Lose that. The rationale I have is to consider whether a translator looking at the

Re: WaitEventSet resource leakage

2023-11-16 Thread Heikki Linnakangas
On 16/11/2023 01:08, Tom Lane wrote: Heikki Linnakangas writes: On 09/03/2023 20:51, Tom Lane wrote: After further thought that seems like a pretty ad-hoc solution. We probably can do no better in the back branches, but shouldn't we start treating WaitEventSets as ResourceOwner-managed

Re: trying again to get incremental backup

2023-11-16 Thread Alvaro Herrera
On 2023-Nov-13, Robert Haas wrote: > On Mon, Nov 13, 2023 at 11:25 AM Alvaro Herrera > wrote: > > Also, it would be good to provide and use a > > function to initialize a BlockRefTableKey from the RelFileNode and > > forknum components, and ensure that any padding bytes are zeroed. > >

Re: Synchronizing slots from primary to standby

2023-11-16 Thread Amit Kapila
On Wed, Nov 15, 2023 at 5:21 PM shveta malik wrote: > > PFA v34. > Few comments on v34-0001* === 1. + char buf[100]; + + buf[0] = '\0'; + + if (MySubscription->twophasestate == LOGICALREP_TWOPHASE_STATE_PENDING) + strcat(buf, "twophase"); + if (MySubscription->failoverstate

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-16 Thread Alvaro Herrera
I just noticed that 0003 does some changes to TransactionGroupUpdateXidStatus() that haven't been adequately explained AFAICS. How do you know that these changes are safe? 0001 contains one typo in the docs, "cotents". I'm not a fan of the fact that some CLOG sizing macros moved to clog.h,

Re: Wrong rows estimations with joins of CTEs slows queries by more than factor 500

2023-11-16 Thread Richard Guo
On Thu, Nov 9, 2023 at 6:45 AM Tom Lane wrote: > The existing RTE_SUBQUERY stanza has most of what we need for this, > so I experimented with extending that to also handle RTE_CTE. It > seems to work, though I soon found out that it needed tweaking for > the case where the CTE is

Re: remaining sql/json patches

2023-11-16 Thread Amit Langote
On Thu, Nov 16, 2023 at 2:11 AM Andres Freund wrote: > On 2023-11-15 22:00:41 +0900, Amit Langote wrote: > > > This causes a nontrivial increase in the size of the parser (~5% in an > > > optimized build here), I wonder if we can do better. > > > > Hmm, sorry if I sound ignorant but what do you

Re: Add new option 'all' to pg_stat_reset_shared()

2023-11-16 Thread torikoshia
On 2023-11-16 16:48, Michael Paquier wrote: On Wed, Nov 15, 2023 at 04:25:14PM +0900, Michael Paquier wrote: Other than that, it looks OK. Tweaked the queries of this one slightly, and applied. So I think that we are now good for this thread. Thanks, all! Thanks for the modification and