Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Wed, Mar 6, 2024 at 8:20 PM John Naylor wrote: > > On Tue, Mar 5, 2024 at 11:12 PM Masahiko Sawada wrote: > > > > > I'd like to push 0001 and 0002 shortly, and then do another sweep over > > > 0003, with remaining feedback, and get that in so we get some > > > buildfarm testing before the

Re: Reducing the log spam

2024-03-06 Thread Aleksander Alekseev
Hi, > So what about a parameter "log_suppress_sqlstates" that suppresses > logging ERROR and FATAL messages with the enumerated SQL states? > > My idea for a default setting would be something like > > log_suppress_sqlstates = '23505,3D000,3F000,42601,42704,42883,42P01' > > but that's of course

Re: [EXTERNAL] Re: Add non-blocking version of PQcancel

2024-03-06 Thread Denis Laxalde
In patch 0004, I noticed a couple of typos in the documentation; please find attached a fixup patch correcting these. Still in the documentation, same patch, the last paragraph documenting PQcancelPoll() ends as: + indicate the current stage of the connection procedure and might be

Re: Add new error_action COPY ON_ERROR "log"

2024-03-06 Thread Bharath Rupireddy
On Tue, Mar 5, 2024 at 4:48 AM Michael Paquier wrote: > > On Mon, Mar 04, 2024 at 05:00:00AM +0530, Bharath Rupireddy wrote: > > How about an extra option to error_action ignore-with-verbose which is > > similar to ignore but when specified emits one NOTICE per malformed > > row? With this, one

Re: CF entries for 17 to be reviewed

2024-03-06 Thread Andrey M. Borodin
> On 4 Mar 2024, at 14:51, Andrey M. Borodin wrote: > > I’ve read other small sections. Here are statuses for "Refactoring" section: * New [relation] options engine Relatively heavy refactoring. Author keeps interest to the patch for some years now. As I understood the main problem

Re: Wrong security context for deferred triggers?

2024-03-06 Thread Laurenz Albe
On Mon, 2023-11-06 at 18:29 +0100, Tomas Vondra wrote: > On 11/6/23 14:23, Laurenz Albe wrote: > > This behavior looks buggy to me. What do you think? > > I cannot imagine that it is a security problem, though. > > How could code getting executed under the wrong role not be a security > issue?

RE: Synchronizing slots from primary to standby

2024-03-06 Thread Zhijie Hou (Fujitsu)
On Wednesday, March 6, 2024 9:13 PM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, March 6, 2024 11:04 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wednesday, March 6, 2024 9:30 AM Masahiko Sawada > > wrote: > > > > Hi, > > > > > On Fri, Mar 1, 2024 at 4:21 PM Zhijie Hou (Fujitsu) > > > > > >

Re: remaining sql/json patches

2024-03-06 Thread jian he
On Wed, Mar 6, 2024 at 12:07 PM Amit Langote wrote: > > Hi Tomas, > > On Wed, Mar 6, 2024 at 6:30 AM Tomas Vondra > wrote: > > > > Hi, > > > > I know very little about sql/json and all the json internals, but I > > decided to do some black box testing. I built a large JSONB table > > (single

RE: Synchronizing slots from primary to standby

2024-03-06 Thread Zhijie Hou (Fujitsu)
On Wednesday, March 6, 2024 11:04 AM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, March 6, 2024 9:30 AM Masahiko Sawada > wrote: > > Hi, > > > On Fri, Mar 1, 2024 at 4:21 PM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > On Friday, March 1, 2024 2:11 PM Masahiko Sawada > > wrote: > > > >

Re: Proposal to add page headers to SLRU pages

2024-03-06 Thread Alvaro Herrera
Hello, I suppose this is important to do if we ever want to move SLRUs into shared buffers. However, I wonder about the extra time this adds to pg_upgrade. Is this something we should be concerned about? Is there any measurement/estimates to tell us how long this would be? Right now, if you

Re: Combine Prune and Freeze records emitted by vacuum

2024-03-06 Thread Heikki Linnakangas
On 25/01/2024 00:49, Melanie Plageman wrote: Generates 30% fewer WAL records and 12% fewer WAL bytes -- which, depending on what else is happening on the system, can lead to vacuum spending substantially less time on WAL writing and syncing (often 15% less time on WAL writes and 10% less time on

Re: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-03-06 Thread Amit Kapila
On Tue, Mar 5, 2024 at 7:59 PM Давыдов Виталий wrote: > > Thank you for the reply. > > On Tuesday, March 05, 2024 12:05 MSK, Heikki Linnakangas > wrote: > > > In a nutshell, this changes PREPARE TRANSACTION so that if > synchronous_commit is 'off', the PREPARE TRANSACTION is not fsync'd to >

Re: Stack overflow issue

2024-03-06 Thread Alexander Korotkov
On Wed, Feb 14, 2024 at 2:00 PM Alexander Korotkov wrote: > On Fri, Jan 12, 2024 at 11:00 PM Robert Haas wrote: > > On Fri, Jan 12, 2024 at 10:12 AM Heikki Linnakangas wrote: > > > Here's one goto-free attempt. It adds a local loop to where the > > > recursion was, so that if you have a chain

Re: Fix race condition in InvalidatePossiblyObsoleteSlot()

2024-03-06 Thread Bharath Rupireddy
On Wed, Mar 6, 2024 at 4:51 PM Michael Paquier wrote: > > On Wed, Mar 06, 2024 at 09:17:58AM +, Bertrand Drouvot wrote: > > Right, somehow out of context here. > > We're not yet in the green yet, one of my animals has complained: >

Re: Proposal to add page headers to SLRU pages

2024-03-06 Thread Li, Yong
Rebase the patch against the latest HEAD. Regards, Yong slru_page_header_v5.patch Description: slru_page_header_v5.patch

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Wed, Mar 6, 2024 at 8:25 PM John Naylor wrote: > > Actually, I forgot -- I had one more question: Masahiko, is there a > reason for this extra local variable, which uses the base type, rather > than the typedef'd parameter? > > +RT_SCOPE RT_RADIX_TREE * > +RT_ATTACH(dsa_area *dsa, RT_HANDLE

Re: remaining sql/json patches

2024-03-06 Thread Himanshu Upadhyaya
On Tue, Mar 5, 2024 at 6:52 AM Amit Langote wrote: Hi, I am doing some random testing with the latest patch and found one scenario that I wanted to share. consider a below case. ‘postgres[102531]=#’SELECT * FROM JSON_TABLE(jsonb '{ "id" : 12345678901, "FULL_NAME" : "JOHN

Re: Potential stack overflow in incremental base backup

2024-03-06 Thread Alvaro Herrera
On 2024-Mar-06, Thomas Munro wrote: > Even on the heap, 16GB is too much to assume we can allocate during a > base backup. I don't claim that's a real-world problem for > incremental backup right now in master, because I don't have any > evidence that anyone ever really uses --with-segsize (do

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
Actually, I forgot -- I had one more question: Masahiko, is there a reason for this extra local variable, which uses the base type, rather than the typedef'd parameter? +RT_SCOPE RT_RADIX_TREE * +RT_ATTACH(dsa_area *dsa, RT_HANDLE handle) +{ + RT_RADIX_TREE *tree; + dsa_pointer control; + + tree

Re: speed up a logical replica setup

2024-03-06 Thread vignesh C
On Sat, 2 Mar 2024 at 02:19, Euler Taveira wrote: > > On Thu, Feb 22, 2024, at 12:45 PM, Hayato Kuroda (Fujitsu) wrote: > > Based on idea from Euler, I roughly implemented. Thought? > > 0001-0013 were not changed from the previous version. > > V24-0014: addressed your comment in the replied

Re: Fix race condition in InvalidatePossiblyObsoleteSlot()

2024-03-06 Thread Michael Paquier
On Wed, Mar 06, 2024 at 09:17:58AM +, Bertrand Drouvot wrote: > Right, somehow out of context here. We're not yet in the green yet, one of my animals has complained: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hachi=2024-03-06%2010%3A10%3A03 This is telling us that the xmin

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Tue, Mar 5, 2024 at 11:12 PM Masahiko Sawada wrote: > > > I'd like to push 0001 and 0002 shortly, and then do another sweep over > > 0003, with remaining feedback, and get that in so we get some > > buildfarm testing before the remaining polishing work on > > tidstore/vacuum. > > Sounds a

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-06 Thread Amit Kapila
On Wed, Mar 6, 2024 at 2:47 PM Bharath Rupireddy wrote: > > > Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change. > @@ -1629,6 +1634,20 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, } } break; + case RS_INVAL_INACTIVE_TIMEOUT: + if

Re: Statistics Import and Export

2024-03-06 Thread Matthias van de Meent
On Wed, 6 Mar 2024 at 11:33, Stephen Frost wrote: > On Wed, Mar 6, 2024 at 11:07 Matthias van de Meent > wrote: >> Or even just one VALUES for the whole statistics loading? > > > I don’t think we’d want to go beyond one relation at a time as then it can be > parallelized, we won’t be trying to

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-06 Thread Amit Kapila
On Mon, Mar 4, 2024 at 3:14 AM Nathan Bossart wrote: > > On Sun, Mar 03, 2024 at 11:40:00PM +0530, Bharath Rupireddy wrote: > > On Sat, Mar 2, 2024 at 3:41 AM Nathan Bossart > > wrote: > >> Would you ever see "conflict" as false and "invalidation_reason" as > >> non-null for a logical slot? > >

Re: Adding deprecation notices to pgcrypto documentation

2024-03-06 Thread Daniel Gustafsson
> On 6 Mar 2024, at 10:57, Peter Eisentraut wrote: > > On 05.03.24 11:50, Daniel Gustafsson wrote: >>> * Should we actually document the exact list of algorithms along with >>> detailed reasons? This list seems prone to becoming outdated. >> If we don't detail the list then I think that it's

Re: pipe_read_line for reading arbitrary strings

2024-03-06 Thread Daniel Gustafsson
> On 6 Mar 2024, at 11:46, Alvaro Herrera wrote: > > On 2024-Mar-06, Daniel Gustafsson wrote: > >> Good catch, that's an incorrect copy/paste, it should use ERRCODE_NO_DATA. >> I'm >> not convinced that a function to read from a pipe should consider not reading >> anything successful by

Re: pipe_read_line for reading arbitrary strings

2024-03-06 Thread Alvaro Herrera
On 2024-Mar-06, Daniel Gustafsson wrote: > Good catch, that's an incorrect copy/paste, it should use ERRCODE_NO_DATA. > I'm > not convinced that a function to read from a pipe should consider not reading > anything successful by default, output is sort expected here. We could add a > flag

Re: Infinite loop in XLogPageRead() on standby

2024-03-06 Thread Alexander Kukushkin
Hi Kyotaro, Oh, now I understand what you mean. Is the retry supposed to happen only when we are reading the very first page from the WAL file? On Wed, 6 Mar 2024 at 09:57, Kyotaro Horiguchi wrote: > > xlogrecovery.c: > @@ -3460,8 +3490,10 @@ retry: > * responsible for the validation.

Re: Proposal to include --exclude-extension Flag in pg_dump

2024-03-06 Thread Dean Rasheed
On Mon, 1 Jan 2024 at 13:28, Ayush Vatsa wrote: > > According to the documentation of pg_dump when the --extension option is not > specified, all non-system extensions in the target database will get dumped. > > Why do we need to explicitly exclude extensions? > Hence to include only a few we

Re: Statistics Import and Export

2024-03-06 Thread Stephen Frost
Greetings, On Wed, Mar 6, 2024 at 11:07 Matthias van de Meent < boekewurm+postg...@gmail.com> wrote: > On Fri, 1 Mar 2024, 04:55 Corey Huinker, wrote: > >> Also per our prior discussion- this makes sense to include in post-data > section, imv, and also because then we have the indexes we may

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-06 Thread Bertrand Drouvot
Hi, On Wed, Mar 06, 2024 at 02:46:57PM +0530, Bharath Rupireddy wrote: > On Wed, Mar 6, 2024 at 2:42 PM Bertrand Drouvot > wrote: > > Yeah, I'm okay with one column. > > Thanks. v8-0001 is how it looks. Please see the v8 patch set with this change. Thanks! A few comments: 1 === + The

Re: Regarding the order of the header file includes

2024-03-06 Thread Bharath Rupireddy
On Wed, Mar 6, 2024 at 3:02 PM Richard Guo wrote: > > I think we generally follow the rule that we include 'postgres.h' or > 'postgres_fe.h' first, followed by system header files, and then > postgres header files ordered in ASCII value. I noticed that in some C > files we fail to follow this

Re: Statistics Import and Export

2024-03-06 Thread Matthias van de Meent
On Fri, 1 Mar 2024, 04:55 Corey Huinker, wrote: >> Also per our prior discussion- this makes sense to include in post-data >> section, imv, and also because then we have the indexes we may wish to load >> stats for, but further that also means it’ll be in the paralleliziable part >> of the

RE: speed up a logical replica setup

2024-03-06 Thread Hayato Kuroda (Fujitsu)
Dear Euler, Thanks for updating the patch! >v24-0003: as I said I don't think we need to add it, however, I won't fight >against it if people want to add this check. OK, let's wait comments from senior members. >Since I applied v24-0004, I realized that extra start / stop service are

Re: Reduce useless changes before reassembly during logical replication

2024-03-06 Thread li jie
Hi, Sorry I replied too late. > This is all true but note that in successful cases (where the table is > published) all the work done by FilterByTable(accessing caches, > transaction-related stuff) can add noticeable overhead as anyway we do > that later in pgoutput_change(). You are correct.

Re: Adding deprecation notices to pgcrypto documentation

2024-03-06 Thread Peter Eisentraut
On 05.03.24 11:50, Daniel Gustafsson wrote: * Should we actually document the exact list of algorithms along with detailed reasons? This list seems prone to becoming outdated. If we don't detail the list then I think that it's not worth doing, doing the research isn't entirely trivial as

Re: pipe_read_line for reading arbitrary strings

2024-03-06 Thread Daniel Gustafsson
> On 6 Mar 2024, at 10:07, Peter Eisentraut wrote: > > On 22.11.23 13:47, Alvaro Herrera wrote: >> On 2023-Mar-07, Daniel Gustafsson wrote: >>> The attached POC diff replace fgets() with pg_get_line(), which may not be >>> an >>> Ok way to cross the streams (it's clearly not a great fit), but

Regarding the order of the header file includes

2024-03-06 Thread Richard Guo
I think we generally follow the rule that we include 'postgres.h' or 'postgres_fe.h' first, followed by system header files, and then postgres header files ordered in ASCII value. I noticed that in some C files we fail to follow this rule strictly. Attached is a patch to fix this. Back in 2019,

Re: Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-06 Thread Jelte Fennema-Nio
This patch would definitely be useful for Citus. We indeed currently copy all of that code into our own explain hook. And it seems we actually have some bug. Because the es->memory branches were not copied (probably because this code didn't exist when we copied it).

Re: Injection points: some tools to wait and wake

2024-03-06 Thread Jelte Fennema-Nio
On Wed, 6 Mar 2024 at 07:17, Michael Paquier wrote: > I'm open to that if there's enough demand for it, but I > don't know how much we should accomodate with the existing > requirements of contrib/ for something that's only developer-oriented. There's quite a few developer-oriented GUCs as well,

Re: Fix race condition in InvalidatePossiblyObsoleteSlot()

2024-03-06 Thread Bertrand Drouvot
Hi, On Wed, Mar 06, 2024 at 02:47:55PM +0900, Michael Paquier wrote: > On Tue, Mar 05, 2024 at 10:17:03AM +, Bertrand Drouvot wrote: > > On Tue, Mar 05, 2024 at 09:42:20AM +0900, Michael Paquier wrote: > > The issue is that then the new ASSERT is > > triggered leading to the standby shutdown.

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-06 Thread Bharath Rupireddy
On Wed, Mar 6, 2024 at 2:42 PM Bertrand Drouvot wrote: > > Hi, > > On Tue, Mar 05, 2024 at 01:44:43PM -0600, Nathan Bossart wrote: > > On Wed, Mar 06, 2024 at 12:50:38AM +0530, Bharath Rupireddy wrote: > > > On Mon, Mar 4, 2024 at 2:11 PM Bertrand Drouvot > > > wrote: > > >> On Sun, Mar 03, 2024

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-06 Thread Bertrand Drouvot
Hi, On Tue, Mar 05, 2024 at 01:44:43PM -0600, Nathan Bossart wrote: > On Wed, Mar 06, 2024 at 12:50:38AM +0530, Bharath Rupireddy wrote: > > On Mon, Mar 4, 2024 at 2:11 PM Bertrand Drouvot > > wrote: > >> On Sun, Mar 03, 2024 at 03:44:34PM -0600, Nathan Bossart wrote: > >> > Unless I am

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:40 PM Masahiko Sawada wrote: > > On Wed, Mar 6, 2024 at 5:33 PM John Naylor wrote: > > I've looked around and it seems clang is more lax on conversions. > > Since it works fine for clang, I think we just need a cast here for > > gcc. I've attached a blind attempt at a

Re: pipe_read_line for reading arbitrary strings

2024-03-06 Thread Peter Eisentraut
On 22.11.23 13:47, Alvaro Herrera wrote: On 2023-Mar-07, Daniel Gustafsson wrote: The attached POC diff replace fgets() with pg_get_line(), which may not be an Ok way to cross the streams (it's clearly not a great fit), but as a POC it provided a neater interface for reading one-off lines from

Re: Missing LWLock protection in pgstat_reset_replslot()

2024-03-06 Thread Bertrand Drouvot
Hi, On Wed, Mar 06, 2024 at 10:24:46AM +0530, shveta malik wrote: > On Tue, Mar 5, 2024 at 6:52 PM Bertrand Drouvot > wrote: > Thanks. Can we try to get rid of multiple LwLockRelease in > pgstat_reset_replslot(). Is this any better? > > /* > -* Nothing to do for physical slots

Re: Add missing error codes to PANIC/FATAL error reports in xlog.c and relcache.c

2024-03-06 Thread Daniel Gustafsson
> On 26 Feb 2024, at 13:42, Nazir Bilal Yavuz wrote: > All of your feedback is addressed in v2. Nothing sticks out from reading through these patches, they seem quite ready to me. Being able to filter and analyze on errorcodes is likely to be more important going forward as more are running

Re: Infinite loop in XLogPageRead() on standby

2024-03-06 Thread Kyotaro Horiguchi
At Tue, 5 Mar 2024 09:36:44 +0100, Alexander Kukushkin wrote in > Please find attached the patch fixing the problem and the updated TAP test > that addresses Nit. Record-level retries happen when the upper layer detects errors. In my previous mail, I cited code that is intended to prevent this

Re: Synchronizing slots from primary to standby

2024-03-06 Thread Amit Kapila
On Wed, Mar 6, 2024 at 12:07 PM Masahiko Sawada wrote: > > On Fri, Mar 1, 2024 at 3:22 PM Peter Smith wrote: > > > > On Fri, Mar 1, 2024 at 5:11 PM Masahiko Sawada > > wrote: > > > > > ... > > > +/* > > > + * "*" is not accepted as in that case primary will not be able > > >

Re: MERGE ... RETURNING

2024-03-06 Thread Peter Eisentraut
On 29.02.24 20:49, Jeff Davis wrote: To summarize, most of the problem has been in retrieving the action (INSERT/UPDATE/DELETE) taken or the WHEN-clause number applied to a particular matched row. The reason this is important is because the row returned is the old row for a DELETE action, and

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Wed, Mar 6, 2024 at 5:33 PM John Naylor wrote: > > On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: > > > > ../../src/include/port/simd.h:326:71: error: incompatible type for > > argument 1 of \342\200\230vshrq_n_s8\342\200\231 > > uint8x16_t masked = vandq_u8(vld1q_u8(mask),

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: > > ../../src/include/port/simd.h:326:71: error: incompatible type for > argument 1 of \342\200\230vshrq_n_s8\342\200\231 > uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7)); >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:06 PM John Naylor wrote: > > (Hmm, I thought we had run this code on Arm already...) CI MacOS uses Clang on aarch64, which has been working fine. The failing animals are on gcc 7.3...

Re: Transaction timeout

2024-03-06 Thread Andrey M. Borodin
> On 25 Feb 2024, at 21:50, Alexander Korotkov wrote: > > Thank you for the patches. I've pushed the 0001 patch to avoid > further failures on buildfarm. Let 0004 wait till injections points > by Mechael are committed. Thanks! All prerequisites are committed. I propose something in a line

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Andres Freund
Hi, On March 6, 2024 9:06:50 AM GMT+01:00, John Naylor wrote: >On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: >> >> On Wed, Mar 6, 2024 at 4:41 PM Andres Freund wrote: > >> > A few ARM buildfarm animals are complaining: >> > >> >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread John Naylor
On Wed, Mar 6, 2024 at 3:02 PM Masahiko Sawada wrote: > > On Wed, Mar 6, 2024 at 4:41 PM Andres Freund wrote: > > A few ARM buildfarm animals are complaining: > > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=parula=2024-03-06%2007%3A34%3A02 > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2024-03-06 Thread Masahiko Sawada
On Wed, Mar 6, 2024 at 4:41 PM Andres Freund wrote: > > Hi, > > On 2024-03-05 16:41:30 +0700, John Naylor wrote: > > I'd like to push 0001 and 0002 shortly, and then do another sweep over > > 0003, with remaining feedback, and get that in so we get some > > buildfarm testing before the remaining

<    1   2