Re: refactoring relation extension and BufferAlloc(), faster COPY

2023-02-28 Thread Andres Freund
Hi, On 2023-02-21 17:33:31 +0100, Alvaro Herrera wrote: > On 2023-Feb-21, Heikki Linnakangas wrote: > > > > +static BlockNumber > > > +BulkExtendSharedRelationBuffered(Relation rel, > > > + SMgrRelation > > > smgr, > > > +

Combine pg_walinspect till_end_of_wal functions with others

2023-02-28 Thread Bharath Rupireddy
Hi, In a recent discussion [1], Michael Paquier asked if we can combine pg_walinspect till_end_of_wal functions with other functions pg_get_wal_records_info and pg_get_wal_stats. The code currently looks much duplicated and the number of functions that pg_walinspect exposes to the users is

Re: Add LZ4 compression in pg_dump

2023-02-28 Thread Michael Paquier
On Tue, Feb 28, 2023 at 05:58:34PM -0600, Justin Pryzby wrote: > I found that e9960732a broke writing of empty gzip-compressed data, > specifically LOs. pg_dump succeeds, but then the restore fails: The number of issues you have been reporting here begins to worries me.. How many of them have

Re: meson: Add equivalent of configure --disable-rpath option

2023-02-28 Thread Peter Eisentraut
On 22.02.23 13:56, Peter Eisentraut wrote: The configure option --disable-rpath currently has no equivalent in meson.  This option is used by packagers, so I think it would be good to have it in meson as well.  I came up with the attached patch. committed

Re: Track Oldest Initialized WAL Buffer Page

2023-02-28 Thread Bharath Rupireddy
On Wed, Mar 1, 2023 at 9:49 AM Nathan Bossart wrote: > > On Tue, Feb 28, 2023 at 11:12:29AM +0530, Bharath Rupireddy wrote: > > On Tue, Feb 28, 2023 at 5:52 AM Nathan Bossart > > wrote: > >> It's confusing to me that OldestInitializedPage is set to NewPageBeginPtr. > >> Doesn't that set it to

Re: pg_upgrade and logical replication

2023-02-28 Thread Julien Rouhaud
On Wed, Mar 01, 2023 at 11:51:49AM +0530, Amit Kapila wrote: > On Tue, Feb 28, 2023 at 10:18 AM Julien Rouhaud wrote: > > > > Well, as I mentioned I'm *not* interested in a logical-replication-only > > scenario. Logical replication is nice but it will always be less efficient > > than physical

Re: Commitfest 2023-03 starting tomorrow!

2023-02-28 Thread Michael Paquier
On Tue, Feb 28, 2023 at 01:45:27PM -0500, Greg Stark wrote: > So I'm not sure if I'll be CFM this month but I'm assuming I will be > at this point Okay, that's OK for me! Thanks for helping out. > The next pass would be to grab any patches not marked Ready for > Committer and if they look

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-28 Thread John Naylor
On Tue, Feb 28, 2023 at 10:09 PM Masahiko Sawada wrote: > > On Tue, Feb 28, 2023 at 10:20 PM Masahiko Sawada wrote: > > > > On Tue, Feb 28, 2023 at 3:42 PM John Naylor > > wrote: > > > > > > > > > On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada < sawada.m...@gmail.com> wrote: > > > > > > > >

Re: LOG: invalid record length at : wanted 24, got 0

2023-02-28 Thread Bharath Rupireddy
On Wed, Mar 1, 2023 at 10:51 AM Harinath Kanchu wrote: > > Hello, > > We are seeing an interesting STANDBY behavior, that’s happening once in 3-4 > days. > > The standby suddenly disconnects from the primary, and it throws the error > “LOG: invalid record length at : wanted 24, got0”. Firstly,

Re: add PROCESS_MAIN to VACUUM

2023-02-28 Thread Michael Paquier
On Thu, Jan 19, 2023 at 11:08:07AM -0800, Nathan Bossart wrote: > rebased PROCESS_TOAST has that: /* sanity check for PROCESS_TOAST */ if ((params->options & VACOPT_FULL) != 0 && (params->options & VACOPT_PROCESS_TOAST) == 0) ereport(ERROR,

Re: pg_upgrade and logical replication

2023-02-28 Thread Amit Kapila
On Tue, Feb 28, 2023 at 10:18 AM Julien Rouhaud wrote: > > On Tue, Feb 28, 2023 at 08:56:37AM +0530, Amit Kapila wrote: > > On Tue, Feb 28, 2023 at 7:55 AM Julien Rouhaud wrote: > > > > > > > > > The scenario I'm interested in is to rely on logical replication only for > > > the > > > upgrade,

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Amit Kapila
On Wed, Mar 1, 2023 at 10:57 AM Masahiko Sawada wrote: > > On Wed, Mar 1, 2023 at 1:55 PM Amit Kapila wrote: > > > > > Won't a malicious user can block the > > replication in other ways as well and let the publisher stall (or > > crash the publisher) even without setting min_send_delay?

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Masahiko Sawada
On Wed, Mar 1, 2023 at 1:55 PM Amit Kapila wrote: > > On Wed, Mar 1, 2023 at 8:18 AM Masahiko Sawada wrote: > > > > On Wed, Mar 1, 2023 at 12:51 AM Hayato Kuroda (Fujitsu) > > wrote: > > > > Thinking of side effects of this feature (no matter where we delay > > applying the changes), on the

LOG: invalid record length at : wanted 24, got 0

2023-02-28 Thread Harinath Kanchu
Hello, We are seeing an interesting STANDBY behavior, that’s happening once in 3-4 days. The standby suddenly disconnects from the primary, and it throws the error “LOG: invalid record length at : wanted 24, got0”. And then it tries to restore the WAL file from the archive. Due to low write

Re: Support logical replication of global object commands

2023-02-28 Thread Zheng Li
On Fri, Feb 17, 2023 at 4:48 AM Amit Kapila wrote: > > On Fri, Feb 17, 2023 at 10:58 AM Zheng Li wrote: > > > > > > > Actually, I intend something for global objects. But the main thing > > > > > that is worrying me about this is that we don't have a clean way to > > > > > untie global object

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Amit Kapila
On Tue, Feb 28, 2023 at 9:21 PM Hayato Kuroda (Fujitsu) wrote: > > > 1. > > + /* > > + * If we've requested to shut down, exit the process. > > + * > > + * Note that WalSndDone() cannot be used here because the delaying > > + * changes will be sent in the function. > > + */ > > + if

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Amit Kapila
On Wed, Mar 1, 2023 at 8:18 AM Masahiko Sawada wrote: > > On Wed, Mar 1, 2023 at 12:51 AM Hayato Kuroda (Fujitsu) > wrote: > > Thinking of side effects of this feature (no matter where we delay > applying the changes), on the publisher, vacuum cannot collect garbage > and WAL cannot be recycled.

Re: Normalization of utility queries in pg_stat_statements

2023-02-28 Thread Michael Paquier
On Mon, Feb 20, 2023 at 11:34:59AM +0900, Michael Paquier wrote: > With the patches.. Attached is an updated patch set, where I have done more refactoring work for the regression tests of pg_stat_statements, splitting pg_stat_statments.sql into the following files: - user_activity.sql for the

Re: stopgap fix for signal handling during restore_command

2023-02-28 Thread Nathan Bossart
On Sun, Feb 26, 2023 at 12:12:27PM -0800, Andres Freund wrote: > On 2023-02-26 11:39:00 -0800, Nathan Bossart wrote: >> What precisely did you have in mind? AFAICT you are asking for a wrapper >> around write(). > > Partially I just want something that can easily be searched for, that can have >

Re: Weird failure with latches in curculio on v15

2023-02-28 Thread Nathan Bossart
On Sat, Feb 25, 2023 at 11:00:31AM -0800, Andres Freund wrote: > TBH, I think the current archive and restore module APIs aren't useful. I > think it was a mistake to add archive modules without having demonstrated that > one can do something useful with them that the restore_command didn't

Re: Track Oldest Initialized WAL Buffer Page

2023-02-28 Thread Nathan Bossart
On Tue, Feb 28, 2023 at 11:12:29AM +0530, Bharath Rupireddy wrote: > On Tue, Feb 28, 2023 at 5:52 AM Nathan Bossart > wrote: >> It's confusing to me that OldestInitializedPage is set to NewPageBeginPtr. >> Doesn't that set it to the beginning of the newest initialized page? > > Yes, that's the

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-02-28 Thread Nathan Bossart
On Tue, Feb 28, 2023 at 10:38:31AM +0530, Bharath Rupireddy wrote: > On Tue, Feb 28, 2023 at 6:14 AM Nathan Bossart > wrote: >> Why do we only read a page at a time in XLogReadFromBuffersGuts()? What is >> preventing us from copying all the data we need in one go? > > Note that most of the

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Amit Kapila
On Wed, Mar 1, 2023 at 8:06 AM Kyotaro Horiguchi wrote: > > At Tue, 28 Feb 2023 08:35:11 +0530, Amit Kapila > wrote in > > On Tue, Feb 28, 2023 at 8:14 AM Kyotaro Horiguchi > > wrote: > > > > > > At Mon, 27 Feb 2023 14:56:19 +0530, Amit Kapila > > > wrote in > > > > The one difference w.r.t

Re: Auth extensions, with an LDAP/SCRAM example [was: Proposal: Support custom authentication methods using hooks]

2023-02-28 Thread Stephen Frost
Greetings, * Jacob Champion (jchamp...@timescale.com) wrote: > On Mon, Feb 27, 2023 at 12:43 PM Stephen Frost wrote: > > * Jacob Champion (jchamp...@timescale.com) wrote: > > > This patchset should ideally have required zero client side changes, but > > > because our SCRAM implementation is

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Masahiko Sawada
On Wed, Mar 1, 2023 at 12:51 AM Hayato Kuroda (Fujitsu) wrote: > > Dear Amit, > > > Few comments: > > Thank you for reviewing! PSA new version. > Note that the starting point of delay for 2PC was not changed, > I think it has been under discussion. > > > 1. > > + /* > > + * If we've requested to

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Kyotaro Horiguchi
At Tue, 28 Feb 2023 08:35:11 +0530, Amit Kapila wrote in > On Tue, Feb 28, 2023 at 8:14 AM Kyotaro Horiguchi > wrote: > > > > At Mon, 27 Feb 2023 14:56:19 +0530, Amit Kapila > > wrote in > > > The one difference w.r.t recovery_min_apply_delay is that here we will > > > hold locks for the

Re: [Proposal] Add foreign-server health checks infrastructure

2023-02-28 Thread Katsuragi Yuta
Hi Kuroda-san, Thank you for updating the patch! I rethought the pqSocketPoll part. Current interpretation of arguments seems a little bit confusing because a specific pattern of arguments has a different meaning. What do you think about introducing a new argument like `int forConnCheck`?

Re: Non-superuser subscription owners

2023-02-28 Thread Stephen Frost
Greetings, * Jeff Davis (pg...@j-davis.com) wrote: > On Tue, 2023-02-28 at 08:37 -0500, Robert Haas wrote: > > The existing SECURITY_RESTRICTED_OPERATION flag basically prevents > > you > > from tinkering with the session state. > > Currently, every time we set that flag we also run all the code

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Peter Smith
Here are some review comments for v9-0001, but these are only very trivial. == Commit Message 1. Nitpick. The new text is jagged-looking. It should wrap at ~80 chars. ~~~ 2. 2. Another reason is for that parallel streaming, the transaction will be opened immediately by the parallel apply

Re: Doc update for pg_stat_statements normalization

2023-02-28 Thread Michael Paquier
On Wed, Mar 01, 2023 at 12:43:40AM +, Imseih (AWS), Sami wrote: > Yes, that makes sense. Okay, thanks. Done that now on HEAD. -- Michael signature.asc Description: PGP signature

Proposal: %T Prompt parameter for psql for current time (like Oracle has)

2023-02-28 Thread Kirk Wolak
I cannot get the last email to show up for the commitfest. This is version 2 of the original patch. [1] Thanks Jim! [1] https://postgr.es/m/CACLU5mSRwHr_8z%3DenMj-nXF1tmC7%2BJn5heZQNiKuLyxYUtL2fg%40mail.gmail.com Regards Kirk. From b9db157177bbdeeeb6d35c3623ca9355141419d7 Mon Sep 17 00:00:00

Re: Minimal logical decoding on standbys

2023-02-28 Thread Jeff Davis
On Mon, 2023-02-27 at 09:40 +0100, Drouvot, Bertrand wrote: > Please find attached V51 tiny rebase due to a6cd1fc692 (for 0001) and > 8a8661828a (for 0005). [ Jumping into this thread late, so I apologize if these comments have already been covered. ] Regarding v51-0004: * Why is the CV sleep

Re: Proposal: %T Prompt parameter for psql for current time (like Oracle has)

2023-02-28 Thread Kirk Wolak
On Thu, Feb 23, 2023 at 2:05 PM Kirk Wolak wrote: > On Thu, Feb 23, 2023 at 9:52 AM Tom Lane wrote: > >> Heikki Linnakangas writes: >> > On 23/02/2023 13:20, Peter Eisentraut wrote: >> >> If you don't have \timing turned on before the query starts, psql won't >> >> record what the time was

Re: Doc update for pg_stat_statements normalization

2023-02-28 Thread Imseih (AWS), Sami
> + > + Queries on which normalization can be applied may be observed with constant > + values in pg_stat_statements, especially when there > + is a high rate of entry deallocations. To reduce the likelihood of this > + happening, consider increasing pg_stat_statements.max. > + The

Re: pg_upgrade and logical replication

2023-02-28 Thread Julien Rouhaud
On Tue, Feb 28, 2023 at 08:02:13AM -0800, Nikolay Samokhvalov wrote: > On Fri, Feb 17, 2023 at 7:35 AM Julien Rouhaud wrote: > > > > Any table later added in the > > publication is either already fully replicated until that LSN on the > > upgraded > > node, so only the delta is needed, or has

Re: Doc update for pg_stat_statements normalization

2023-02-28 Thread Michael Paquier
On Tue, Feb 28, 2023 at 11:11:30PM +, Imseih (AWS), Sami wrote: > I agree. We introduce the concept of a plannable statement in a > previous section and we can then make this distinction in the new > paragraph. > > I also added a link to pg_stat_statements_info since that is introduced >

Re: Add LZ4 compression in pg_dump

2023-02-28 Thread Justin Pryzby
On Thu, Feb 23, 2023 at 09:24:46PM +0100, Tomas Vondra wrote: > On 2/23/23 16:26, Tomas Vondra wrote: > > Thanks for v30 with the updated commit messages. I've pushed 0001 after > > fixing a comment typo and removing (I think) an unnecessary change in an > > error message. > > > > I'll give the

Re: [PATCH] Add `verify-system` sslmode to use system CA pool for server cert

2023-02-28 Thread Jacob Champion
On 2/16/23 10:38, Jacob Champion wrote: > Thanks for the nudge, v7 rebases over the configure conflict from 9244c11afe. I think/hope this is well-baked enough for a potential commit this CF, so I've adjusted the target version. Let me know if there are any concerns about the approach. Thanks!

Re: Ability to reference other extensions by schema in extension scripts

2023-02-28 Thread Sandro Santilli
The following review has been posted through the commitfest application: make installcheck-world: tested, failed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed I've applied the patch attached to message

Re: Beautify pg_walinspect docs a bit

2023-02-28 Thread Michael Paquier
On Tue, Feb 28, 2023 at 11:57:40AM -0800, Nathan Bossart wrote: > It looks like 58597ed accidentally added an "end_lsn" to the docs for > pg_get_wal_stats_till_end_of_wal(). Indeed. Fixed, thanks! -- Michael signature.asc Description: PGP signature

Re: [PoC] Let libpq reject unexpected authentication requests

2023-02-28 Thread Jacob Champion
On 2/16/23 10:57, Jacob Champion wrote: > v14 rebases over the test and solution conflicts from 9244c11afe2. Since we're to the final CF for PG16, here's a rough summary. This patchset provides two related features: 1) the ability for a client to explicitly allow or deny particular methods of

Re: Refactoring SysCacheGetAttr to know when attr cannot be NULL

2023-02-28 Thread Tom Lane
Daniel Gustafsson writes: > The attached refactoring introduce SysCacheGetAttrNotNull as a wrapper around > SysCacheGetAttr where a NULL value triggers an elog(). +1, seems like a good idea. (I didn't review the patch details.) > This will reduce granularity of error messages, and as the patch

Re: Marking options deprecated in help output

2023-02-28 Thread Daniel Gustafsson
> On 24 Feb 2023, at 21:31, Heikki Linnakangas wrote: > On 05/12/2022 11:42, Daniel Gustafsson wrote: > How about putting the deprecated option on a separate line like this: > >> -h, --host=HOSTNAMEdatabase server host or socket directory >> -H (same as -h,

Re: Doc update for pg_stat_statements normalization

2023-02-28 Thread Imseih (AWS), Sami
> I am OK with an addition to the documentation to warn that one may > have to increase the maximum number of entries that can be stored if > seeing a non-normalized entry that should have been normalized. I agree. We introduce the concept of a plannable statement in a previous section and we

Re: Stale references to guc.c in comments/tests

2023-02-28 Thread Tom Lane
Daniel Gustafsson writes: > Specifying the GUCs in question is a good idea, done in the attached. I'm not > sure the phrasing is spot-on though, but I can't think of a better one. If > you > can think of a better one I'm all ears. I'd just change "the definition of" to "the definitions of".

RE: Ability to reference other extensions by schema in extension scripts

2023-02-28 Thread Regina Obe
> On Sun, Feb 26, 2023 at 01:39:24AM -0500, Regina Obe wrote: > > > > 1) Just don't allow any extensions referenced by other > > >extensions to be relocatable. > > > > Attached is my revision 3 patch, which follows the proposed #1. > > Don't allow schema relocation of an extension if another

Re: Stale references to guc.c in comments/tests

2023-02-28 Thread Daniel Gustafsson
> On 27 Feb 2023, at 17:59, Tom Lane wrote: > The grammar is a bit off ("the GUC definition" would read better), > but really I think the wording was vague already and we should tighten > it up. Can we specify exactly which GUC variable(s) we're talking about? Specifying the GUCs in question

Re: RFC: logical publication via inheritance root?

2023-02-28 Thread Jacob Champion
Hi, I'm going to register this in CF for feedback. Summary for potential reviewers: we don't use declarative partitions in the Timescale partitioning scheme, but it'd be really nice to be able to replicate between our tables and standard tables, or between two Timescale-partitioned tables with

Re: Ability to reference other extensions by schema in extension scripts

2023-02-28 Thread Sandro Santilli
On Sun, Feb 26, 2023 at 01:39:24AM -0500, Regina Obe wrote: > > 1) Just don't allow any extensions referenced by other > >extensions to be relocatable. > > Attached is my revision 3 patch, which follows the proposed #1. > Don't allow schema relocation of an extension if another extension >

Re: Ability to reference other extensions by schema in extension scripts

2023-02-28 Thread Sandro Santilli
On Sat, Feb 25, 2023 at 03:40:24PM -0500, Regina Obe wrote: > > On Mon, Feb 06, 2023 at 05:19:39AM -0500, Regina Obe wrote: > > > > I was thinking: how about using the "refobjsubid" to encode the "level" of > > dependency on an extension ? Right now "refobjsubid" is always 0 when the > >

Making empty Bitmapsets always be NULL

2023-02-28 Thread Tom Lane
When I designed the Bitmapset module, I set things up so that an empty Bitmapset could be represented either by a NULL pointer, or by an allocated object all of whose bits are zero. I've recently come to the conclusion that that was a bad idea and we should instead have a convention like the

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-02-28 Thread Alexander Korotkov
On Wed, Mar 1, 2023 at 12:03 AM Gregory Stark wrote: > It looks like this patch received some feedback from Andres and hasn't > had any further work posted. I'm going to move it to "Waiting on > Author". I'll post the updated version in the next couple of days. > It doesn't sound like this is

Re: Non-superuser subscription owners

2023-02-28 Thread Jeff Davis
On Tue, 2023-02-28 at 08:37 -0500, Robert Haas wrote: > The existing SECURITY_RESTRICTED_OPERATION flag basically prevents > you > from tinkering with the session state. Currently, every time we set that flag we also run all the code as the table owner. You're suggesting using the

Re: Non-superuser subscription owners

2023-02-28 Thread Jeff Davis
On Tue, 2023-02-28 at 11:28 -0800, Andres Freund wrote: > I can only repeat myself in stating that SECURITY DEFINER solves none > of the > relevant issues. I included several examples of why it doesn't in the > recent > thread about "blocking SECURITY INVOKER". E.g. that default arguments > of >

Re: WIN32 pg_import_system_collations

2023-02-28 Thread Andrew Dunstan
On 2023-02-28 Tu 11:40, Juan José Santamaría Flecha wrote: On Tue, Feb 28, 2023 at 12:55 PM Andrew Dunstan wrote: On 2023-02-27 Mo 17:20, Juan José Santamaría Flecha wrote: Hmm, yeah. I'm not sure I understand the point of this test anyway: SELECT to_char(date '2010-03-01',

Refactoring SysCacheGetAttr to know when attr cannot be NULL

2023-02-28 Thread Daniel Gustafsson
Today we have two fairly common patterns around extracting an attr from a cached tuple: a = SysCacheGetAttr(OID, tuple, Anum_pg_foo_bar, ); Assert(!isnull); a = SysCacheGetAttr(OID, tuple, Anum_pg_foo_bar, ); if (isnull) elog(ERROR, ".."); The error message in the elog() cases also

Re: Beautify pg_walinspect docs a bit

2023-02-28 Thread Nathan Bossart
It looks like 58597ed accidentally added an "end_lsn" to the docs for pg_get_wal_stats_till_end_of_wal(). diff --git a/doc/src/sgml/pgwalinspect.sgml b/doc/src/sgml/pgwalinspect.sgml index 22677e54f2..3d7cdb95cc 100644 --- a/doc/src/sgml/pgwalinspect.sgml +++ b/doc/src/sgml/pgwalinspect.sgml @@

Re: Memory leak from ExecutorState context?

2023-02-28 Thread Tomas Vondra
On 2/28/23 19:06, Jehan-Guillaume de Rorthais wrote: > Hello all, > > A customer is facing out of memory query which looks similar to this > situation: > > > https://www.postgresql.org/message-id/flat/12064.1555298699%40sss.pgh.pa.us#eb519865575bbc549007878a5fb7219b > > This PostgreSQL

Re: Auth extensions, with an LDAP/SCRAM example [was: Proposal: Support custom authentication methods using hooks]

2023-02-28 Thread Jacob Champion
On Mon, Feb 27, 2023 at 12:43 PM Stephen Frost wrote: > * Jacob Champion (jchamp...@timescale.com) wrote: > > This patchset should ideally have required zero client side changes, but > > because our SCRAM implementation is slightly nonstandard too -- it > > doesn't embed the username into the

Re: Non-superuser subscription owners

2023-02-28 Thread Andres Freund
Hi, On 2023-02-22 09:18:34 -0800, Jeff Davis wrote: > I can't resist mentioning that these are all SECURITY INVOKER problems. > SECURITY INVOKER is insecure unless the invoker absolutely trusts the > definer, and that only really makes sense if the definer is a superuser > (or something very

Re: Experiments with Postgres and SSL

2023-02-28 Thread Jacob Champion
On Tue, Feb 28, 2023 at 10:33 AM Greg Stark wrote: > On Wed, 22 Feb 2023 at 07:27, Heikki Linnakangas wrote: > > Good idea. Do we want to just require the protocol to be "postgres", or > > perhaps "postgres/3.0"? Need to register that with IANA, I guess. > > I had never heard of this before, it

Re: Memory leak from ExecutorState context?

2023-02-28 Thread Tomas Vondra
On 2/28/23 19:25, Justin Pryzby wrote: > On Tue, Feb 28, 2023 at 07:06:43PM +0100, Jehan-Guillaume de Rorthais wrote: >> Hello all, >> >> A customer is facing out of memory query which looks similar to this >> situation: >> >> >>

Commitfest 2023-03 starting tomorrow!

2023-02-28 Thread Greg Stark
So I'm not sure if I'll be CFM this month but I'm assuming I will be at this point Regardless, Commitfest 2023-03 starts tomorrow! So this is a good time to check your submitted patches and ensure they're actually in building and don't need a rebase. Take a look at http://cfbot.cputube.org/

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-02-28 Thread Andres Freund
Hi, On 2023-02-25 16:00:05 +0530, Amit Kapila wrote: > On Tue, Feb 21, 2023 at 7:55 PM Önder Kalacı wrote: > >> I think this overhead seems to be mostly due to the need to perform > >> tuples_equal multiple times for duplicate values. I think more work needs to be done to determine the source

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-02-28 Thread Kuntal Ghosh
On Tue, Feb 28, 2023 at 10:38 AM Bharath Rupireddy wrote: > +/* + * Guts of XLogReadFromBuffers(). + * + * Read 'count' bytes into 'buf', starting at location 'ptr', from WAL + * fetched WAL buffers on timeline 'tli' and return the read bytes. + */ s/fetched WAL buffers/fetched from WAL buffers

Re: Experiments with Postgres and SSL

2023-02-28 Thread Greg Stark
On Wed, 22 Feb 2023 at 07:27, Heikki Linnakangas wrote: > > On 20/01/2023 03:28, Jacob Champion wrote: > > On Wed, Jan 18, 2023 at 7:16 PM Greg Stark wrote: > >> * "Service Mesh" type tools that hide multiple services behind a > >> single host/port ("Service Mesh" is just a new buzzword for

Re: Memory leak from ExecutorState context?

2023-02-28 Thread Justin Pryzby
On Tue, Feb 28, 2023 at 07:06:43PM +0100, Jehan-Guillaume de Rorthais wrote: > Hello all, > > A customer is facing out of memory query which looks similar to this > situation: > > > https://www.postgresql.org/message-id/flat/12064.1555298699%40sss.pgh.pa.us#eb519865575bbc549007878a5fb7219b >

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-02-28 Thread Heikki Linnakangas
On 28/02/2023 16:17, Maxim Orlov wrote: Either I do not understand something, or the files from pg_commit_ts directory are not copied. Huh, yeah you're right, pg_upgrade doesn't copy pg_commit_ts. - Heikki

Re: WIN32 pg_import_system_collations

2023-02-28 Thread Juan José Santamaría Flecha
On Tue, Feb 28, 2023 at 12:55 PM Andrew Dunstan wrote: > On 2023-02-27 Mo 17:20, Juan José Santamaría Flecha wrote: > > > Hmm, yeah. I'm not sure I understand the point of this test anyway: > > > SELECT to_char(date '2010-03-01', 'DD TMMON ' COLLATE "de_DE"); > Uhm, they probably don't make

Re: pg_upgrade and logical replication

2023-02-28 Thread Nikolay Samokhvalov
On Fri, Feb 17, 2023 at 7:35 AM Julien Rouhaud wrote: > > Any table later added in the > publication is either already fully replicated until that LSN on the upgraded > node, so only the delta is needed, or has been created after that LSN. In the > latter case, the entirety of the table will be

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Hayato Kuroda (Fujitsu)
Dear Amit, > Few comments: Thank you for reviewing! PSA new version. Note that the starting point of delay for 2PC was not changed, I think it has been under discussion. > 1. > + /* > + * If we've requested to shut down, exit the process. > + * > + * Note that WalSndDone() cannot be used here

Re: Maybe we can remove the type cast in typecache.c

2023-02-28 Thread Tom Lane
qinghao huang writes: > When I was reading postgres code, I found there is a wierd type cast. I'm > pondering if it is necessary. > ``` > /* Allocate a new typmod number. This will be wasted if we error out. */ > typmod = (int) > >

Avoid multiple SetLatch() calls in procsignal_sigusr1_handler()

2023-02-28 Thread Bharath Rupireddy
Hi, Most of the multiplexed SIGUSR1 handlers are setting latch explicitly when the procsignal_sigusr1_handler() can do that for them at the end. These multiplexed handlers are currently being used as SIGUSR1 handlers, not as independent handlers, so no problem if SetLatch() is removed from them.

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-28 Thread Masahiko Sawada
On Tue, Feb 28, 2023 at 10:20 PM Masahiko Sawada wrote: > > On Tue, Feb 28, 2023 at 3:42 PM John Naylor > wrote: > > > > > > On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada > > wrote: > > > > > > On Wed, Feb 22, 2023 at 6:55 PM John Naylor > > > wrote: > > > > > > > > That doesn't seem

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2023-02-28 Thread Damir Belyalov
Hello Tested patch on all cases: CIM_SINGLE, CIM_MULTI, CIM_MULTI_CONDITION. As expected it works. Also added a description to copy.sgml and made a review on patch. I added 'ignored_errors' integer parameter that should be output after the option is finished. All errors were added to the system

Re: Allow logical replication to copy tables in binary format

2023-02-28 Thread Melih Mutlu
Hi, Attached v11. vignesh C , 28 Şub 2023 Sal, 13:59 tarihinde şunu yazdı: > Thanks for the patch, Few comments: > 1) Are primary key required for the tables, if not required we could > remove the primary key which will speed up the test by not creating > the indexes and inserting in the

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-02-28 Thread Maxim Orlov
On Mon, 27 Feb 2023 at 12:10, Heikki Linnakangas wrote: > (CC list trimmed, gmail wouldn't let me send otherwise) > > That sounds right for pg_serial, pg_notify, and pg_subtrans. But not for > pg_commit_ts and the pg_multixacts. > > This needs tests for pg_upgrading those SLRUs, after 0, 1 and N

Re: Adding CommandID to heap xlog records

2023-02-28 Thread Heikki Linnakangas
I took another stab at this from a different angle, and tried to use this to simplify logical decoding. The theory was that if we included the command ID in the WAL records, we wouldn't need the separate HEAP2_NEW_CID record anymore, and could remove much of the code in reorderbuffer.c that's

Re: Non-superuser subscription owners

2023-02-28 Thread Robert Haas
On Mon, Feb 27, 2023 at 7:37 PM Jeff Davis wrote: > > Yeah. That's the idea I was floating, at least. > > Isn't that a hard problem; maybe impossible? It doesn't seem that hard to me; maybe I'm missing something. The existing SECURITY_RESTRICTED_OPERATION flag basically prevents you from

Re: Allow logical replication to copy tables in binary format

2023-02-28 Thread Jelte Fennema
> 3. I think the newly added tests must verify if the binary COPY is > picked up when enabled. Perhaps, looking at the publisher's server log > for 'COPY ... WITH BINARY format'? Maybe it's an overkill, otherwise, > we have no way of testing that the option took effect. Another way to test that

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-02-28 Thread Masahiko Sawada
On Tue, Feb 28, 2023 at 3:42 PM John Naylor wrote: > > > On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada > wrote: > > > > On Wed, Feb 22, 2023 at 6:55 PM John Naylor > > wrote: > > > > > > That doesn't seem useful to me. If we've done enough testing to reassure > > > us the new way always

Re: pg_rewind: warn when checkpoint hasn't happened after promotion

2023-02-28 Thread James Coleman
On Mon, Feb 27, 2023 at 2:33 AM Heikki Linnakangas wrote: > > On 16/11/2022 07:17, kuroda.keis...@nttcom.co.jp wrote: > >> The issue here is pg_rewind looks into control file to determine the > >> soruce timeline, because the control file is not updated until the > >> first checkpoint ends after

Re: Add shared buffer hits to pg_stat_io

2023-02-28 Thread Drouvot, Bertrand
Hi, On 2/25/23 9:16 PM, Melanie Plageman wrote: Hi, As suggested in [1], the attached patch adds shared buffer hits to pg_stat_io. Thanks for the patch! BufferDesc * LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, -bool

Re: Allow logical replication to copy tables in binary format

2023-02-28 Thread Bharath Rupireddy
On Tue, Feb 28, 2023 at 1:22 AM Melih Mutlu wrote: > > Hi, > > Thanks for all of your reviews! > > So, I made some changes in the v10 according to your comments. Thanks. Some quick comments on v10: 1. + + If true, initial data synchronization will be performed in binary format +

Re: WIN32 pg_import_system_collations

2023-02-28 Thread Andrew Dunstan
On 2023-02-27 Mo 17:20, Juan José Santamaría Flecha wrote: El lun, 27 feb 2023, 23:05, Juan José Santamaría Flecha escribió: The command that's failing is "SET lc_time TO 'de_DE';", and that area of code is untouched by this patch. As mentioned in [1], the problem seems to

Re: Allow logical replication to copy tables in binary format

2023-02-28 Thread vignesh C
On Tue, 28 Feb 2023 at 01:22, Melih Mutlu wrote: > > Hi, > > Thanks for all of your reviews! > > So, I made some changes in the v10 according to your comments. > > 1- copy_format is changed to a boolean parameter copy_binary. > It sounds easier to use a boolean to enable/disable binary copy.

Re: Make some xlogreader messages more accurate

2023-02-28 Thread Jeevan Ladhe
+1 for the changes. >1. Why is "wanted >=%u" any better than "wanted at least %u"? IMO, the >wording as opposed to >= symbol in the user-facing messages works >better. I think I agree with Bharath on this: "wanted at least %u" sounds better for user error than "wanted >=%u". Regards, Jeevan

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-02-28 Thread Kartyshov Ivan
Intro== The main purpose of the feature is to achieve read-your-writes-consistency, while using async replica for reads and primary for writes. In that case lsn of last modification is stored inside application. We cannot store this lsn inside database, since reads are distributed

Maybe we can remove the type cast in typecache.c

2023-02-28 Thread qinghao huang
Hi hackers, When I was reading postgres code, I found there is a wierd type cast. I'm pondering if it is necessary. ``` /* Allocate a new typmod number. This will be wasted if we error out. */ typmod = (int) pg_atomic_fetch_add_u32(>shared_typmod_registry->next_typmod,

Re: Track IO times in pg_stat_io

2023-02-28 Thread Drouvot, Bertrand
Hi, On 2/26/23 5:03 PM, Melanie Plageman wrote: Hi, As suggested in [1], the attached patch adds IO times to pg_stat_io; Thanks for the patch! I started to have a look at it and figured out that a tiny rebase was needed (due to 728560db7d and b9f0e54bc9), so please find the rebase (aka V2)

Re: Provide PID data for "cannot wait on a latch owned by another process" in latch.c

2023-02-28 Thread Michael Paquier
On Tue, Feb 28, 2023 at 08:18:16AM +0100, Peter Eisentraut wrote: > I would also have asked for some kind of prefix that introduces the numbers. Okay. > I wonder what these numbers are useful for though? Is this a development > aid? Yes. > Can you do anything with these numbers? Yes. They

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

2023-02-28 Thread Bharath Rupireddy
On Thu, Feb 23, 2023 at 3:03 AM Melanie Plageman wrote: > > Hi, > > So, I attached a rough implementation of both the autovacuum failsafe > reverts to shared buffers and the vacuum option (no tests or docs or > anything). Thanks for the patches. I have some comments. 0001: 1. I don't quite

Add documentation for coverage reports with meson

2023-02-28 Thread Michael Paquier
Hi all, I have mentioned on a different thread of -docs that we have no documentation to achieve $subject, so attached is a patch to add something. This can be done with the following steps: meson setup -Db_coverage=true .. blah ninja meson test ninja coverage-html As far as I can see, there is

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-02-28 Thread Amit Kapila
On Mon, Feb 27, 2023 at 2:21 PM Hayato Kuroda (Fujitsu) wrote: > Few comments: 1. + /* + * If we've requested to shut down, exit the process. + * + * Note that WalSndDone() cannot be used here because the delaying + * changes will be sent in the function. + */ + if (got_STOPPING) + { +

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

2023-02-28 Thread Bharath Rupireddy
On Thu, Jan 12, 2023 at 6:06 AM Andres Freund wrote: > > On 2023-01-11 17:26:19 -0700, David G. Johnston wrote: > > Should we just add "ring_buffers" to the existing "shared_buffers" and > > "temp_buffers" settings? > > The different types of ring buffers have different sizes, for good reasons.