Re: Improving worst-case merge join performance with often-null foreign key

On Sun, Apr 23, 2023 at 11:30 AM Richard Guo wrote: > On Sat, Apr 22, 2023 at 11:21 PM Tom Lane wrote: >> Hmm. I don't entirely understand why the existing stop-at-nulls logic >> in nodeMergejoin.c didn't fix this for you. Maybe somebody has broken >> that? See the commentary for MJEvalOuterVa

Re: pg_recvlogical prints bogus error when interrupted

On Thu, Apr 27, 2023 at 11:24:52AM +0530, Bharath Rupireddy wrote: > IMO, +1 for HEAD/PG16 and +0.5 for backpatching as it may not be so > critical to backpatch all the way down. What may happen without this > patch is that the output file isn't fsync-ed upon SIGINT/SIGTERM. > Well, is it a critica

Re: Autogenerate some wait events code and documentation

On Wed, Apr 26, 2023 at 08:36:44PM +0200, Drouvot, Bertrand wrote: > Please find attached V5 addressing the previous comments except > the "ordering" one (need to look deeper at this). I was putting my hands into that, and I see now what you mean here.. Among the nine types of wait events, Lock, L

Re: pg_recvlogical prints bogus error when interrupted

On Tue, Apr 11, 2023 at 11:42 AM Michael Paquier wrote: > > On Mon, Oct 24, 2022 at 08:15:11AM +0530, Bharath Rupireddy wrote: > > The attached patch (pg_recvlogical_graceful_interrupt.text) has a > > couple of problems, I believe. We're losing prepareToTerminate() with > > keepalive true and we'r

Re: proposal: psql: show current user in prompt

Hi rebased version + fix warning possibly uninitialized variable Regards Pavel diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index b11d9a6ba3..f774ffa310 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -5401,6 +5401,43 @@ psql "dbname=postgres r

Re: Add LZ4 compression in pg_dump

On Wed, Apr 26, 2023 at 08:50:46AM +, gkokola...@pm.me wrote: > For what is worth, I think this would be the best approach. +1 Thanks. I have gone with that, then! -- Michael signature.asc Description: PGP signature

Re: Doc limitation update proposal: include out-of-line OID usage per TOAST-ed columns

On Wed, Apr 26, 2023 at 5:18 PM Jakub Wartak wrote: > OK, so here is the documentation patch proposal. I've also added two > rows touching the subject of pg_largeobjects, as it is also related to > the OIDs topic. -partition keys -32 -can be increased by recompiling PostgreSQL +

Re: Add two missing tests in 035_standby_logical_decoding.pl

On Wed, Apr 26, 2023 at 4:41 PM Drouvot, Bertrand wrote: > > On 4/26/23 12:27 PM, Alvaro Herrera wrote: > >> diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm > >> b/src/test/perl/PostgreSQL/Test/Cluster.pm > >> index 6f7f4e5de4..819667d42a 100644 > >> --- a/src/test/perl/PostgreSQL/Test/Clus

RE: Initial Schema Sync for Logical Replication

On Fri, Apr 21, 2023 at 16:48 PM Masahiko Sawada wrote: > On Thu, Apr 20, 2023 at 8:16 PM Amit Kapila wrote: > > > > On Mon, Apr 17, 2023 at 9:12 AM Masahiko Sawada > wrote: > > > > > > On Fri, Apr 7, 2023 at 6:37 PM Amit Kapila > > > wrote: > > > > > > > > On Thu, Apr 6, 2023 at 6:57 PM Masah

Re: pg_stat_io not tracking smgrwriteback() is confusing

At Wed, 26 Apr 2023 17:08:14 -0400, Melanie Plageman wrote in > On Mon, Apr 24, 2023 at 9:29 PM Melanie Plageman > wrote: > > I've yet to cook up a client backend test case (e.g. with COPY). I've taken > > that as a todo. > > It was trivial to see client backend writebacks in almost any scenar

Re: Add PQsendSyncMessage() to libpq

Hello, On 25/04/2023 15:23, Denis Laxalde wrote: > This sounds like a useful addition to me. I've played a bit with it in > Psycopg and it works fine. Thank you very much for reviewing my patch! I have attached a new version of it that addresses your comments and that has been rebased on top of t

Re: pg_stat_io not tracking smgrwriteback() is confusing

On Mon, Apr 24, 2023 at 9:29 PM Melanie Plageman wrote: > I've yet to cook up a client backend test case (e.g. with COPY). I've taken > that as a todo. It was trivial to see client backend writebacks in almost any scenario once I set backend_flush_after. I wonder if it is worth mentioning the var

Re: run pgindent on a regular basis / scripted manner

Andrew Dunstan writes: > On 2023-04-26 We 09:27, Tom Lane wrote: >> Yeah, I agree, there is no case where that doesn't suck. I don't >> mind it imposing specific placements of brackets and so on --- >> that's very analogous to what pgindent will do. But it likes to >> re-flow comma-separated lis

Re: run pgindent on a regular basis / scripted manner

On 2023-04-26 We 09:27, Tom Lane wrote: Peter Eisentraut writes: On 24.04.23 16:14, Tom Lane wrote: I certainly don't like its current behavior where adding/changing one line can have side-effects on nearby lines. But we have a proposal to clean that up, and I'm cautiously optimistic that it

Re: issue with meson builds on msys2

On 2023-04-26 We 11:30, Tom Lane wrote: Andrew Dunstan writes: If I redirect the output to a file (which is what the buildfarm client actually does), it seems like it completes successfully, but I still get a non-zero exit: pgrunner@EC2AMAZ-GCB871B UCRT64 ~/bf $ /usr/bin/perl -e 'chdir "root/H

Re: pg_stat_io for the startup process

Hi all, Robert Haas , 26 Nis 2023 Çar, 15:34 tarihinde şunu yazdı: > On Wed, Apr 26, 2023 at 5:47 AM Kyotaro Horiguchi > wrote: > > 3. When should we call pgstat_report_stats on the startup process? > > > > During recovery, I think we can call pgstat_report_stats() (or a > > subset of it) right

Re: Autogenerate some wait events code and documentation

Hi, On 4/26/23 6:51 PM, Drouvot, Bertrand wrote: Hi, On 4/25/23 7:15 AM, Michael Paquier wrote: Will do, no problem at all. Please find attached V5 addressing the previous comments except the "ordering" one (need to look deeper at this). Regards, -- Bertrand Drouvot PostgreSQL Contributor

Re: Autogenerate some wait events code and documentation

Hi, On 4/25/23 7:15 AM, Michael Paquier wrote: On Mon, Apr 24, 2023 at 09:03:53AM +0200, Drouvot, Bertrand wrote: Oh right, fixed. I may tweak a few things if I put my hands on it, but that looks pretty solid seen from here.. Glad to hear! ;-) I have spotted a few extra issues. One thin

Re: issue with meson builds on msys2

Andrew Dunstan writes: > If I redirect the output to a file (which is what the buildfarm client > actually does), it seems like it completes successfully, but I still get > a non-zero exit: > pgrunner@EC2AMAZ-GCB871B UCRT64 ~/bf > $ /usr/bin/perl -e 'chdir "root/HEAD/instkeep.2023-04-25_11-09-4

Re: Fix for visibility check on 14.5 fails on tpcc with high concurrency

Hi hackers, I was wondering whether there are any updates on the bug in visibility check introduced in version 14.5. Many thanks, Dimos [ServiceNow]

Re: issue with meson builds on msys2

On 2023-04-26 We 10:58, Tom Lane wrote: I wrote: Looking at the pg_ctl source code, the only way I can explain that printout is that do_stop called wait_for_postmaster_stop which, after one or more loops, exited via one of its exit() calls. Ah, a little too hasty there: it's get_pgpid() that h

Re: issue with meson builds on msys2

I wrote: > Looking at the pg_ctl source code, the only way I can explain that > printout is that do_stop called wait_for_postmaster_stop which, > after one or more loops, exited via one of its exit() calls. Ah, a little too hasty there: it's get_pgpid() that has to be reaching an exit().

Re: issue with meson builds on msys2

Andrew Dunstan writes: >> For some reason which makes no sense to me the buildfarm animal fails >> at the first Stop-Db step. The DB is actually stopped, but pg_ctl >> returns a non-zero status. The thing that's really odd is that meson >> isn't at all involved in this step. But it's happened e

Re: Add two missing tests in 035_standby_logical_decoding.pl

Hi, On 4/26/23 11:58 AM, Yu Shi (Fujitsu) wrote: On Mon, Apr 24, 2023 8:07 PM Drouvot, Bertrand wrote: I think that's because when replaying a checkpoint record, the startup process of standby only saves the information of the checkpoint, and we need to wait for the checkpointer to perform

Re: vector search support

Hi Nathan, I find the patches really interesting. Personally, as Data/MLOps Engineer, I'm involved in a project where we use embedding techniques to generate vectors from documents, and use clustering and kNN searches to find similar documents basing on spatial neighbourhood of generated vectors.

Re: run pgindent on a regular basis / scripted manner

Peter Eisentraut writes: > On 24.04.23 16:14, Tom Lane wrote: >> I certainly don't like its current behavior where adding/changing one >> line can have side-effects on nearby lines. But we have a proposal >> to clean that up, and I'm cautiously optimistic that it'll be better >> in future. Did y

Re: [PATCH] Compression dictionaries for JSONB

Hi Nikita, > The External TOAST pointer is very limited to the amount of service data > it could keep, that's why we introduced the Custom TOAST pointers in the > Pluggable TOAST. But keep in mind that changing the TOAST pointer > structure requires a lot of quite heavy modifications in the core -

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

On Wed, Apr 26, 2023 at 8:31 AM Daniel Gustafsson wrote: > > On 26 Apr 2023, at 13:26, David Rowley wrote: > > On Wed, 26 Apr 2023, 8:48 pm Masahiko Sawada, > wrote: > > > It works but I think we might want to add the unit kB for > > understandability and consisten

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

Hi, > I agree that we can't simply widen varatt_external to use 8 bytes for > the toast ID in all cases. +1 Note that the user may have a table with multiple TOASTable attributes. If we simply widen the TOAST pointer it may break the existing tables in the edge case. Also this may be a reason wh

Re: pg_stat_io for the startup process

On Wed, Apr 26, 2023 at 5:47 AM Kyotaro Horiguchi wrote: > 3. When should we call pgstat_report_stats on the startup process? > > During recovery, I think we can call pgstat_report_stats() (or a > subset of it) right before invoking WaitLatch and at segment > boundaries. I think this kind of idea

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

> On 26 Apr 2023, at 13:26, David Rowley wrote: > On Wed, 26 Apr 2023, 8:48 pm Masahiko Sawada, > wrote: > It works but I think we might want to add the unit kB for > understandability and consistency with other GUC_UNIT_KB parameters. > I've attached a small patch

Re: Find dangling membership roles in pg_dumpall

> On 26 Apr 2023, at 13:02, Daniel Gustafsson wrote: >> On 26 Apr 2023, at 12:18, Andreas 'ads' Scherbaum wrote: >> The attached patch fixes this problem, and updates prev_remaining inside >> the loop. > > Nice catch, that indeed seems like a proper fix. This was introduced in > ce6b672e44 and

Re: [PATCH] Extend the length of BackgroundWorker.bgw_library_name

Hi, > You're absolutely right. Here's v3. Please avoid using top posting [1]. The commit message may require a bit of tweaking by the committer but other than that the patch seems to be fine. I'm going to mark it as RfC in a bit unless anyone objects. [1]: https://wiki.postgresql.org/wiki/Maili

RE: [PoC] pg_upgrade: allow to upgrade publisher node

Dear Peter, > A suggestion: You could write some/most tests against test_decoding > rather than the publication/subscription system. That way, you can > avoid many timing issues in the tests and you can check more exactly > that the slots produce the output you want. This would also help ensure

Re: Doc limitation update proposal: include out-of-line OID usage per TOAST-ed columns

On Sun, 23 Apr 2023, 3:42 am Gurjeet Singh, wrote: > I anticipate that edits to Appendix K Postgres Limits will prompt > improving the note in there about the maximum column limit, That note > is too wordy, and sometimes confusing, especially for the audience > that it's written for: newcomers to

RE: Support logical replication of DDLs

On Wednesday, April 26, 2023 2:32 PM Masahiko Sawada > Subject: Re: Support logical replication of DDLs > > On Tue, Mar 28, 2023 at 3:22 PM houzj.f...@fujitsu.com > wrote: > > > > On Tuesday, March 28, 2023 1:41 PM Amit Kapila > wrote: > > > > > > On Mon, Mar 27, 2023 at 5:37 PM Amit Kapila >

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

On Wed, 26 Apr 2023, 8:48 pm Masahiko Sawada, wrote: > I realized that the value of vacuum_buffer_usage_limit parameter in > postgresql.conf.sample doesn't have the unit: > > #vacuum_buffer_usage_limit = 256 # size of vacuum and analyze buffer > access strategy ring. >

Re: Perform streaming logical transactions by background workers and parallel apply

On Wed, Apr 26, 2023 at 4:11 PM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, April 26, 2023 5:00 PM Alexander Lakhin > wrote: > > Thanks for reporting the issue. > > I think the problem is that it tried to release locks in > logicalrep_worker_onexit() before the initialization of the process is

Re: Add two missing tests in 035_standby_logical_decoding.pl

Hi, On 4/26/23 11:12 AM, vignesh C wrote: On Wed, 26 Apr 2023 at 13:45, Drouvot, Bertrand There was one typo in the commit message, subscribtion should be subscription, the rest of the changes looks good to me: Subject: [PATCH v5] Add subscribtion to the standby test in 035_standby_logical_de

Re: Add two missing tests in 035_standby_logical_decoding.pl

Hi, On 4/26/23 12:27 PM, Alvaro Herrera wrote: diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index 6f7f4e5de4..819667d42a 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -2644,7 +2644,16 @

Re: Find dangling membership roles in pg_dumpall

> On 26 Apr 2023, at 12:18, Andreas 'ads' Scherbaum wrote: > > > Hello, > > pg_dumpall.c has a function dumpRoleMembership() which dumps all > membership roles. This function includes a piece of code which checks > if the membership tree has an open end which can't be resolved. > However that c

RE: Perform streaming logical transactions by background workers and parallel apply

On Wednesday, April 26, 2023 5:00 PM Alexander Lakhin wrote: > Please look at a new anomaly that can be observed starting from 216a7848. > > The following script: > echo "CREATE SUBSCRIPTION testsub CONNECTION 'dbname=nodb' > PUBLICATION testpub WITH (connect = false); > ALTER SUBSCRIPTION tests

Re: Add two missing tests in 035_standby_logical_decoding.pl

> diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm > b/src/test/perl/PostgreSQL/Test/Cluster.pm > index 6f7f4e5de4..819667d42a 100644 > --- a/src/test/perl/PostgreSQL/Test/Cluster.pm > +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm > @@ -2644,7 +2644,16 @@ sub wait_for_catchup > } >

Re: Doc limitation update proposal: include out-of-line OID usage per TOAST-ed columns

Hi, >> These 2 discussions show that it's a painful experience to run into >> this problem, and that the hackers have ideas on how to fix it, but >> those fixes haven't materialized for years. So I would say that, yes, >> this info belongs in the hard-limits section, because who knows how >> long

Find dangling membership roles in pg_dumpall

Hello, pg_dumpall.c has a function dumpRoleMembership() which dumps all membership roles. This function includes a piece of code which checks if the membership tree has an open end which can't be resolved. However that code is never used. The variable prev_remaining is initially set to 0, and t

RE: Add two missing tests in 035_standby_logical_decoding.pl

On Mon, Apr 24, 2023 8:07 PM Drouvot, Bertrand wrote: > > On 4/24/23 11:45 AM, Amit Kapila wrote: > > On Mon, Apr 24, 2023 at 11:54 AM Amit Kapila > wrote: > >> > >> On Mon, Apr 24, 2023 at 11:24 AM Drouvot, Bertrand > >> wrote: > >>> > >> > >> Few comments: > >> > >> > > > > +# W

Re: pg_stat_io for the startup process

At Tue, 25 Apr 2023 16:04:23 -0700, Andres Freund wrote in > I refreshed my memory: The startup process has indeed behaved that way for > much longer than pg_stat_io existed - but it's harder to spot, because the > stats are more coarsely aggregated :/. And it's very oddly inconsistent: > > The

Re: Support logical replication of DDLs

On Wed, 26 Apr 2023 at 12:02, Masahiko Sawada wrote: > > On Tue, Mar 28, 2023 at 3:22 PM houzj.f...@fujitsu.com > wrote: > > > > On Tuesday, March 28, 2023 1:41 PM Amit Kapila > > wrote: > > > > > > On Mon, Mar 27, 2023 at 5:37 PM Amit Kapila > > > wrote: > > > > > > > > On Mon, Mar 27, 2023 a

Re: Add two missing tests in 035_standby_logical_decoding.pl

On Wed, 26 Apr 2023 at 13:45, Drouvot, Bertrand wrote: > > Hi, > > On 4/26/23 6:06 AM, vignesh C wrote: > > On Tue, 25 Apr 2023 at 12:51, Drouvot, Bertrand > > wrote: > > Thanks for the updated patch. > > Few comments: > > Thanks for looking at it! > > > 1) subscriber_stdout and subscriber_stder

Re: Perform streaming logical transactions by background workers and parallel apply

Hello hackers, Please look at a new anomaly that can be observed starting from 216a7848. The following script: echo "CREATE SUBSCRIPTION testsub CONNECTION 'dbname=nodb' PUBLICATION testpub WITH (connect = false); ALTER SUBSCRIPTION testsub ENABLE;" | psql sleep 1 rm $PGINST/lib/libpqwalreceiv

Re: Support logical replication of DDLs

On Wed, Apr 26, 2023 at 12:01 PM Masahiko Sawada wrote: > > On Wed, Apr 26, 2023 at 2:56 PM Amit Kapila wrote: > > > > On Wed, Apr 26, 2023 at 10:01 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Apr 25, 2023 at 12:58 PM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > Aport from above com

Re: Add LZ4 compression in pg_dump

--- Original Message --- On Tuesday, April 25th, 2023 at 8:02 AM, Michael Paquier wrote: > > > On Wed, Apr 12, 2023 at 07:53:53PM -0500, Justin Pryzby wrote: > > > I doubt it - in the !HAVE_LIBZ case, it's currently an "if" statement > > with nothing but a comment, which isn't

Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

Hi, On Sun, Apr 16, 2023 at 9:09 AM David Rowley wrote: > > On Sat, 15 Apr 2023 at 12:59, David Rowley wrote: > > These are all valid points. I've attached a patch aiming to address > > each of them. > > I tweaked this a little further and pushed it. > I realized that the value of vacuum_buffer

Re: [pg_rewind] use the passing callback instead of global function

> On 26 Apr 2023, at 10:33, Richard Guo wrote: > > On Wed, Apr 26, 2023 at 9:51 AM Junwang Zhao > wrote: > `local_traverse_files` and `libpq_traverse_files` both have a > callback parameter but instead use the global process_source_file > which is no good for function e

Re: [pg_rewind] use the passing callback instead of global function

On Wed, Apr 26, 2023 at 9:51 AM Junwang Zhao wrote: > `local_traverse_files` and `libpq_traverse_files` both have a > callback parameter but instead use the global process_source_file > which is no good for function encapsulation. Nice catch. This should be a typo introduced by 37d2ff38. Whil

Re: Add two missing tests in 035_standby_logical_decoding.pl

Hi, On 4/26/23 6:06 AM, vignesh C wrote: On Tue, 25 Apr 2023 at 12:51, Drouvot, Bertrand wrote: Thanks for the updated patch. Few comments: Thanks for looking at it! 1) subscriber_stdout and subscriber_stderr are not required for this test case, we could remove it, I was able to remove tho

Re: run pgindent on a regular basis / scripted manner

On 24.04.23 16:14, Tom Lane wrote: Peter Eisentraut writes: Does anyone find perltidy useful? To me, it functions more like a JavaScript compiler in that once you process the source code, it is no longer useful for manual editing. If we are going to have the buildfarm check indentation and th

Re: [PoC] pg_upgrade: allow to upgrade publisher node

On 24.04.23 14:03, Hayato Kuroda (Fujitsu) wrote: so at least there's a good chance that they will still be at shutdown, and will therefore send all the data to the subscribers? Having a regression tests for that scenario would also be a good idea. Having an uncommitted write transaction should

Re: Overhauling "Routine Vacuuming" docs, particularly its handling of freezing

On Tue, Apr 25, 2023 at 4:58 AM Peter Geoghegan wrote: > > There are also very big structural problems with "Routine Vacuuming", > that I also propose to do something about. Honestly, it's a huge mess > at this point. It's nobody's fault in particular; there has been > accretion after accretion ad