Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-12 Thread Michael Paquier
On Mon, Jul 11, 2022 at 02:26:46PM +0200, Matthias van de Meent wrote: > Thanks for reviewing. I think that v6 is over-engineered because there should be no need to add a check in xlogreader.c as long as the origin of the problem is blocked, no? And the origin here is when the record is

Re: Reducing the chunk header sizes on all memory context types

2022-07-12 Thread Andres Freund
Hi, On 2022-07-12 10:42:07 -0700, Andres Freund wrote: > On 2022-07-12 17:01:18 +1200, David Rowley wrote: > > There is at least one. It might be major; to reduce the AllocSet chunk > > header from 16 bytes down to 8 bytes I had to get rid of the freelist > > pointer that was reusing the "aset"

Re: Reducing the chunk header sizes on all memory context types

2022-07-12 Thread David Rowley
On Wed, 13 Jul 2022 at 05:42, Andres Freund wrote: > > There is at least one. It might be major; to reduce the AllocSet chunk > > header from 16 bytes down to 8 bytes I had to get rid of the freelist > > pointer that was reusing the "aset" field in the chunk header struct. > > This works now by

Re: Reducing the chunk header sizes on all memory context types

2022-07-12 Thread David Rowley
On Wed, 13 Jul 2022 at 05:44, Andres Freund wrote: > On 2022-07-12 20:22:57 +0300, Yura Sokolov wrote: > > I don't get, why "large chunk" needs additional fields for size and > > offset. > > Large allocation sizes are certainly rounded to page size. > > And allocations which doesn't fit 1GB we

Re: minor change for create_list_bounds()

2022-07-12 Thread David Rowley
On Thu, 30 Jun 2022 at 11:41, Nathan Bossart wrote: > > On Tue, Mar 08, 2022 at 11:05:10AM -0800, Zhihong Yu wrote: > > I was looking at commit db632fbca and noticed that, > > in create_list_bounds(), if index is added to boundinfo->interleaved_parts > > in the first if statement, there is no

Re: PATCH: Add Table Access Method option to pgbench

2022-07-12 Thread Michel Pelletier
On Thu, 30 Jun 2022 at 18:09, Michael Paquier wrote: > On Fri, Jul 01, 2022 at 10:06:49AM +0900, Michael Paquier wrote: > > And the conclusion back then is that one can already achieve this by > > using PGOPTIONS: > > PGOPTIONS='-c default_table_access_method=wuzza' pgbench [...] > > > > So

Re: Perform streaming logical transactions by background workers and parallel apply

2022-07-12 Thread Peter Smith
Below are my review comments for the v16* patch set: v16-0001 1.0 There are places (comments, docs, errmsgs, etc) in the patch referring to "parallel mode". I think every one of those references should be found and renamed to "parallel streaming mode" or "streaming=parallel"

Re: Documentation about PL transforms

2022-07-12 Thread Pavel Stehule
Hi I am doing an review of this patch and I have two comments 1. I am not sure if get_call_trftypes is a good name - the prefix get_call is used when some runtime data is processed. This function just returns reformatted data from the system catalogue. Maybe get_func_trftypes_list, or just

Re: PG15 beta1 sort performance regression due to Generation context change

2022-07-12 Thread David Rowley
On Tue, 12 Jul 2022 at 17:15, David Rowley wrote: > So far only Robert has raised concerns with this regression for PG15 > (see [2]). Tom voted for leaving things as they are for PG15 in [3]. > John agrees, as quoted above. Does anyone else have any opinion? Let me handle this slightly

Re: remove_useless_groupby_columns is too enthusiastic

2022-07-12 Thread David Rowley
On Wed, 13 Jul 2022 at 05:31, Tom Lane wrote: > I tried the attached quick-hack patch that just prevents > remove_useless_groupby_columns from removing anything that > appears in ORDER BY. That successfully fixes the complained-of > case, and it doesn't change any existing regression test

Re: making relfilenodes 56 bits

2022-07-12 Thread Dilip Kumar
On Tue, Jul 12, 2022 at 7:21 PM Robert Haas wrote: > > In this version, I also removed the struct padding, changed the limit > on the number of entries to a nice round 64, and made some comment > updates. I considered trying to go further and actually make the file > variable-size, so that we

Re: proposal: Allocate work_mem From Pool

2022-07-12 Thread Joseph D Wagner
Before I try to answer that, I need to know how the scheduler works. As I understand the term used, there is no scheduler inside Postgres for user connections -- they're handled by the OS kernel. Then, I'm probably using the wrong term. Right now, I have max_worker_processes set to 16. What

Re: POC: GROUP BY optimization

2022-07-12 Thread David Rowley
On Thu, 31 Mar 2022 at 12:19, Tomas Vondra wrote: > Pushed, after going through the patch once more, running check-world > under valgrind, and updating the commit message. I'm just in this general area of the code again today and wondered about the header comment for the preprocess_groupclause()

Re: "ERROR: latch already owned" on gharial

2022-07-12 Thread Sandeep Thakkar
Thanks Robert. We are receiving the alerts from buildfarm-admins for anole and gharial not reporting. Who can help to stop these? Thanks On Wed, Jul 6, 2022 at 1:27 AM Robert Haas wrote: > On Sun, Jul 3, 2022 at 11:51 PM Thomas Munro > wrote: > > On Wed, Jun 1, 2022 at 12:55 AM Robert Haas >

Re: Add --{no-,}bypassrls flags to createuser

2022-07-12 Thread Michael Paquier
On Thu, May 26, 2022 at 04:47:46PM +0900, Kyotaro Horiguchi wrote: > FWIW, the "fancy" here causes me to think about something likely to > cause syntax breakage of the query to be sent. > > createuser -a 'user"1' -a 'user"2' 'user"3' > createuser -v "2023-1-1'; DROP TABLE public.x; select '" hoge

Re: enable/disable broken for statement triggers on partitioned tables

2022-07-12 Thread Amit Langote
Hi, On Fri, Jul 8, 2022 at 3:44 AM Dmitry Koval wrote: > I've looked through the code and everything looks good. > But there is one thing I doubt. > Patch changes result of test: > > > create function trig_nothing() returns trigger language plpgsql >as $$ begin return null; end $$; >

Re: Some clean-up work in get_cheapest_group_keys_order()

2022-07-12 Thread David Rowley
On Wed, 13 Jul 2022 at 13:12, Tom Lane wrote: > > David Rowley writes: > > On Wed, 13 Jul 2022 at 11:02, Tom Lane wrote: > >> Agreed, but I think there are other instances of that idiom that > >> should be cleaned up while you're at it. > > > Agreed. I imagine we should just do the remaining

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-07-12 Thread Kyotaro Horiguchi
At Tue, 12 Jul 2022 19:18:22 -0700, Andres Freund wrote in > Hi, > > On 2022-07-13 11:00:07 +0900, Kyotaro Horiguchi wrote: > > I imagined to use B_INVALID as a kind of "default" partition, which > > accepts all unknown backend types. > > There shouldn't be any unknown backend types. Something

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-07-12 Thread Andres Freund
Hi, On 2022-07-13 11:00:07 +0900, Kyotaro Horiguchi wrote: > I imagined to use B_INVALID as a kind of "default" partition, which > accepts all unknown backend types. There shouldn't be any unknown backend types. Something has gone wrong if we get far without a backend type set. > We can just

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-07-12 Thread Kyotaro Horiguchi
At Tue, 12 Jul 2022 12:19:06 -0400, Melanie Plageman wrote in > > + > > >io_ops.stats[backend_type_get_idx(MyBackendType)]; > > > > backend_type_get_idx(x) is actually (x - 1) plus assertion on the > > value range. And the only use-case is here. There's an reverse > > function and also used

Re: DropRelFileLocatorBuffers

2022-07-12 Thread Kyotaro Horiguchi
At Tue, 12 Jul 2022 10:30:20 -0400, Robert Haas wrote in > On Tue, Jul 12, 2022 at 12:07 AM Dilip Kumar wrote: > > I think the naming used in your patch looks better to me. So +1 for the > > change. > > Committed. Thank you, Robert and Dilip. regards. -- Kyotaro Horiguchi NTT Open Source

Re: Some clean-up work in get_cheapest_group_keys_order()

2022-07-12 Thread Tom Lane
David Rowley writes: > On Wed, 13 Jul 2022 at 11:02, Tom Lane wrote: >> Agreed, but I think there are other instances of that idiom that >> should be cleaned up while you're at it. > Agreed. I imagine we should just do the remaining cleanup in master > only. Do you agree? No objection.

Re: test_oat_hooks bug (was: Re: pgsql: Add copy/equal support for XID lists)

2022-07-12 Thread Michael Paquier
On Tue, Jul 12, 2022 at 05:20:59PM +0200, Alvaro Herrera wrote: > While looking for a place to host a test for XID lists support, I > noticed a mistake in test_oat_hooks, fixed as per the attached. Indeed. Good catch. -- Michael signature.asc Description: PGP signature

Re: Some clean-up work in get_cheapest_group_keys_order()

2022-07-12 Thread David Rowley
On Wed, 13 Jul 2022 at 11:02, Tom Lane wrote: > > David Rowley writes: > > * I think list_truncate(list_copy(list), n) is a pretty bad way to > > copy the first n elements of a list, especially when n is likely to be > > 0 most of the time. I think we should just add a function called > >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 7:59 PM Amit Kapila wrote: > > On Tue, Jul 12, 2022 at 2:53 PM Masahiko Sawada wrote: > > > > On Tue, Jul 12, 2022 at 5:58 PM shiy.f...@fujitsu.com > > wrote: > > > > > > > > > It happened when executing the following code because it tried to free a > > > NULL > > >

Re: pg15b2: large objects lost on upgrade

2022-07-12 Thread Michael Paquier
On Tue, Jul 12, 2022 at 04:51:44PM -0400, Robert Haas wrote: > I spent a bunch of time looking at this today and I have more sympathy > for Justin's previous proposal now. I found it somewhat hacky that he > was relying on the hard-coded value of LargeObjectRelationId and >

Re: [PATCH] Log details for client certificate failures

2022-07-12 Thread Jacob Champion
On Mon, Jul 11, 2022 at 6:09 AM Peter Eisentraut wrote: > I squashed those two together. I also adjusted the error message a bit > more for project style. (We can put both lines into detail.) Oh, okay. Log parsers don't have any issues with that? > I had to read up on this "ex_data" API.

Re: [PATCH] Log details for client certificate failures

2022-07-12 Thread Jacob Champion
On Sat, Jul 9, 2022 at 6:49 AM Graham Leggett wrote: > Please don’t invent another format, or try and truncate the data. This is a > huge headache when troubleshooting. I hear you, and I agree that correlating these things across machines is something we should be making easier. I'm just not

Re: Some clean-up work in get_cheapest_group_keys_order()

2022-07-12 Thread Tom Lane
David Rowley writes: > * I think list_truncate(list_copy(list), n) is a pretty bad way to > copy the first n elements of a list, especially when n is likely to be > 0 most of the time. I think we should just add a function called > list_copy_head(). We already have list_copy_tail(). Agreed, but

Some clean-up work in get_cheapest_group_keys_order()

2022-07-12 Thread David Rowley
I was rebasing a patch which requires me to make some changes in get_cheapest_group_keys_order(). I noticed a few things in there that I think we could do a little better on: * The code uses pfree() on a list and it should be using list_free() * There's a manually coded for loop over a list

Re: Extending outfuncs support to utility statements

2022-07-12 Thread Tom Lane
Peter Eisentraut writes: > This is also needed to be able to store utility statements in (unquoted) > SQL function bodies. I have some in-progress code for that that I need > to dust off. IIRC, there are still some nontrivial issues to work > through on the reading side. I don't have a

Re: making relfilenodes 56 bits

2022-07-12 Thread Robert Haas
On Tue, Jul 12, 2022 at 6:02 PM Andres Freund wrote: > MAX_FORKNUM is way lower right now. And hardcoded. So this doesn't imply a new > restriction. As we iterate over 0..MAX_FORKNUM in a bunch of places (with > filesystem access each time), it's not feasible to make that number large. Yeah.

Re: making relfilenodes 56 bits

2022-07-12 Thread Andres Freund
Hi, Please don't top quote - as mentioned a couple times recently. On 2022-07-12 23:00:22 +0200, Hannu Krosing wrote: > Re: staticAssertStmt(MAX_FORKNUM <= INT8_MAX); > > Have you really thought through making the ForkNum 8-bit ? MAX_FORKNUM is way lower right now. And hardcoded. So this

Re: Extending outfuncs support to utility statements

2022-07-12 Thread Tom Lane
I wrote: > There might be enough node types that are raw-parse-tree-only, > but not involved in utility statements, to make it worth > continuing to suppress readfuncs support for them. But I kinda > doubt it. I'll try to get some numbers later today. Granting that we want write/read support

Re: System catalog documentation chapter

2022-07-12 Thread Tom Lane
Bruce Momjian writes: > On Tue, Jul 12, 2022 at 08:56:36PM +0200, Peter Eisentraut wrote: >> views.sgml is a pretty generic name for a chapter that just contains system >> views. > Yes, I struggled with that. What made me choose "views" is that the > current name was catalogs.sgml, not

Re: Reducing Memory Consumption (aset and generation)

2022-07-12 Thread Ranier Vilela
Em ter., 12 de jul. de 2022 às 02:35, David Rowley escreveu: > On Mon, 11 Jul 2022 at 20:48, Matthias van de Meent > wrote: > > > 2) v1-002-generation-reduces-memory-consumption.patch > > > Reduces memory used by struct GenerationBlock, by minus 8 bits, > > > > That seems fairly

Re: System catalog documentation chapter

2022-07-12 Thread Bruce Momjian
On Tue, Jul 12, 2022 at 08:56:36PM +0200, Peter Eisentraut wrote: > On 08.07.22 21:32, Bruce Momjian wrote: > > On Fri, Jul 8, 2022 at 12:07:45PM -0400, Bruce Momjian wrote: > > > On Fri, Jul 8, 2022 at 11:49:47AM -0400, Tom Lane wrote: > > > > Bruce Momjian writes: > > > > > Agreed. I don't

Re: System catalog documentation chapter

2022-07-12 Thread Bruce Momjian
On Tue, Jul 12, 2022 at 08:56:01PM +0200, Peter Eisentraut wrote: > On 08.07.22 18:07, Bruce Momjian wrote: > > so I guess we can backpatch this with no issues. > > It inserts a new chapter, which would renumber all other chapters. That's a > pretty big change to backpatch. I'm against that.

Re: should check interrupts in BuildRelationExtStatistics ?

2022-07-12 Thread Tom Lane
Zhihong Yu writes: > Looking at the files under src/backend/utils/sort/, looks like license > header is missing from qsort_interruptible.c [ shrug ... ] qsort.c and qsort_arg.c don't have that either. regards, tom lane

Re: making relfilenodes 56 bits

2022-07-12 Thread Hannu Krosing
Re: staticAssertStmt(MAX_FORKNUM <= INT8_MAX); Have you really thought through making the ForkNum 8-bit ? For example this would limit a columnar storage with each column stored in it's own fork (which I'd say is not entirely unreasonable) to having just about ~250 columns. And there can easily

Re: should check interrupts in BuildRelationExtStatistics ?

2022-07-12 Thread Zhihong Yu
On Tue, Jul 12, 2022 at 1:31 PM Tom Lane wrote: > I wrote: > > Thomas Munro writes: > >> On Wed, Jul 6, 2022 at 11:37 AM Tom Lane wrote: > >>> qsort_interruptible > > >> +1 > > > So here's a patch that does it that way. > > Hearing no comments, pushed. > > regards, tom

Re: pg15b2: large objects lost on upgrade

2022-07-12 Thread Robert Haas
On Mon, Jul 11, 2022 at 9:16 AM Robert Haas wrote: > I am not saying we shouldn't try to fix this up more thoroughly, just > that I think you are overestimating the consequences. I spent a bunch of time looking at this today and I have more sympathy for Justin's previous proposal now. I found it

Re: making relfilenodes 56 bits

2022-07-12 Thread Robert Haas
On Tue, Jul 12, 2022 at 1:09 PM Andres Freund wrote: > What does currently happen if we exceed that? elog > > diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h > > index b578e2ec75..5d3775ccde 100644 > > --- a/src/include/utils/wait_event.h > > +++

Re: should check interrupts in BuildRelationExtStatistics ?

2022-07-12 Thread Tom Lane
I wrote: > Thomas Munro writes: >> On Wed, Jul 6, 2022 at 11:37 AM Tom Lane wrote: >>> qsort_interruptible >> +1 > So here's a patch that does it that way. Hearing no comments, pushed. regards, tom lane

Re: automatically generating node support functions

2022-07-12 Thread Tom Lane
Peter Eisentraut writes: > On 11.07.22 19:57, Tom Lane wrote: >> So at this point I'm rather attracted to the idea of reverting to >> a manually-maintained NodeTag enum. We know how to avoid ABI >> breakage with that, and it's not exactly the most painful part >> of adding a new node type. >

Re: Remove trailing newlines from pg_upgrade's messages

2022-07-12 Thread Tom Lane
Kyotaro Horiguchi writes: > FWIW, the following change makes sense to me according to the spec of > validate_exec()... > diff --git a/src/bin/pg_upgrade/exec.c b/src/bin/pg_upgrade/exec.c > index fadeea12ca..3cff186213 100644 > --- a/src/bin/pg_upgrade/exec.c > +++ b/src/bin/pg_upgrade/exec.c >

Re: Remove trailing newlines from pg_upgrade's messages

2022-07-12 Thread Tom Lane
Peter Eisentraut writes: > On 14.06.22 20:57, Tom Lane wrote: >> Hence, the patch below removes trailing newlines from all of >> pg_upgrade's message strings, and teaches its logging infrastructure >> to print them where appropriate. As in logging.c, there's now an >> Assert that no format

Re: WIN32 pg_import_system_collations

2022-07-12 Thread Juan José Santamaría Flecha
Please find attached a rebased version. I have split the patch into two parts trying to make it easier to review, one with the code changes and the other with the test. Other than that, there are minimal changes from the previous version to the code due to the update of _WIN32_WINNT and enabling

Re: automatically generating node support functions

2022-07-12 Thread Peter Eisentraut
On 11.07.22 19:57, Tom Lane wrote: So at this point I'm rather attracted to the idea of reverting to a manually-maintained NodeTag enum. We know how to avoid ABI breakage with that, and it's not exactly the most painful part of adding a new node type. One of the nicer features is that you now

Re: Making CallContext and InlineCodeBlock less special-case-y

2022-07-12 Thread Peter Eisentraut
On 12.07.22 01:01, Tom Lane wrote: I wrote: Peter Eisentraut writes: On 10.07.22 01:50, Tom Lane wrote: As committed, gen_node_support.pl excludes CallContext and InlineCodeBlock from getting unneeded support functions via some very ad-hoc code. Couldn't we just enable those support

Re: System catalog documentation chapter

2022-07-12 Thread Peter Eisentraut
On 08.07.22 21:32, Bruce Momjian wrote: On Fri, Jul 8, 2022 at 12:07:45PM -0400, Bruce Momjian wrote: On Fri, Jul 8, 2022 at 11:49:47AM -0400, Tom Lane wrote: Bruce Momjian writes: Agreed. I don't want to break links into the documentation in final released versions, so head and PG15 seem

Re: System catalog documentation chapter

2022-07-12 Thread Peter Eisentraut
On 08.07.22 18:07, Bruce Momjian wrote: so I guess we can backpatch this with no issues. It inserts a new chapter, which would renumber all other chapters. That's a pretty big change to backpatch. I'm against that.

Re: First draft of the PG 15 release notes

2022-07-12 Thread Bruce Momjian
On Mon, Jul 11, 2022 at 11:31:32PM -0700, Noah Misch wrote: > On Mon, Jul 11, 2022 at 12:39:57PM -0400, Bruce Momjian wrote: > > I had trouble reading the sentences in the order you used so I > > restructured it: > > > > The new default is one of the secure schema usage patterns that >

Re: Cleaning up historical portability baggage

2022-07-12 Thread Tom Lane
Andres Freund writes: > Redefining functions, be it by linking in something or by redefining function > names via macros, is a mess. There's places where we then have to undefine > some of these things to be able to include external headers etc. Some > functions are only replaced in backends,

Re: Reducing the chunk header sizes on all memory context types

2022-07-12 Thread Andres Freund
Hi, On 2022-07-12 20:22:57 +0300, Yura Sokolov wrote: > I don't get, why "large chunk" needs additional fields for size and > offset. > Large allocation sizes are certainly rounded to page size. > And allocations which doesn't fit 1GB we could easily round to 1MB. > Then we could simply store

Re: Reducing the chunk header sizes on all memory context types

2022-07-12 Thread Andres Freund
Hi, On 2022-07-12 17:01:18 +1200, David Rowley wrote: > I've taken Andres' patch and made some quite significant changes to > it. In the patch's current state, the sort performance regression in > PG15 vs PG14 is fixed. The generation.c context chunk header has been > reduced to 8 bytes from the

Re: Cleaning up historical portability baggage

2022-07-12 Thread Andres Freund
Hi, On 2022-07-12 08:01:40 -0400, Robert Haas wrote: > On Mon, Jul 11, 2022 at 9:11 PM Thomas Munro wrote: > > Hmm, but that's not what we're doing in general. For example, on > > Windows we're redirecting open() to a replacement function of our own, > > we're not using "pg_open()" in our code.

remove_useless_groupby_columns is too enthusiastic

2022-07-12 Thread Tom Lane
I looked into the complaint at [1] about the planner being much stupider when one side of a JOIN USING is referenced than the other side. It seemed to me that that shouldn't be happening, because the relevant decisions are made on the basis of EquivalenceClasses and both USING columns should be

Re: Reducing the chunk header sizes on all memory context types

2022-07-12 Thread Yura Sokolov
Good day, David. В Вт, 12/07/2022 в 17:01 +1200, David Rowley пишет: > Over on [1], I highlighted that 40af10b57 (Use Generation memory > contexts to store tuples in sorts) could cause some performance > regressions for sorts when the size of the tuple is exactly a power of > 2. The reason for

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-07-12 Thread Andres Freund
Hi, On 2022-07-11 22:22:28 -0400, Melanie Plageman wrote: > Yes, per an off list suggestion by you, I have changed the tests to use a > sum of writes. I've also added a test for IOPATH_LOCAL and fixed some of > the missing calls to count IO Operations for IOPATH_LOCAL and > IOPATH_STRATEGY. > >

Re: making relfilenodes 56 bits

2022-07-12 Thread Andres Freund
Hi, On 2022-07-12 09:51:12 -0400, Robert Haas wrote: > On Mon, Jul 11, 2022 at 7:22 PM Andres Freund wrote: > > I guess I'm not enthused in duplicating the necessary knowledge in evermore > > places. We've forgotten one of the magic incantations in the past, and > > needing > > to find all the

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-07-12 Thread Andres Freund
Hi, On 2022-07-12 12:19:06 -0400, Melanie Plageman wrote: > > > I also realized that I am not differentiating between IOPATH_SHARED and > > > IOPATH_STRATEGY for IOOP_FSYNC. But, given that we don't know what type > > > of buffer we are fsync'ing by the time we call register_dirty_segment(), > >

test_oat_hooks bug (was: Re: pgsql: Add copy/equal support for XID lists)

2022-07-12 Thread Alvaro Herrera
While looking for a place to host a test for XID lists support, I noticed a mistake in test_oat_hooks, fixed as per the attached. -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ >From 7005187f58a2a8b0300eceaa8fae8f825d5525a9 Mon Sep 17 00:00:00 2001 From: Alvaro

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-07-12 Thread Melanie Plageman
Thanks for the review! On Tue, Jul 12, 2022 at 4:06 AM Kyotaro Horiguchi wrote: > At Mon, 11 Jul 2022 22:22:28 -0400, Melanie Plageman < > melanieplage...@gmail.com> wrote in > > Hi, > > > > In the attached patch set, I've added in missing IO operations for > > certain IO Paths as well as

Re: making relfilenodes 56 bits

2022-07-12 Thread Dilip Kumar
On Mon, Jul 11, 2022 at 9:49 PM Robert Haas wrote: > > It also makes me wonder why we're using macros rather than static > inline functions in buf_internals.h. I wonder whether we could do > something like this, for example, and keep InvalidForkNumber as -1: > > static inline ForkNumber >

Re: Introduce "log_connection_stages" setting.

2022-07-12 Thread Euler Taveira
On Tue, Jul 12, 2022, at 10:52 AM, Sergey Dudoladov wrote: > The problem we face is excessive logging of connection information > that clutters the logs and in corner cases with many short-lived > connections leads to disk space exhaustion. You are proposing a fine-grained control over connection

Re: proposal: Allocate work_mem From Pool

2022-07-12 Thread Justin Pryzby
On Tue, Jul 12, 2022 at 03:55:39AM -0700, Joseph D Wagner wrote: > Before I try to answer that, I need to know how the scheduler works. > > Let's say there's a max of 8 worker process, and 12 queries trying to run. > When does query #9 run? After the first of 1-8 completes, simple FIFO? > Or

Re: DropRelFileLocatorBuffers

2022-07-12 Thread Robert Haas
On Tue, Jul 12, 2022 at 12:07 AM Dilip Kumar wrote: > I think the naming used in your patch looks better to me. So +1 for the > change. Committed. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Cleaning up historical portability baggage

2022-07-12 Thread Tom Lane
Robert Haas writes: > On Mon, Jul 11, 2022 at 9:11 PM Thomas Munro wrote: >> Hmm, but that's not what we're doing in general. For example, on >> Windows we're redirecting open() to a replacement function of our own, >> we're not using "pg_open()" in our code. That's not an example based >> on

Introduce "log_connection_stages" setting.

2022-07-12 Thread Sergey Dudoladov
Hello, The problem we face is excessive logging of connection information that clutters the logs and in corner cases with many short-lived connections leads to disk space exhaustion. Current connection log lines share significant parts of the information - host, port, very close timestamps etc.

Re: making relfilenodes 56 bits

2022-07-12 Thread Robert Haas
On Mon, Jul 11, 2022 at 7:22 PM Andres Freund wrote: > I guess I'm not enthused in duplicating the necessary knowledge in evermore > places. We've forgotten one of the magic incantations in the past, and needing > to find all the places that need to be patched is a bit bothersome. > > Perhaps we

Re: Making Vars outer-join aware

2022-07-12 Thread Tom Lane
Richard Guo writes: > Note that the evaluation of expression 'b.j + 1' now occurs below the > outer join. Is this something we need to be concerned about? It seems more formally correct to me, but perhaps somebody would complain about possibly-useless expression evals. We could likely

[PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2022-07-12 Thread Önder Kalacı
Hi hackers, It is often not feasible to use `REPLICA IDENTITY FULL` on the publication, because it leads to full table scan per tuple change on the subscription. This makes `REPLICA IDENTITY FULL` impracticable -- probably other than some small number of use cases. With this patch, I'm

Re: proposal: Allocate work_mem From Pool

2022-07-12 Thread John Naylor
On Tue, Jul 12, 2022 at 5:55 PM Joseph D Wagner wrote: > Before I try to answer that, I need to know how the scheduler works. As I understand the term used, there is no scheduler inside Postgres for user connections -- they're handled by the OS kernel. That's probably why it'd be a difficult

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2022-07-12 Thread Amit Kapila
On Fri, Jul 8, 2022 at 10:26 PM Melih Mutlu wrote: > >> I think that won't work because each time on restart the slot won't be >> fixed. Now, it is possible that we may drop the wrong slot if that >> state of copying rel is SUBREL_STATE_DATASYNC. Also, it is possible >> that while creating a

Re: [PATCH] Compression dictionaries for JSONB

2022-07-12 Thread Aleksander Alekseev
Hi Nikita, > Aleksander, please point me in the right direction if it was mentioned > before, I have a few questions: Thanks for your feedback. These are good questions indeed. > 1) It is not clear for me, how do you see the life cycle of such a > dictionary? If it is meant to keep growing

Re: Cleaning up historical portability baggage

2022-07-12 Thread Robert Haas
On Mon, Jul 11, 2022 at 9:11 PM Thomas Munro wrote: > Hmm, but that's not what we're doing in general. For example, on > Windows we're redirecting open() to a replacement function of our own, > we're not using "pg_open()" in our code. That's not an example based > on AC_REPLACE_FUNCS, but there

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Amit Kapila
On Tue, Jul 12, 2022 at 2:53 PM Masahiko Sawada wrote: > > On Tue, Jul 12, 2022 at 5:58 PM shiy.f...@fujitsu.com > wrote: > > > > > > It happened when executing the following code because it tried to free a > > NULL > > pointer (catchange_xip). > > > > /* be tidy */ > > if

RE: proposal: Allocate work_mem From Pool

2022-07-12 Thread Joseph D Wagner
>> I think it would be better if work_mem was allocated from a pool >> of memory > I think this has been proposed before, and the issue/objection > with this idea is probably that query plans will be inconsistent, > and end up being sub-optimal. > work_mem is considered at planning time, but I

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2022-07-12 Thread Alexander Korotkov
Hi Aleksander! Thank you for your efforts reviewing this patch. On Thu, Jul 7, 2022 at 12:43 PM Aleksander Alekseev wrote: > > I'm going to need more time to meditate on the proposed changes and to > > figure out the performance impact. > > OK, turned out this patch is slightly more

Re: [PATCH] Compression dictionaries for JSONB

2022-07-12 Thread Nikita Malakhov
Hi hackers! Aleksander, please point me in the right direction if it was mentioned before, I have a few questions: 1) It is not clear for me, how do you see the life cycle of such a dictionary? If it is meant to keep growing without cleaning up/rebuilding it could affect performance in an

Re: CREATE TABLE ( .. STORAGE ..)

2022-07-12 Thread Aleksander Alekseev
Hi Peter, > The "safety check: do not allow toasted storage modes unless column > datatype is TOAST-aware" could be moved into GetAttributeStorage(), so > it doesn't have to be repeated. (Note that GetAttributeCompression() > does similar checking.) Good point. Fixed. > ATExecSetStorage()

Re: Cleaning up historical portability baggage

2022-07-12 Thread Peter Eisentraut
On 12.07.22 03:10, Thomas Munro wrote: AFAIK we generally only use pg_whatever() when there's a good reason, such as an incompatibility, a complication or a different abstraction that you want to highlight to a reader. The reason here was temporary: we couldn't implement standard pread/pwrite

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 5:58 PM shiy.f...@fujitsu.com wrote: > > On Tue, Jul 12, 2022 8:49 AM Masahiko Sawada wrote: > > > > I've attached an updated patch. > > > > Hi, > > I met a segmentation fault in test_decoding test after applying the patch for > master > branch. Attach the backtrace.

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread shiy.f...@fujitsu.com
On Tue, Jul 12, 2022 8:49 AM Masahiko Sawada wrote: > > I've attached an updated patch. > Hi, I met a segmentation fault in test_decoding test after applying the patch for master branch. Attach the backtrace. It happened when executing the following code because it tried to free a NULL

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Amit Kapila
On Tue, Jul 12, 2022 at 1:13 PM Masahiko Sawada wrote: > > On Tue, Jul 12, 2022 at 3:25 PM Amit Kapila wrote: > > > > On Tue, Jul 12, 2022 at 11:38 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Jul 12, 2022 at 10:28 AM Masahiko Sawada > > > wrote: > > > > > > > > > > > > I'm doing

Re: Weird behaviour with binary copy, arrays and column count

2022-07-12 Thread James Vanns
That's a good tip! Can't believe I hadn't even thought of that! :/ Cheers Jim On Mon, 11 Jul 2022 at 21:59, Greg Stark wrote: > > On Fri, 8 Jul 2022 at 13:09, James Vanns wrote: > > > > It does seem to smell of an alignment, padding, buffer overrun, parsing > > kind of error. > > It does

Re: replacing role-level NOINHERIT with a grant-level option

2022-07-12 Thread tushar
On 7/11/22 11:01 PM, Robert Haas wrote: Oops. Here is a rebased version of v3 which aims to fix this bug. Thanks, Issue seems to be fixed with this patch. -- regards,tushar EnterpriseDB https://www.enterprisedb.com/ The Enterprise PostgreSQL Company

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-07-12 Thread Kyotaro Horiguchi
At Mon, 11 Jul 2022 22:22:28 -0400, Melanie Plageman wrote in > Hi, > > In the attached patch set, I've added in missing IO operations for > certain IO Paths as well as enumerating in the commit message which IO > Paths and IO Operations are not currently counted and or not possible. > >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 3:25 PM Amit Kapila wrote: > > On Tue, Jul 12, 2022 at 11:38 AM Masahiko Sawada > wrote: > > > > On Tue, Jul 12, 2022 at 10:28 AM Masahiko Sawada > > wrote: > > > > > > > > > I'm doing benchmark tests and will share the results. > > > > > > > I've done benchmark tests

Re: Support TRUNCATE triggers on foreign tables

2022-07-12 Thread Yugo NAGATA
On Tue, 12 Jul 2022 09:24:20 +0900 Fujii Masao wrote: > > > On 2022/07/08 17:13, Ian Lawrence Barwick wrote: > >> If we want to add such prevention, we will need similar checks for > >> INSERT/DELETE/UPDATE not only TRUNCATE. However, I think such fix is > >> independent > >> from this and it

Re: Making Vars outer-join aware

2022-07-12 Thread Richard Guo
On Mon, Jul 11, 2022 at 3:38 AM Tom Lane wrote: > Here's v2 of this patch series. It's functionally identical to v1, > but I've rebased it over the recent auto-node-support-generation > changes, and also extracted a few separable bits in hopes of making > the main planner patch smaller. (It's

Re: [PATCH] Optimize json_lex_string by batching character copying

2022-07-12 Thread John Naylor
On Mon, Jul 11, 2022 at 11:07 PM Andres Freund wrote: > I wonder if we can add a somewhat more general function for scanning until > some characters are found using SIMD? There's plenty other places that could > be useful. In simple cases, we could possibly abstract the entire loop. With this

Re: First draft of the PG 15 release notes

2022-07-12 Thread Noah Misch
On Mon, Jul 11, 2022 at 12:39:57PM -0400, Bruce Momjian wrote: > I had trouble reading the sentences in the order you used so I > restructured it: > > The new default is one of the secure schema usage patterns thatlinkend="ddl-schemas-patterns"/> has recommended since the security >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Amit Kapila
On Tue, Jul 12, 2022 at 11:38 AM Masahiko Sawada wrote: > > On Tue, Jul 12, 2022 at 10:28 AM Masahiko Sawada > wrote: > > > > > > I'm doing benchmark tests and will share the results. > > > > I've done benchmark tests to measure the overhead introduced by doing > bsearch() every time when

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 10:28 AM Masahiko Sawada wrote: > > On Tue, Jul 12, 2022 at 9:48 AM Masahiko Sawada wrote: > > > > On Fri, Jul 8, 2022 at 8:20 PM Masahiko Sawada > > wrote: > > > > > > On Fri, Jul 8, 2022 at 5:59 PM Amit Kapila > > > wrote: > > > > > > > > On Fri, Jul 8, 2022 at