Re: Minimal logical decoding on standbys

2023-03-02 Thread Jeff Davis
On Thu, 2023-03-02 at 11:45 -0800, Jeff Davis wrote: > In this case it looks easier to add the right API than to be sure > about > whether it's needed or not. I attached a sketch of one approach. I'm not very confident that it's the right API or even that it works as I intended it, but if others l

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread Önder Kalacı
Hi Peter, all > > == > Commit Message > > 1. > There is no smart mechanism to pick the index. Instead, we choose > the first index that fulfils the requirements mentioned above. > > ~ > > 1a. > I think this paragraph should immediately follow the earlier one > ("With this patch...") which talk

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread Önder Kalacı
Hi Hou zj, all > > 1. > + usableIndexContext = AllocSetContextCreate(CurrentMemoryContext, > + > "usableIndexContext", > + > ALLOCSET_DEFAULT_SIZES); > + oldctx = MemoryContextSwitchTo(usableIndexContext); > + > + /* Get index list of the local re

Re: meson: Optionally disable installation of test modules

2023-03-02 Thread Peter Eisentraut
On 02.03.23 08:09, Nazir Bilal Yavuz wrote: On Wed, 1 Mar 2023 at 22:21, Peter Eisentraut wrote: Looks good to me. I did a small pass over it to adjust some namings. For example, I renamed test_install_files to test_install_data, so it's consistent with the overall meson naming: -install_dat

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread vignesh C
On Thu, 2 Mar 2023 at 20:53, Önder Kalacı wrote: > > Hi, > >> >> Is it just a matter of testing other >> index types > > > Yes, there are more to it. build_replindex_scan_key() > only works for btree indexes, as it does BTEqualStrategyNumber. > > I might expect a few more limitations like that. I

Re: Force testing of query jumbling code in TAP tests

2023-03-02 Thread Michael Paquier
On Tue, Feb 28, 2023 at 02:06:05PM +0900, Michael Paquier wrote: > Are there any objections to do what's proposed in the patch and > improve the testing coverage of query jumbling by default? Well, done this one as of d28a449. More validation tests could always be added later if there are better

Re: Add SHELL_EXIT_CODE to psql

2023-03-02 Thread Corey Huinker
On Thu, Mar 2, 2023 at 1:35 PM Tom Lane wrote: > Corey Huinker writes: > > [ v9-0003-Add-psql-variables-SHELL_ERROR-and-SHELL_EXIT_COD.patch ] > > I took a brief look through this. I'm on board with the general > concept, but I think you've spent too much time on an ultimately > futile attempt

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread Amit Kapila
On Thu, Mar 2, 2023 at 8:52 PM Önder Kalacı wrote: > > Alright, attached v27_0001_use_index_on_subs_when_pub_rep_ident_full.patch and > v27_0002_use_index_on_subs_when_pub_rep_ident_full.patch. > Few comments on 0001 1. + such suitable indexes, the search on the subscriber

Re: Real config values for bytes needs quotes?

2023-03-02 Thread Kyotaro Horiguchi
At Thu, 2 Mar 2023 11:54:10 +0100, Peter Eisentraut wrote in > On 27.02.23 09:32, Kyotaro Horiguchi wrote: > > I found it frustrating that the line "shared_buffers = 0.1GB" in > > postgresql.conf postgresql.conf was causing an error and that the > > value required (additional) surrounding single

Re: Add pg_walinspect function with block info columns

2023-03-02 Thread Michael Paquier
On Thu, Mar 02, 2023 at 11:17:05AM -0500, Melanie Plageman wrote: > Thinking about this more, it could make sense to have a function which > gives you this extended block information and has a parameter like > with_fpi which would include the information returned by > pg_get_wal_fpi_info(). It migh

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread Peter Smith
Here are some review comments for v27-0001 (not the tests) == Commit Message 1. There is no smart mechanism to pick the index. Instead, we choose the first index that fulfils the requirements mentioned above. ~ 1a. I think this paragraph should immediately follow the earlier one ("With this

Re: [EXTERNAL] Re: Support load balancing in libpq

2023-03-02 Thread Andrey Borodin
On Wed, Mar 1, 2023 at 12:03 PM Jelte Fennema wrote: > > done and updated cf entry > Hi Jelte! I've looked into the patch. Although so many improvements can be suggested, It definitely makes sense as-is too. These improvements might be, for example, sorting hosts according to ping latency or som

Re: Remove source code display from \df+?

2023-03-02 Thread Isaac Morland
On Thu, 2 Mar 2023 at 17:20, Tom Lane wrote: > Isaac Morland writes: > > [ 0001-Remove-source-code-display-from-df-v6.patch ] > > Pushed after some editorialization on the test case. > Thanks! One thing I noticed while testing is that if you apply \df+ to an > aggregate function, it will show

Re: Date-time extraneous fields with reserved keywords

2023-03-02 Thread Keisuke Kuroda
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed Hi Joseph, Good catch. Of the reserved words that are specia

RE: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread houzj.f...@fujitsu.com
On Thursday, March 2, 2023 11:23 PM Önder Kalacı wrote: > Both the patches are numbered 0001. It would be better to number them > as 0001 and 0002. > > Alright, attached v27_0001_use_index_on_subs_when_pub_rep_ident_full.patch > and > v27_0002_use_index_on_subs_when_pub_rep_ident_full.patch.

Re: Making empty Bitmapsets always be NULL

2023-03-02 Thread David Rowley
On Fri, 3 Mar 2023 at 15:17, Tom Lane wrote: > (Is it worth carrying both "allocated words" and "nonzero words" > fields to avoid useless memory-management effort? Dunno.) It would have been a more appealing thing to do before Bitmapset became a node type as we'd have had empty space in the stru

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread Amit Kapila
On Thu, Mar 2, 2023 at 6:50 PM Önder Kalacı wrote: > >> >> >> In the above profile number of calls to index_fetch_heap(), >> heapam_index_fetch_tuple() explains the reason for the regression you >> are seeing with the index scan. Because the update will generate dead >> tuples in the same transact

Re: Rework LogicalOutputPluginWriterUpdateProgress

2023-03-02 Thread Peter Smith
On Fri, Mar 3, 2023 at 1:27 PM houzj.f...@fujitsu.com wrote: > > On Friday, March 3, 2023 8:18 AM Peter Smith wrote: ... > > Anyway, I think this exposes another problem. If you still want the patch > > to pass > > the 'finshed_xact' parameter separately then AFAICT the first parameter > > (ctx

RE: Rework LogicalOutputPluginWriterUpdateProgress

2023-03-02 Thread houzj.f...@fujitsu.com
On Friday, March 3, 2023 8:18 AM Peter Smith wrote: > On Wed, Mar 1, 2023 at 9:16 PM wangw.f...@fujitsu.com > wrote: > > > > On Tues, Feb 28, 2023 at 9:12 AM Peter Smith > wrote: > > > Here are some comments for the v2-0001 patch. > > > > > > (I haven't looked at the v3 that was posted overnight

Re: Making empty Bitmapsets always be NULL

2023-03-02 Thread Tom Lane
David Rowley writes: > I suggest tightening the rule even further so instead of just empty > sets having to be represented as NULL, the rule should be that sets > should never contain any trailing zero words, which is effectively a > superset of the "empty is NULL" rule that you've just changed.

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread Peter Smith
FYI, After applying only the 0001 patch I received a TAP test error. t/032_subscribe_use_index.pl ... 1/? # Tests were run but no plan was declared and done_testing() was not seen. t/032_subscribe_use_index.pl ... Dubious, test returned 29 (wstat 7424, 0x1d00) All 1 subtests passed t/100_

Re: Understanding, testing and improving our Windows filesystem code

2023-03-02 Thread Thomas Munro
On Thu, Jan 5, 2023 at 1:06 AM vignesh C wrote: > On Tue, 25 Oct 2022 at 09:42, Thomas Munro wrote: > > I pushed the bug fixes from this series, without their accompanying > > tests. Here's a rebase of the test suite, with all those tests now > > squashed into the main test patch, and also the >

Re: Making empty Bitmapsets always be NULL

2023-03-02 Thread David Rowley
On Wed, 1 Mar 2023 at 10:59, Tom Lane wrote: > When I designed the Bitmapset module, I set things up so that an empty > Bitmapset could be represented either by a NULL pointer, or by an > allocated object all of whose bits are zero. I've recently come to > the conclusion that that was a bad idea

Re: Normalization of utility queries in pg_stat_statements

2023-03-02 Thread Michael Paquier
On Thu, Mar 02, 2023 at 08:12:24AM +0100, Drouvot, Bertrand wrote: > Applying 0001 produces: > > Applying: Split more regression tests of pg_stat_statements > .git/rebase-apply/patch:1735: new blank line at EOF. > + > .git/rebase-apply/patch:2264: new blank line at EOF. > + > warning: 2 lines add

Re: Rework LogicalOutputPluginWriterUpdateProgress

2023-03-02 Thread Andres Freund
Hi, On 2023-03-03 11:18:04 +1100, Peter Smith wrote: > - Why is reducing members of LogicalDecodingContext even a goal? I > thought the LogicalDecodingContext is intended to be the one-stop > place to hold *all* things related to the "Context" (including that > member that was deleted). There's n

Re: Allow logical replication to copy tables in binary format

2023-03-02 Thread Peter Smith
On Thu, Mar 2, 2023 at 4:00 PM Amit Kapila wrote: > > On Thu, Mar 2, 2023 at 7:27 AM Peter Smith wrote: > > ... > > IIUC most people seem to be coming down in favour of there being a > > single unified option (the existing 'binary==true/false) which would > > apply to both the COPY and the data r

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-03-02 Thread Andres Freund
On 2023-03-02 14:41:26 -0500, reid.thomp...@crunchydata.com wrote: > Patch has been rebased to master. Quite a few prior review comments seem to not have been addressed. There's not much point in posting new versions without that. I think there's zero chance 0002 can make it into 16. If 0001 is c

Re: Evaluate arguments of correlated SubPlans in the referencing ExprState

2023-03-02 Thread Andres Freund
Hi, On 2023-03-02 13:00:31 -0800, Andres Freund wrote: > I'm not opposed to EXPR_PARAM_SET, to be clear. I'll send an updated > version later. I was just thinking about the correctness in the current > world. Attached. I named the set EEOP_PARAM_SET EEOP_PARAM_EXEC_SET or such, because I was won

Re: Rework LogicalOutputPluginWriterUpdateProgress

2023-03-02 Thread Peter Smith
On Wed, Mar 1, 2023 at 9:16 PM wangw.f...@fujitsu.com wrote: > > On Tues, Feb 28, 2023 at 9:12 AM Peter Smith wrote: > > Here are some comments for the v2-0001 patch. > > > > (I haven't looked at the v3 that was posted overnight; maybe some of > > my comments have already been addressed.) > > Tha

Re: Should vacuum process config file reload more often

2023-03-02 Thread Melanie Plageman
On Thu, Mar 2, 2023 at 2:36 AM Masahiko Sawada wrote: > > On Thu, Mar 2, 2023 at 10:41 AM Melanie Plageman > wrote: > > On another topic, I've just realized that when autovacuuming we only > > update tab->at_vacuum_cost_delay/limit from > > autovacuum_vacuum_cost_delay/limit for each table (in >

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Tomas Vondra
On 3/2/23 23:57, Jehan-Guillaume de Rorthais wrote: > On Thu, 2 Mar 2023 19:53:14 +0100 > Tomas Vondra wrote: >> On 3/2/23 19:15, Jehan-Guillaume de Rorthais wrote: > ... > >>> There was some thoughts about how to make a better usage of the memory. As >>> memory is exploding way beyond work_me

Re: Introduce "log_connection_stages" setting.

2023-03-02 Thread Jacob Champion
On 3/2/23 14:56, Tom Lane wrote: > Jacob Champion writes: >> If I've understood Tom correctly in [1], both of these guc_mallocs >> should be using a loglevel less than ERROR, to avoid forcing a >> postmaster exit when out of memory. (I used WARNING in that thread >> instead, which seemed to be acc

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Jehan-Guillaume de Rorthais
On Thu, 2 Mar 2023 19:53:14 +0100 Tomas Vondra wrote: > On 3/2/23 19:15, Jehan-Guillaume de Rorthais wrote: ... > > There was some thoughts about how to make a better usage of the memory. As > > memory is exploding way beyond work_mem, at least, avoid to waste it with > > too many buffers of BufF

Re: Introduce "log_connection_stages" setting.

2023-03-02 Thread Tom Lane
Jacob Champion writes: > This is looking very good. One bigger comment: >> +myextra = (int *) guc_malloc(ERROR, sizeof(int)); >> +*myextra = newlogconnect; > If I've understood Tom correctly in [1], both of these guc_mallocs > should be using a loglevel less than ERROR, to avoid forcing

Re: buildfarm + meson

2023-03-02 Thread Andrew Dunstan
On 2023-03-02 Th 17:06, Andres Freund wrote: Hi On 2023-03-02 17:00:47 -0500, Andrew Dunstan wrote: On 2023-03-01 We 16:32, Andres Freund wrote: This is now working on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. I do get a whole lot of annoying messages like this:

Re: Introduce "log_connection_stages" setting.

2023-03-02 Thread Jacob Champion
On 2/1/23 11:59, Sergey Dudoladov wrote: > Justin, thank you for the fast review. The new version is attached. This is looking very good. One bigger comment: > + myextra = (int *) guc_malloc(ERROR, sizeof(int)); > + *myextra = newlogconnect; If I've understood Tom correctly in [1], both

Documentation of psql's \df no longer matches reality

2023-03-02 Thread Tom Lane
While preparing 3dfae91f7 I couldn't help noticing that what psql-ref.sgml has to say about \df's "function type" column: ... and function types, which are classified as agg (aggregate), normal, procedure, trigger, or window. no longer corresponds very well to what the code actua

Re: Remove source code display from \df+?

2023-03-02 Thread Tom Lane
Isaac Morland writes: > [ 0001-Remove-source-code-display-from-df-v6.patch ] Pushed after some editorialization on the test case. One thing I noticed while testing is that if you apply \df+ to an aggregate function, it will show "Internal name" of "aggregate_dummy". While that's an accurate desc

Re: buildfarm + meson

2023-03-02 Thread Andres Freund
Hi On 2023-03-02 17:00:47 -0500, Andrew Dunstan wrote: > > On 2023-03-01 We 16:32, Andres Freund wrote: > > > This is now working > > > on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP > > > tests. > > > I do get a whole lot of annoying messages like this: > > > > > > Unknow

Re: File descriptors in exec'd subprocesses

2023-03-02 Thread Thomas Munro
On Thu, Mar 2, 2023 at 9:57 AM Thomas Munro wrote: > On Thu, Mar 2, 2023 at 9:49 AM Gregory Stark (as CFM) > wrote: > > On Mon, 20 Feb 2023 at 23:04, Thomas Munro wrote: > > > Done like that in this version. This is the version I'm thinking of > > > committing, unless someone wants to argue for

Re: buildfarm + meson

2023-03-02 Thread Andrew Dunstan
On 2023-03-01 We 16:32, Andres Freund wrote: This is now working on my MSVC test rig (WS2019, VS2019, Strawberry Perl), including TAP tests. I do get a whole lot of annoying messages like this: Unknown TAP version. The first line MUST be `TAP version `. Assuming version 12. The newest minor ve

Re: Operation log for major operations

2023-03-02 Thread Kirk Wolak
On Thu, Mar 2, 2023 at 4:09 PM Dmitry Koval wrote: > I'll try to expand my explanation. > I fully understand and accept the arguments about "limited sense to go > into the control file" and "about recording *anything* in the control > file". This is totally correct for vanilla. > But vendors have

Re: Operation log for major operations

2023-03-02 Thread Dmitry Koval
I'll try to expand my explanation. I fully understand and accept the arguments about "limited sense to go into the control file" and "about recording *anything* in the control file". This is totally correct for vanilla. But vendors have forks of PostgreSQL with custom features and extensions. S

Re: Evaluate arguments of correlated SubPlans in the referencing ExprState

2023-03-02 Thread Andres Freund
Hi, On 2023-03-02 15:10:31 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-03-02 14:33:35 -0500, Tom Lane wrote: > >> I looked through this, and there is one point that is making me really > >> uncomfortable. This bit is assuming that we can bind the address of > >> the es_param_exec_

Re: proposal: psql: show current user in prompt

2023-03-02 Thread Kirk Wolak
On Sat, Feb 4, 2023 at 3:33 PM Pavel Stehule wrote: > Hi > > pá 3. 2. 2023 v 21:43 odesílatel Pavel Stehule > napsal: > >> >> >> pá 3. 2. 2023 v 21:21 odesílatel Tom Lane napsal: >> >>> Pavel Stehule writes: >>> > Both patches are very simple - and they use almost already prepared >>> > infras

Re: Add support for unit "B" to pg_size_pretty()

2023-03-02 Thread Dean Rasheed
On Thu, 2 Mar 2023 at 19:58, David Rowley wrote: > > I think I'd prefer to see the size_bytes_unit_alias struct have an > index into size_pretty_units[] array. i.e: > > struct size_bytes_unit_alias > { > const char *alias; /* aliased unit name */ > const int unit_index; /* corresponding si

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-03-02 Thread Alexander Korotkov
On Thu, Mar 2, 2023 at 9:17 PM Pavel Borisov wrote: > On Thu, 2 Mar 2023 at 18:53, Alexander Korotkov wrote: > > On Thu, Mar 2, 2023 at 1:29 PM Pavel Borisov wrote: > > > > Let's see the performance results for the patchset. I'll properly > > > > revise the comments if results will be good. > >

Re: Evaluate arguments of correlated SubPlans in the referencing ExprState

2023-03-02 Thread Tom Lane
Andres Freund writes: > On 2023-03-02 14:33:35 -0500, Tom Lane wrote: >> I looked through this, and there is one point that is making me really >> uncomfortable. This bit is assuming that we can bind the address of >> the es_param_exec_vals array right into the compiled expression: > Yea, I wasn

Re: Evaluate arguments of correlated SubPlans in the referencing ExprState

2023-03-02 Thread Andres Freund
Hi, On 2023-03-02 14:33:35 -0500, Tom Lane wrote: > Andres Freund writes: > > Around > > https://www.postgresql.org/message-id/20230224015417.75yimxbksejpffh3%40awork3.anarazel.de > > I suggested that we should evaluate the arguments of correlated SubPlans as > > part of the expression referencin

Re: Auth extensions, with an LDAP/SCRAM example [was: Proposal: Support custom authentication methods using hooks]

2023-03-02 Thread Jacob Champion
On Tue, Feb 28, 2023 at 6:53 PM Stephen Frost wrote: > * Jacob Champion (jchamp...@timescale.com) wrote: > > Yes. Interoperable authentication is going to have to solve those > > sorts of problems somehow, though. And there are a bunch of levers to > > pull: clients aren't required to run their se

Re: Add support for unit "B" to pg_size_pretty()

2023-03-02 Thread David Rowley
On Mon, 27 Feb 2023 at 21:34, Peter Eisentraut wrote: > > On 22.02.23 03:39, David Rowley wrote: > > I think you'll need to find another way to make the aliases work. > > Maybe another array with the name and an int to reference the > > corresponding index in size_pretty_units. > > Ok, here is a n

Re: Minimal logical decoding on standbys

2023-03-02 Thread Jeff Davis
On Thu, 2023-03-02 at 10:20 +0100, Drouvot, Bertrand wrote: > Right, but in our case, right after the wakeup (the one due to the CV > broadcast, > aka the one that will remove it from the wait queue) we'll exit the > loop due to: > > " > /* check whether we're done */ > if (loc <

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-03-02 Thread reid . thompson
On Mon, 2023-02-13 at 16:26 -0800, Andres Freund wrote: > Hi, > > The tests recently started to fail: > > https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3867 > > I marked this as waiting on author. > > Greetings, > > Andres Freund Patch has been rebased to master.

Re: libpq: PQgetCopyData() and allocation overhead

2023-03-02 Thread Jeroen Vermeulen
My apologies. The wiki said to discuss early, even before writing the code if possible, but I added a link to the PR for those who really wanted to see the details. I'm attaching a diff now. This is not a patch, it's just a discussion piece. The problem was that PQgetCopyData loops use a lot of

Re: Operation log for major operations

2023-03-02 Thread Tom Lane
Justin Pryzby writes: > On Thu, Mar 02, 2023 at 08:57:43PM +0300, Dmitry Koval wrote: >> These changes did not interest the community. It was expected (topic is very >> specifiс: vendor's technical support). So no sense to distract developers > Actually, I think there is interest, but it has to b

Re: Evaluate arguments of correlated SubPlans in the referencing ExprState

2023-03-02 Thread Tom Lane
Andres Freund writes: > Around > https://www.postgresql.org/message-id/20230224015417.75yimxbksejpffh3%40awork3.anarazel.de > I suggested that we should evaluate the arguments of correlated SubPlans as > part of the expression referencing the subplan. > Here's a patch for that. I looked through

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Tomas Vondra
On 3/2/23 19:15, Jehan-Guillaume de Rorthais wrote: > Hi! > > On Thu, 2 Mar 2023 13:44:52 +0100 > Tomas Vondra wrote: >> Well, yeah and no. >> >> In principle we could/should have allocated the BufFiles in a different >> context (possibly hashCxt). But in practice it probably won't make any >>

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Jehan-Guillaume de Rorthais
On Thu, 2 Mar 2023 19:15:30 +0100 Jehan-Guillaume de Rorthais wrote: [...] > For what it worth, these two patches seems really interesting to me. Do you > need any help to revive it? To avoid confusion, the two patches I meant were: * 0001-move-BufFile-stuff-into-separate-context.patch * v4-p

Re: Operation log for major operations

2023-03-02 Thread Justin Pryzby
On Thu, Mar 02, 2023 at 08:57:43PM +0300, Dmitry Koval wrote: > These changes did not interest the community. It was expected (topic is very > specifiс: vendor's technical support). So no sense to distract developers Actually, I think there is interest, but it has to be phrased in a limited sense

Re: Add SHELL_EXIT_CODE to psql

2023-03-02 Thread Tom Lane
Corey Huinker writes: > [ v9-0003-Add-psql-variables-SHELL_ERROR-and-SHELL_EXIT_COD.patch ] I took a brief look through this. I'm on board with the general concept, but I think you've spent too much time on an ultimately futile attempt to get a committable test case, and not enough time on makin

Re: Sort optimizations: Making in-memory sort cache-aware

2023-03-02 Thread Ankit Kumar Pandey
Hi Andres, I took a stab at naive version of this but been stuck for sometime now. I have added logic to sort on first column at first pass, realloc all tuples and do full sort at second pass, but I am not seeing any benefit (it is actually regressing) at all. Tried doing above both at bulk

Re: Add LZ4 compression in pg_dump

2023-03-02 Thread Tomas Vondra
On 3/2/23 18:18, Justin Pryzby wrote: > On Wed, Mar 01, 2023 at 05:20:05PM +0100, Tomas Vondra wrote: >> On 2/25/23 15:05, Justin Pryzby wrote: >>> On Fri, Feb 24, 2023 at 11:02:14PM -0600, Justin Pryzby wrote: I have some fixes (attached) and questions while polishing the patch for zs

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Jehan-Guillaume de Rorthais
Hi! On Thu, 2 Mar 2023 13:44:52 +0100 Tomas Vondra wrote: > Well, yeah and no. > > In principle we could/should have allocated the BufFiles in a different > context (possibly hashCxt). But in practice it probably won't make any > difference, because the query will probably run all the hashjoins

Re: Operation log for major operations

2023-03-02 Thread Dmitry Koval
Hi! These changes did not interest the community. It was expected (topic is very specifiс: vendor's technical support). So no sense to distract developers ... I'll move patch to Withdrawn. -- With best regards, Dmitry Koval Postgres Professional: http://postgrespro.com

Re: Add LZ4 compression in pg_dump

2023-03-02 Thread Justin Pryzby
On Wed, Mar 01, 2023 at 05:20:05PM +0100, Tomas Vondra wrote: > On 2/25/23 15:05, Justin Pryzby wrote: > > On Fri, Feb 24, 2023 at 11:02:14PM -0600, Justin Pryzby wrote: > >> I have some fixes (attached) and questions while polishing the patch for > >> zstd compression. The fixes are small and cou

Re: Making empty Bitmapsets always be NULL

2023-03-02 Thread Tom Lane
Richard Guo writes: > It seems that the Bitmapset checked by bms_is_empty_internal cannot be > NULL from how it is computed by a function. So I wonder if we can > remove the check of 'a' being NULL in that function, or reduce it to an > Assert. Yeah, I think just removing it is sufficient. The

Re: is_superuser is not documented

2023-03-02 Thread Joseph Koshakow
On Thu, Mar 2, 2023 at 11:53 AM Fujii Masao wrote: > >On 2022/09/14 14:27, bt22kawamotok wrote: >> I update patch to reflect master update. > >Thanks for updating the patch! > >+ >+Shows whether the current user is a superuser or not. >+ > >How abo

Re: Add LZ4 compression in pg_dump

2023-03-02 Thread gkokolatos
--- Original Message --- On Wednesday, March 1st, 2023 at 5:20 PM, Tomas Vondra wrote: > > > > > On 2/25/23 15:05, Justin Pryzby wrote: > > > On Fri, Feb 24, 2023 at 11:02:14PM -0600, Justin Pryzby wrote: > > > > > I have some fixes (attached) and questions while polishing th

Re: Add pg_walinspect function with block info columns

2023-03-02 Thread Melanie Plageman
On Wed, Mar 1, 2023 at 12:51 PM Melanie Plageman wrote: > When using pg_walinspect, and calling functions like > pg_get_wal_records_info(), I often wish that the various information in > the block_ref column was separated out into columns so that I could > easily access them and pass them to vario

Re: Proposal: %T Prompt parameter for psql for current time (like Oracle has)

2023-03-02 Thread Kirk Wolak
On Thu, Mar 2, 2023 at 9:56 AM Laurenz Albe wrote: > On Wed, 2023-03-01 at 11:13 -0500, Kirk Wolak wrote: > > Thanks, corrected, and confirmed Unix line endings. > > The patch builds fine and works as intended. > > I leave it to the committers to decide whether the patch is worth the > effort or

Re: Allow tailoring of ICU locales with custom rules

2023-03-02 Thread Laurenz Albe
On Wed, 2023-02-22 at 18:35 +0100, Peter Eisentraut wrote: > > - there doesn't seem to be a way to add rules to template1. > > If someone wants to have icu rules and initial contents to their new > > databases, I think they need to create a custom template database > > (from template0) for that pur

Re: Allow logical replication to copy tables in binary format

2023-03-02 Thread Kuntal Ghosh
On Thu, Mar 2, 2023 at 10:30 AM Amit Kapila wrote: > > > TBH I am not sure anymore if the complications justify the patch. > > > > It seems we have to choose from 2 bad choices: > > - separate options = this works but would be more confusing for the user > > - unified option = this would be simple

Re: Proposal: %T Prompt parameter for psql for current time (like Oracle has)

2023-03-02 Thread Laurenz Albe
On Wed, 2023-03-01 at 11:13 -0500, Kirk Wolak wrote: > Thanks, corrected, and confirmed Unix line endings. The patch builds fine and works as intended. I leave it to the committers to decide whether the patch is worth the effort or not, given that you can get a similar effect with %`date`. It add

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-03-02 Thread Alexander Korotkov
Hi, Pavel! On Thu, Mar 2, 2023 at 1:29 PM Pavel Borisov wrote: > > Let's see the performance results for the patchset. I'll properly > > revise the comments if results will be good. > > > > Pavel, could you please re-run your tests over revised patchset? > > Since last time I've improved the tes

Re: Refactoring SysCacheGetAttr to know when attr cannot be NULL

2023-03-02 Thread Tom Lane
Peter Eisentraut writes: > I think an error message like > "unexpected null value in system cache %d column %d" > is sufficient. Since these are "can't happen" errors, we don't need to > spend too much extra effort to make it prettier. I'd at least like to see it give the catalog's OID. T

Re: Refactoring SysCacheGetAttr to know when attr cannot be NULL

2023-03-02 Thread Tom Lane
Daniel Gustafsson writes: >> On 1 Mar 2023, at 00:20, Tom Lane wrote: >> Also ... at least in assert-enabled builds, maybe we could check that >> the column being fetched this way is actually marked attnotnull? > We could, but that would limit the API to attnotnull columns, rather than when > th

Request for comment on setting binary format output per session

2023-03-02 Thread Dave Cramer
Greetings, In [1] I proposed a patch that used a GUC to request a list of OID's to be returned in binary format. In [2] Peter Eisentraut proposed a very similar solution to the problem. In [2] there was some discussion regarding whether this should be set via GUC or a new protocol message. I'd l

Re: Improving inferred query column names

2023-03-02 Thread Peter Eisentraut
On 22.02.23 21:38, Andres Freund wrote: On 2023-02-20 16:08:00 +0100, Peter Eisentraut wrote: On 11.02.23 20:24, Andres Freund wrote: I think we should just do it and not care about what breaks. There has never been any guarantee about these. FWIW, "most" other SQL implementations appear to ge

Re: generic plans and "initial" pruning

2023-03-02 Thread Amit Langote
On Wed, Feb 8, 2023 at 7:31 PM Amit Langote wrote: > On Tue, Feb 7, 2023 at 23:38 Andres Freund wrote: >> The tests seem to frequently hang on freebsd: >> https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3478 > > Thanks for the heads up. I’ve noticed this one too, thoug

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
> Yeah, min_send_delay and max_slots_wal_keep_size should be easily tunable > because > the appropriate value depends on the enviroment and workload. > However, pg_replication_slots.pg_replication_slots cannot show the exact amout > of WALs, > so it may not suitable for tuning. I think user can com

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
Dear Amit, > Fair point but I think the current comment should explain why we are > doing something different here. How about extending the existing > comments to something like: "If we've requested to shut down, exit the > process. This is unlike handling at other places where we allow > complete

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
Dear Vignesh, Thank you for reviewing! New version can be available at [1]. > 1) Currently we have added the delay during the decode of commit, > while decoding the commit walsender process will stop decoding any > further transaction until delay is completed. There might be a > possibility that

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-03-02 Thread Hayato Kuroda (Fujitsu)
Dear Peter, Thank you for reviewing! PSA new version. > 1. > Nitpick. The new text is jagged-looking. It should wrap at ~80 chars. Addressed. > > 2. > 2. Another reason is for that parallel streaming, the transaction will be > opened > immediately by the parallel apply worker. Therefore, if t

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread Önder Kalacı
Hi Amit, Shi Yu > > > > b. Executed SQL. > > I executed TRUNCATE and INSERT before each UPDATE. I am not sure if you > did the > > same, or just executed 50 consecutive UPDATEs. If the latter one, there > would be > > lots of old tuples and this might have a bigger impact on sequential > scan. I

Re: Fix comments in gistxlogDelete, xl_heap_freeze_page and xl_btree_delete

2023-03-02 Thread Drouvot, Bertrand
Hi, On 1/6/23 11:05 AM, Drouvot, Bertrand wrote: Hi hackers, Please find attached a patch to $SUBJECT. The wrong comments have been discovered by Robert in [1]. Submitting this here as a separate thread so it does not get lost in the logical decoding on standby thread. [1]: https://www.pos

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Tomas Vondra
On 3/2/23 13:08, Jehan-Guillaume de Rorthais wrote: > ... > [...] >> But I have another idea - put a breakpoint on makeBufFile() which is the >> bit that allocates the temp files including the 8kB buffer, and print in >> what context we allocate that. I have a hunch we may be allocating it in >>

Re: libpq: PQgetCopyData() and allocation overhead

2023-03-02 Thread Daniel Gustafsson
> On 1 Mar 2023, at 15:23, Jeroen Vermeulen wrote: > PR for easy discussion: https://github.com/jtv/postgres/pull/1 The process for discussing work on pgsql-hackers is to attach the patch to the email and discuss it inline in the thread. That way all versions of the patch as well as the discuss

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-03-02 Thread Amit Kapila
On Thu, Mar 2, 2023 at 3:00 PM Önder Kalacı wrote: >> >> Few comments: >> === >> 1. >> + identity. When replica identity FULL is specified, >> + indexes can be used on the subscriber side for searching the rows. These >> + indexes should be btree, >> >> Why only btree and not ot

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Jehan-Guillaume de Rorthais
On Thu, 2 Mar 2023 01:30:27 +0100 Tomas Vondra wrote: > On 3/2/23 00:18, Jehan-Guillaume de Rorthais wrote: > >>> ExecHashIncreaseNumBatches was really chatty, having hundreds of thousands > >>> of calls, always short-cut'ed to 1048576, I guess because of the > >>> conditional block «/* safety che

Re: Make some xlogreader messages more accurate

2023-03-02 Thread Bharath Rupireddy
On Thu, Mar 2, 2023 at 1:51 PM Peter Eisentraut wrote: > > On 28.02.23 07:15, Bharath Rupireddy wrote: > >> Going through the remaining report_invalid_record() calls I then > >> adjusted the use of "invalid" vs. "incorrect" in one case. The message > >> "record with invalid length" makes it sound

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-03-02 Thread Bharath Rupireddy
On Wed, Mar 1, 2023 at 2:39 PM Bharath Rupireddy wrote: > > Please find the attached v5 patch set for further review. I simplified the code largely by moving the logic of reading the WAL buffer page from a separate function to the main while loop. This enabled me to get rid of XLogReadFromBuffers

Re: Refactoring SysCacheGetAttr to know when attr cannot be NULL

2023-03-02 Thread Daniel Gustafsson
> On 2 Mar 2023, at 10:59, Peter Eisentraut > wrote: > > On 28.02.23 21:14, Daniel Gustafsson wrote: >> The attached refactoring introduce SysCacheGetAttrNotNull as a wrapper around >> SysCacheGetAttr where a NULL value triggers an elog(). This removes a lot of >> boilerplate error handling whi

Re: Refactoring SysCacheGetAttr to know when attr cannot be NULL

2023-03-02 Thread Daniel Gustafsson
> On 1 Mar 2023, at 00:20, Tom Lane wrote: > Also ... at least in assert-enabled builds, maybe we could check that > the column being fetched this way is actually marked attnotnull? > That would help to catch misuse. We could, but that would limit the API to attnotnull columns, rather than when

Re: Real config values for bytes needs quotes?

2023-03-02 Thread Peter Eisentraut
On 27.02.23 09:32, Kyotaro Horiguchi wrote: I found it frustrating that the line "shared_buffers = 0.1GB" in postgresql.conf postgresql.conf was causing an error and that the value required (additional) surrounding single quotes. The attached patch makes the parser accept the use of non-quoted r

Re: pg_upgrade and logical replication

2023-03-02 Thread Julien Rouhaud
On Thu, Mar 02, 2023 at 03:47:53PM +0530, Amit Kapila wrote: > On Wed, Mar 1, 2023 at 12:25 PM Julien Rouhaud wrote: > > > > 1) setup a normal physical replication cluster (pg_basebackup, restoring > > PITR, > > whatever), let's call the primary node "A" and replica node "B" > > 2) ensure W

Re: typedef struct LogicalDecodingContext

2023-03-02 Thread Peter Eisentraut
On 02.03.23 03:46, Tom Lane wrote: Peter Smith writes: Apparently, not all C99 compilers can be assumed to work using the strict C99 rules. While googling this issue I came across a statement that clang currently defaults to C17 rules. Even relatively old compilers might default to C11. But

Re: typedef struct LogicalDecodingContext

2023-03-02 Thread Peter Eisentraut
On 02.03.23 04:00, Tom Lane wrote: I wrote: I'm a little inclined to see if I can turn on -std=gnu99 on my clang-based buildfarm animals. I use that with gcc for my normal development activities, but now that I see that clang catches some things gcc doesn't ... FTR: done on sifaka and longfin

Re: meson: Non-feature feature options

2023-03-02 Thread Nazir Bilal Yavuz
Hi, Thanks for the review. On Wed, 1 Mar 2023 at 18:52, Peter Eisentraut wrote: > > Maybe we can make some of the logic less nested. Right now there is > > if sslopt != 'none' > >if not ssl.found() and sslopt in ['auto', 'openssl'] > > I think at that point, ssl.found() is never tr

Re: [HACKERS] make async slave to wait for lsn to be replayed

2023-03-02 Thread Peter Eisentraut
On 28.02.23 11:10, Kartyshov Ivan wrote: 3) Procedure style: Tom Lane and Kyotaro (wait_proc_v1.patch) https://www.postgresql.org/message-id/27171.1586439221%40sss.pgh.pa.us https://www.postgresql.org/message-id/20210121.173009.235021120161403875.horikyota.ntt%40gmail.com == advantages: n

Re: pg_upgrade and logical replication

2023-03-02 Thread Amit Kapila
On Wed, Mar 1, 2023 at 12:25 PM Julien Rouhaud wrote: > > On Wed, Mar 01, 2023 at 11:51:49AM +0530, Amit Kapila wrote: > > On Tue, Feb 28, 2023 at 10:18 AM Julien Rouhaud wrote: > > > > > > > Okay, but it would be better if you list out your detailed steps. It > > would be useful to support the n

  1   2   >