Re: [HACKERS] logical decoding of two-phase transactions

2021-03-17 Thread Amit Kapila
On Wed, Mar 17, 2021 at 11:27 AM Amit Kapila wrote: > > 5. I have modified the comments atop worker.c to explain the design > and some of the problems clearly. See attached. If you are fine with > this, please include it in the next version of the patch. > I have further expanded these comments

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-17 Thread Fujii Masao
On 2021/03/18 11:59, kuroda.hay...@fujitsu.com wrote: Dear Ikeda-san, I confirmed new patch and no problem was found. Thanks. (I'm not a native English speaker, so I cannot check your comments correctly, sorry) One user-visible side-effect by this change is; with the patch, the stats is

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Pavel Stehule
st 17. 3. 2021 v 23:16 odesílatel Hannu Krosing napsal: > why are you using yet another special syntax for this ? > > would it not be better to do something like this: > CREATE FUNCTION a_reall_long_and_winding_function_name(i int, out o int) > LANGUAGE plpgsql AS $plpgsql$ > DECLARE >args

Re: pg_amcheck contrib application

2021-03-17 Thread Tom Lane
Mark Dilger writes: >> On Mar 16, 2021, at 12:52 PM, Robert Haas wrote: >> Since we now know that shutting autovacuum off makes the problem go >> away, I don't see a reason to commit 0001. We should fix pg_amcheck >> instead, if, as presently seems to be the case, that's where the >> problem is.

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Pavel Stehule
st 17. 3. 2021 v 23:32 odesílatel Michael Paquier napsal: > On Wed, Mar 17, 2021 at 05:04:48PM +0100, Pavel Stehule wrote: > > This tree has a different direction than is usual, and then replacing the > > root node is not simple. > > Yeah, it is not like we should redesign this whole part just

Re: pg_amcheck contrib application

2021-03-17 Thread Mark Dilger
> On Mar 16, 2021, at 12:52 PM, Robert Haas wrote: > > On Mon, Mar 15, 2021 at 10:10 PM Mark Dilger > wrote: >> It is unfortunate that the failing test only runs pg_amcheck after creating >> numerous corruptions, as we can't know if pg_amcheck would have complained >> about pg_statistic

Re: PITR promote bug: Checkpointer writes to older timeline

2021-03-17 Thread Michael Paquier
On Wed, Mar 17, 2021 at 05:09:50PM +0900, Michael Paquier wrote: > Currently with HEAD and back branches, nothing would be broken as > logical contexts cannot exist in recovery. Still it would be easy > to miss the new behavior for anybody attempting to work more on this > feature in the future

RE: libpq debug log

2021-03-17 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > In pqTraceOutputString(), you can use the return value from fprintf to > move the cursor -- no need to count chars. Yes, precisely, 2 bytes for the double quotes needs to be subtracted as follows: len = fprintf(...); *cursor += (len - 2); > I still think

RE: Parallel INSERT (INTO ... SELECT ...)

2021-03-17 Thread houzj.f...@fujitsu.com
> > If a table parameter value is set and the > > equivalent toast. parameter is not, the TOAST table > > will use the table's parameter value. > > -Specifying these parameters for partitioned tables is not supported, > > -but you may specify them for individual leaf

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-17 Thread Amit Kapila
On Thu, Mar 18, 2021 at 8:30 AM Justin Pryzby wrote: > > diff --git a/doc/src/sgml/ref/create_table.sgml > b/doc/src/sgml/ref/create_table.sgml > index ff1b642722..d5d356f2de 100644 > --- a/doc/src/sgml/ref/create_table.sgml > +++ b/doc/src/sgml/ref/create_table.sgml > @@ -1338,8 +1338,10 @@

Re: New IndexAM API controlling index vacuum strategies

2021-03-17 Thread Peter Geoghegan
On Wed, Mar 17, 2021 at 7:16 PM Masahiko Sawada wrote: > Since I was thinking that always skipping index vacuuming on > anti-wraparound autovacuum is legitimate, skipping index vacuuming > only when we're really close to the point of going into read-only mode > seems a bit conservative, but maybe

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: > The most glaring case is the RelationInitTableAccessMethod() call in > RelationBuildLocalRelation(). Seems like the best fix is to just move > the MemoryContextSwitchTo() to just before the > RelationInitTableAccessMethod(). Although I wonder if we shouldn't go > further,

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
On 2021-03-17 22:33:59 -0400, Tom Lane wrote: > >> I've not yet tried to grok the comment that purports to justify it, > >> but I fail to see why it'd ever be useful to drop values inherited > >> from the postmaster. > > > I can't really make sense of it either. I think it may be trying to > >

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 00:01:55 -0400, Tom Lane wrote: > As for the particular point about ParallelBlockTableScanWorkerData, > I agree with your question to David about why that's in TableScanDesc > not HeapScanDesc, but I can't get excited about it not being freed in > heap_endscan. That's mainly

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Michael Paquier
On Wed, Mar 17, 2021 at 07:30:04PM -0700, Andres Freund wrote: > I suspect it might be easier to reproduce the issue with smaller WAL > segments, a short checkpoint_timeout, and multiple jobs generating WAL > and then sleeping for random amounts of time. Not sure if that's the > sole ingredient,

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-17 Thread Justin Pryzby
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index ff1b642722..d5d356f2de 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -1338,8 +1338,10 @@ WITH ( MODULUS numeric_literal, REM If a table parameter value is

RE: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-17 Thread kuroda.hay...@fujitsu.com
Dear Ikeda-san, I confirmed new patch and no problem was found. Thanks. (I'm not a native English speaker, so I cannot check your comments correctly, sorry) Best Regards, Hayato Kuroda FUJITSU LIMITED

Re: New IndexAM API controlling index vacuum strategies

2021-03-17 Thread Peter Geoghegan
On Mon, Mar 15, 2021 at 4:11 PM Andres Freund wrote: > I kinda wonder whether this case should just be handled by just gotoing > back to the start of the blkno loop, and redoing the pruning. The only > thing that makes that a bit more complicatd is that we've already > incremented

RE: libpq debug log

2021-03-17 Thread tsunakawa.ta...@fujitsu.com
I'll look at the comments from Alvaro-san and Horiguchi-san. Here are my review comments: (23) + /* Trace message only when there is first 1 byte */ + if (conn->Pfdebug && conn->outCount < conn->outMsgStart) + { + if (conn->outCount < conn->outMsgStart) +

Re: Parallel INSERT (INTO ... SELECT ...)

2021-03-17 Thread Justin Pryzby
On Fri, Mar 12, 2021 at 04:05:09PM +0900, Amit Langote wrote: > On Fri, Mar 12, 2021 at 6:10 AM Justin Pryzby wrote: > > Note also this CF entry > > https://commitfest.postgresql.org/32/2987/ > > | Allow setting parallel_workers on partitioned tables I'm rebasing that other patch on top of

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: > On 2021-03-17 21:30:48 -0400, Tom Lane wrote: >> I believe I've just tracked down the cause of that. Those errors >> are (as far as I've seen) only happening in parallel workers, and >> the reason is this gem in RestoreGUCState: ... > Ouch. At least it's a short lived

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Andres Freund
Hi, On 2021-03-18 09:55:46 +0900, Michael Paquier wrote: > Let's see how it goes from this point, but, FWIW, I have not been able > to reproduce again my similar problem with the archive command :/ -- I suspect it might be easier to reproduce the issue with smaller WAL segments, a short

Re: New IndexAM API controlling index vacuum strategies

2021-03-17 Thread Masahiko Sawada
On Wed, Mar 17, 2021 at 7:21 AM Peter Geoghegan wrote: > > On Tue, Mar 16, 2021 at 6:40 AM Masahiko Sawada wrote: > > > Note that I've merged multiple existing functions in vacuumlazy.c into > > > one: the patch merges lazy_vacuum_all_indexes() and lazy_vacuum_heap() > > > into a single function

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 21:30:48 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2021-03-17 18:07:36 -0400, Tom Lane wrote: > >> Huh, interesting. I wonder why that makes the ident problem go away? > > > I spent a bit of time looking at valgrind, and it looks to be explicit > > behaviour: > >

Re: EXPLAIN/EXPLAIN ANALYZE REFRESH MATERIALIZED VIEW

2021-03-17 Thread Japin Li
On Tue, 16 Mar 2021 at 20:13, Bharath Rupireddy wrote: > On Tue, Mar 16, 2021 at 1:15 AM Tom Lane wrote: >> >> [ Sorry for not looking at this thread sooner ] >> >> Bharath Rupireddy writes: >> > Currently, $subject is not allowed. We do plan the mat view query >> > before every refresh. I

Re: replication slot stats memory bug

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 19:36:46 -0400, Tom Lane wrote: > > But it seems like we just shouldn't allocate it dynamically at all? > > max_replication_slots doesn't change during postmaster lifetime, so it > > seems like it should just be allocated once? > > Meh. I don't see a need to wire in such an

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Andres Freund
Hi, On 2021-03-16 16:20:37 +0900, Michael Paquier wrote: > Fujii-san has mentioned that on twitter, but one area that has changed > during the v13 cycle is aaa3aed, where the code recycling segments has > been switched from a pgrename() (with a retry loop) to a > CreateHardLinkA()+pgunlink()

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Justin Pryzby
On Wed, Mar 17, 2021 at 02:50:34PM -0700, Andres Freund wrote: > On 2021-03-17 16:01:58 -0400, Robert Haas wrote: > > > - why is HIDE_TOAST_COMPRESSION useful? Doesn't quite seem to be > > > comparable to HIDE_TABLEAM? > > > > Andres, what do you mean by this exactly? It's exactly the same

Re: replication slot stats memory bug

2021-03-17 Thread Amit Kapila
On Thu, Mar 18, 2021 at 4:55 AM Andres Freund wrote: > > Hi, > > On 2021-03-17 16:04:47 -0700, Andres Freund wrote: > > I'll push the minimal fix of forcing the allocation to happen in > > pgStatLocalContext and setting it to NULL in pgstat_clear_snapshot(). > > Done: >

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: > On 2021-03-17 18:07:36 -0400, Tom Lane wrote: >> Huh, interesting. I wonder why that makes the ident problem go away? > I spent a bit of time looking at valgrind, and it looks to be explicit > behaviour: > >// Our goal is to construct a set of chunks that includes

Re: Calendar support in localization

2021-03-17 Thread Vik Fearing
On 3/17/21 3:48 PM, Tom Lane wrote: > Also, the SQL spec says in so many words > that the SQL-defined datetime types follow the Gregorian calendar. We already don't follow the SQL spec for timestamps (and I, for one, think we are better for it) so I don't think we should worry about that. -- Vik

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Michael Paquier
On Tue, Mar 16, 2021 at 11:40:12AM +0100, Magnus Hagander wrote: > If we can provide a new .EXE built with exactly the same flags as the > EDB downloads that they can just drop into a directory, I think it's a > lot easier to get that done. Yeah, multiple people have been complaining about that

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 18:07:36 -0400, Tom Lane wrote: > Andres Freund writes: > >> I found a way around that late last night. Need to mark the context > >> itself as an allocation. But I made a mess on the way to that and need > >> to clean the patch up before sending it (and need to drop my > >>

Re: replication slot stats memory bug

2021-03-17 Thread Tom Lane
Andres Freund writes: > I saw a leak in pgstat_read_statsfiles(), more precisely: > /* Allocate the space for replication slot statistics */ > replSlotStats = palloc0(max_replication_slots * > sizeof(PgStat_ReplSlotStats)); Yeah, I just found that myself. I think your fix is good.

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Julien Rouhaud
On Wed, Mar 17, 2021 at 06:32:16PM -0400, Bruce Momjian wrote: > On Wed, Mar 17, 2021 at 04:04:44PM -0400, Robert Haas wrote: > > On Wed, Mar 17, 2021 at 12:48 PM Julien Rouhaud wrote: > > > I originally suggested to make it clearer by having an enum GUC rather > > > than a > > > boolean, say

Re: replication slot stats memory bug

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 16:04:47 -0700, Andres Freund wrote: > I'll push the minimal fix of forcing the allocation to happen in > pgStatLocalContext and setting it to NULL in pgstat_clear_snapshot(). Done:

replication slot stats memory bug

2021-03-17 Thread Andres Freund
Hi, in the course of https://postgr.es/m/3471359.1615937770%40sss.pgh.pa.us I saw a leak in pgstat_read_statsfiles(), more precisely: /* Allocate the space for replication slot statistics */ replSlotStats = palloc0(max_replication_slots * sizeof(PgStat_ReplSlotStats)); the issue

Re: WIP: WAL prefetch (another approach)

2021-03-17 Thread Tomas Vondra
Hi, On 3/17/21 10:43 PM, Stephen Frost wrote: > Greetings, > > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: >> Right, I was just going to point out the FPIs are not necessary - what >> matters is the presence of long streaks of WAL records touching the same >> set of blocks. But people

Re: Calendar support in localization

2021-03-17 Thread Thomas Munro
On Thu, Mar 18, 2021 at 3:48 AM Tom Lane wrote: > Robert Haas writes: > > It's not very obvious how to scale this kind of approach to a wide > > variety of calendar types, and as Thomas says, it would much cooler to > > be able to handle all of the ones that ICU knows how to support rather > >

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Michael Paquier
On Wed, Mar 17, 2021 at 05:04:48PM +0100, Pavel Stehule wrote: > This tree has a different direction than is usual, and then replacing the > root node is not simple. Yeah, it is not like we should redesign this whole part just for the feature discussed here, and that may impact performance as the

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Wed, Mar 17, 2021 at 04:04:44PM -0400, Robert Haas wrote: > On Wed, Mar 17, 2021 at 12:48 PM Julien Rouhaud wrote: > > I originally suggested to make it clearer by having an enum GUC rather than > > a > > boolean, say compute_queryid = [ none | core | external ], and if set to > > external

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Hannu Krosing
why are you using yet another special syntax for this ? would it not be better to do something like this: CREATE FUNCTION a_reall_long_and_winding_function_name(i int, out o int) LANGUAGE plpgsql AS $plpgsql$ DECLARE args function_name_alias BEGIN args.o = 2 * args.i END; $plpgsql$; or at

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: >> I found a way around that late last night. Need to mark the context >> itself as an allocation. But I made a mess on the way to that and need >> to clean the patch up before sending it (and need to drop my >> girlfriend off first). > Unfortunately I didn't immediately

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 16:01:58 -0400, Robert Haas wrote: > > - why is HIDE_TOAST_COMPRESSION useful? Doesn't quite seem to be > > comparable to HIDE_TABLEAM? > > Andres, what do you mean by this exactly? It's exactly the same issue: > without this, if you change the default compression method,

Re: WIP: WAL prefetch (another approach)

2021-03-17 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > Right, I was just going to point out the FPIs are not necessary - what > matters is the presence of long streaks of WAL records touching the same > set of blocks. But people with workloads where this is common likely > don't need

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Tomas Vondra
On 3/17/21 9:58 PM, Dean Rasheed wrote: > On Wed, 17 Mar 2021 at 20:48, Dean Rasheed wrote: >> >> For reference, here is the test case I was using (which isn't really very >> good for >> catching dependence between columns): >> > > And here's a test case with much more dependence between the

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Dean Rasheed
On Wed, 17 Mar 2021 at 20:48, Dean Rasheed wrote: > > For reference, here is the test case I was using (which isn't really very > good for > catching dependence between columns): > And here's a test case with much more dependence between the columns: DROP TABLE IF EXISTS foo; CREATE TABLE foo

Re: WIP: BRIN multi-range indexes

2021-03-17 Thread John Naylor
On Wed, Mar 17, 2021 at 3:16 PM Tomas Vondra wrote: > Ummm, no attachment ;-) Oops, here it is. -- John Naylor EDB: http://www.enterprisedb.com jcn-costing-test.sql Description: Binary data

Re: non-HOT update not looking at FSM for large tuple update

2021-03-17 Thread John Naylor
On Fri, Mar 12, 2021 at 8:45 AM Matthias van de Meent < boekewurm+postg...@gmail.com> wrote: > > If this case isn't added, the lower else branch will fail to find > fitting pages for len > maxPaddedFsmRequest tuples; potentially > extending the relation when there is actually still enough space >

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2021-03-17 Thread Simon Riggs
On Fri, 12 Mar 2021 at 22:16, Tomas Vondra wrote: > > On 1/28/21 2:33 PM, Simon Riggs wrote: > > On Thu, 28 Jan 2021 at 12:53, Masahiko Sawada wrote: > > > >> This entry has been "Waiting on Author" status and the patch has not > >> been updated since Nov 30. Are you still planning to work on

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Dean Rasheed
On Wed, 17 Mar 2021 at 19:07, Tomas Vondra wrote: > > On 3/17/21 7:54 PM, Dean Rasheed wrote: > > > > it might have been better to estimate the first case as > > > > ndistinct((a+b)) * ndistinct(c) * ndistinct(d) > > > > and the second case as > > > > ndistinct((a+b)) * ndistinct((c+d))

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Robert Haas
On Wed, Mar 17, 2021 at 12:48 PM Julien Rouhaud wrote: > I originally suggested to make it clearer by having an enum GUC rather than a > boolean, say compute_queryid = [ none | core | external ], and if set to > external then a hook would be explicitely called. Right now, "none" and > "external"

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Robert Haas
).On Mon, Mar 15, 2021 at 6:58 PM Andres Freund wrote: > - Adding all these indirect function calls via toast_compression[] just > for all of two builtin methods isn't fun either. Yeah, it feels like this has too many layers of indirection now. Like, toast_decompress_datum() first gets

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Robert Haas
On Wed, Mar 17, 2021 at 2:17 PM Andres Freund wrote: > OTOH heap_form_flattened_tuple() has the advantage that we can optimize > it further (e.g. to do the conversion to flattened values in fill_val()) > without changing the outside API. Well, in my view, that does change the outside API,

Re: WIP: BRIN multi-range indexes

2021-03-17 Thread Tomas Vondra
On 3/17/21 7:59 PM, John Naylor wrote: > On Thu, Mar 11, 2021 at 12:26 PM Tomas Vondra > mailto:tomas.von...@enterprisedb.com>> > wrote: >> >> Hi, >> >> Here is an updated version of the patch series. >> >> It fixes the assert failure (just remove the multiplication from it) and >> adds a

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Tomas Vondra
On 3/17/21 7:54 PM, Dean Rasheed wrote: > On Wed, 17 Mar 2021 at 17:26, Tomas Vondra > wrote: >> >> My concern is that the current behavior (where we prefer expression >> stats over multi-column stats to some extent) works fine as long as the >> parts are independent, but once there's

Re: WIP: BRIN multi-range indexes

2021-03-17 Thread John Naylor
On Thu, Mar 11, 2021 at 12:26 PM Tomas Vondra wrote: > > Hi, > > Here is an updated version of the patch series. > > It fixes the assert failure (just remove the multiplication from it) and > adds a simple regression test that exercises this. > > Based on the discussion so far, I've decided to

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Dean Rasheed
On Wed, 17 Mar 2021 at 17:26, Tomas Vondra wrote: > > My concern is that the current behavior (where we prefer expression > stats over multi-column stats to some extent) works fine as long as the > parts are independent, but once there's dependency it's probably more > likely to produce

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-17 Thread Justin Pryzby
The v8 patch has the "broken constraint" problem. Also, it "fails to avoid" adding duplicate constraints: Check constraints: "c" CHECK (i IS NOT NULL AND i > 1 AND i < 2) "cc" CHECK (i IS NOT NULL AND i >= 1 AND i < 2) "p1_check" CHECK (true) "p1_i_check" CHECK (i IS NOT NULL AND

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 13:31:14 -0400, Robert Haas wrote: > On Wed, Mar 17, 2021 at 7:41 AM Dilip Kumar wrote: > > 0002: > > - Wrapper over heap_form_tuple and used in ExecEvalRow() and > > ExecEvalFieldStoreForm() > > Instead of having heap_form_flattened_tuple(), how about >

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, (really need to fix my mobile phone mail program to keep the CC list...) On 2021-03-17 08:15:43 -0700, Andres Freund wrote: > On Wed, Mar 17, 2021, at 07:16, Tom Lane wrote: > > Andres Freund writes: > > > On 2021-03-16 20:50:17 -0700, Andres Freund wrote: > > Meanwhile, I'm still trying to

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-03-17 Thread Alvaro Herrera
On 2021-Mar-15, Alvaro Herrera wrote: > Here's a fixup patch to do it that way. I tried running the commands > you showed and one of them immediately dies with the new error message; > I can't cause the bogus constraint to show up anymore. Actually, that was a silly fix that didn't actually

Re: Index Skip Scan (new UniqueKeys)

2021-03-17 Thread Tomas Vondra
On 3/17/21 6:02 PM, Dmitry Dolgov wrote: >> On Wed, Mar 17, 2021 at 03:28:00AM +0100, Tomas Vondra wrote: >> Hi, >> >> I took a look at the new patch series, focusing mostly on the uniquekeys >> part. It'd be a bit tedious to explain all the review comments here, so >> attached is a patch

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Robert Haas
On Wed, Mar 17, 2021 at 7:41 AM Dilip Kumar wrote: > 0002: > - Wrapper over heap_form_tuple and used in ExecEvalRow() and > ExecEvalFieldStoreForm() Instead of having heap_form_flattened_tuple(), how about heap_flatten_values(tupleDesc, values, isnull) that is documented to modify the values

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Tomas Vondra
Hi, On 3/17/21 4:55 PM, Dean Rasheed wrote: > On Sun, 7 Mar 2021 at 21:10, Tomas Vondra > wrote: >> >> 2) ndistinct >> >> There's one thing that's bugging me, in how we handle "partial" matches. >> For each expression we track both the original expression and the Vars >> we extract from it. If

Re: pg_dump new feature: exporting functions only. Bad or good idea ?

2021-03-17 Thread Tom Lane
Vik Fearing writes: > On 3/17/21 6:00 PM, Lætitia Avrot wrote: >> However, when I finally got the time to look at it in detail, I found out >> there was no way to solve the dependencies in the functions and procedures, >> so that the exported file, when re-played could lead to invalid objects. >>

Re: pg_dump new feature: exporting functions only. Bad or good idea ?

2021-03-17 Thread Fabrízio de Royes Mello
On Wed, Mar 17, 2021 at 2:00 PM Lætitia Avrot wrote: > > Hey hackers, > > I had this idea, that I raised and cherished like my baby to add a switch in `pg_dump` to allow exporting stored functions (and procedures) only. > > However, when I finally got the time to look at it in detail, I found out

Re: [patch] [doc] Minor variable related cleanup and rewording of plpgsql docs

2021-03-17 Thread Tom Lane
I wrote: > I took a stab at doing that, just to see what it might look like. > I thought it comes out pretty well, really -- see what you think. Hearing nothing further, I pushed that after another round of copy-editing. There's still plenty of time to revise it if anybody has further comments.

Re: pg_dump new feature: exporting functions only. Bad or good idea ?

2021-03-17 Thread Vik Fearing
On 3/17/21 6:00 PM, Lætitia Avrot wrote: > Hey hackers, > > I had this idea, that I raised and cherished like my baby to add a switch > in `pg_dump` to allow exporting stored functions (and procedures) only. > > However, when I finally got the time to look at it in detail, I found out > there

pg_dump new feature: exporting functions only. Bad or good idea ?

2021-03-17 Thread Lætitia Avrot
Hey hackers, I had this idea, that I raised and cherished like my baby to add a switch in `pg_dump` to allow exporting stored functions (and procedures) only. However, when I finally got the time to look at it in detail, I found out there was no way to solve the dependencies in the functions and

Re: Index Skip Scan (new UniqueKeys)

2021-03-17 Thread Dmitry Dolgov
> On Wed, Mar 17, 2021 at 03:28:00AM +0100, Tomas Vondra wrote: > Hi, > > I took a look at the new patch series, focusing mostly on the uniquekeys > part. It'd be a bit tedious to explain all the review comments here, so > attached is a patch series with a "review" patch for some of the parts.

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Julien Rouhaud
On Wed, Mar 17, 2021 at 12:24:44PM -0400, Bruce Momjian wrote: > On Wed, Mar 17, 2021 at 05:16:50PM +0100, Pavel Stehule wrote: > > > > > > st 17. 3. 2021 v 17:03 odesílatel Tom Lane napsal: > > > > Bruce Momjian writes: > > > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote:

Re: libpq debug log

2021-03-17 Thread Alvaro Herrera
Hello In pqTraceOutputString(), you can use the return value from fprintf to move the cursor -- no need to count chars. I still think that the message-type specific functions should print the message type, rather than having the string arrays. -- Álvaro Herrera Valdivia, Chile "La gente

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Wed, Mar 17, 2021 at 05:16:50PM +0100, Pavel Stehule wrote: > > > st 17. 3. 2021 v 17:03 odesílatel Tom Lane napsal: > > Bruce Momjian writes: > > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: > >> I still say that it's a serious mistake to sanctify a query ID >

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Pavel Stehule
st 17. 3. 2021 v 17:03 odesílatel Tom Lane napsal: > Bruce Momjian writes: > > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: > >> I still say that it's a serious mistake to sanctify a query ID > calculation > >> method that was designed only for pg_stat_statement's needs as the one

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Wed, Mar 17, 2021 at 12:01:38PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: > >> I still say that it's a serious mistake to sanctify a query ID calculation > >> method that was designed only for pg_stat_statement's needs as the

Re: WIP: WAL prefetch (another approach)

2021-03-17 Thread Tomas Vondra
On 2/15/21 12:18 AM, Stephen Frost wrote: > Greetings, > > ... > I think there's potential for some significant optimization going forward, but I think it's basically optimization over what we're doing today. As this is already a nontrivial patch, I'd argue for doing so

Re: pl/pgsql feature request: shorthand for argument and local variable references

2021-03-17 Thread Pavel Stehule
st 17. 3. 2021 v 9:20 odesílatel Michael Paquier napsal: > On Wed, Mar 17, 2021 at 02:06:57PM +0800, Julien Rouhaud wrote: > > I also think that there should be a single usable top label, otherwise > it will > > lead to confusing code and it can be a source of bug. > > Okay, that's fine by me.

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Tom Lane
Bruce Momjian writes: > On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: >> I still say that it's a serious mistake to sanctify a query ID calculation >> method that was designed only for pg_stat_statement's needs as the one >> true way to do it. But that's what exposing it in a core

Re: PoC/WIP: Extended statistics on expressions

2021-03-17 Thread Dean Rasheed
On Sun, 7 Mar 2021 at 21:10, Tomas Vondra wrote: > > 2) ndistinct > > There's one thing that's bugging me, in how we handle "partial" matches. > For each expression we track both the original expression and the Vars > we extract from it. If we can't find a statistics matching the whole >

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Wed, Mar 17, 2021 at 11:28:38AM -0400, Tom Lane wrote: > Bruce Momjian writes: > > We are reaching the two-year mark on this feature, that everyone seems > > to agree is needed. Is any committer going to work on this to get it > > into PG 14? Should I take it? > > I still say that it's a

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Tom Lane
Bruce Momjian writes: > We are reaching the two-year mark on this feature, that everyone seems > to agree is needed. Is any committer going to work on this to get it > into PG 14? Should I take it? I still say that it's a serious mistake to sanctify a query ID calculation method that was

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2021-03-17 Thread Bruce Momjian
On Sun, Mar 14, 2021 at 04:06:45PM +0800, Julien Rouhaud wrote: > Recent conflict, thanks to cfbot. v18 attached. We are reaching the two-year mark on this feature, that everyone seems to agree is needed. Is any committer going to work on this to get it into PG 14? Should I take it? I just

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On Wed, Mar 17, 2021, at 07:16, Tom Lane wrote: > Andres Freund writes: > > On 2021-03-16 20:50:17 -0700, Andres Freund wrote: > Meanwhile, I'm still trying to understand why valgrind is whining > about the rd_indexcxt identifier strings. AFAICS it shouldn't. I found a way around that late

RE: libpq debug log

2021-03-17 Thread iwata....@fujitsu.com
Hi Tsunakawa san, Alvaro san, Thank you very much for your review. It helped me a lot to make the patch better. I update patch to v26. This patch has been updated according to Tsunakawa san and Alvaro san review comments. The output is following; ``` 2021-03-17 14:43:16.411238 > Terminate

Re: Calendar support in localization

2021-03-17 Thread Tom Lane
Robert Haas writes: > It's not very obvious how to scale this kind of approach to a wide > variety of calendar types, and as Thomas says, it would much cooler to > be able to handle all of the ones that ICU knows how to support rather > than just one. But, the problem I see with using timestamptz

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Tom Lane
Andres Freund writes: > On 2021-03-16 20:50:17 -0700, Andres Freund wrote: >> What I meant was that I didn't understand how there's not a leak >> danger when compilation fails halfway through, given that the context >> in question is below TopMemoryContext and that I didn't see a relevant >> TRY

Re: Calendar support in localization

2021-03-17 Thread Robert Haas
On Wed, Mar 17, 2021 at 9:54 AM Surafel Temesgen wrote: > As you mention above whatever the calendar type is we ended up storing an > integer that represent the date so rather than re-implementing every > function and operation for every calendar we can use existing Gerigorian >

Re: Calendar support in localization

2021-03-17 Thread Surafel Temesgen
On Tue, Mar 16, 2021 at 12:20 PM Thomas Munro wrote: > On Wed, Mar 17, 2021 at 6:31 AM Surafel Temesgen > wrote: > > Ethiopice calendar have 13 months so it can not be stored as date and > timestamp type and you approach seems more complicated and i suggest to > have this feature on the purpose

Re: pgsql: Add libpq pipeline mode support to pgbench

2021-03-17 Thread Daniel Verite
Fabien COELHO wrote: > For consistency with the existing \if … \endif, ISTM that it could have > been named \batch … \endbatch or \pipeline … \endpipeline? "start" mirrors "end". To me, the analogy with \if-\endif is not obvious. Grammatically \if is meant to introduce the expression

Re: Get memory contexts of an arbitrary backend process

2021-03-17 Thread torikoshia
On 2021-03-05 14:22, Fujii Masao wrote: On 2021/03/04 18:32, torikoshia wrote: On 2021-01-14 19:11, torikoshia wrote: Since pg_get_target_backend_memory_contexts() waits to dump memory and it could lead dead lock as below.   - session1   BEGIN; TRUNCATE t;   - session2   BEGIN; TRUNCATE t;

Re: Assertion failure with barriers in parallel hash join

2021-03-17 Thread Thomas Munro
On Wed, Mar 17, 2021 at 6:58 PM Thomas Munro wrote: > According to BF animal elver there is something wrong with this > commit. Looking into it. Assertion failure reproduced here and understood, but unfortunately it'll take some more time to fix this. I've reverted the commit for now to

Re: proposal: schema variables - doc

2021-03-17 Thread Erik Rijkers
> On 2021.03.13. 07:01 Pavel Stehule wrote: > Hi > fresh rebase > [schema-variables-20210313.patch.gz] Hi Pavel, I notice that the phrase 'schema variable' is not in the index at the end ('bookindex.html'). Not good. It is also not in the index at the front of the manual - also not good.

Re: a verbose option for autovacuum

2021-03-17 Thread Euler Taveira
On Wed, Mar 10, 2021, at 12:46 AM, Masahiko Sawada wrote: > Attached a patch. I've slightly modified the format for consistency > with heap statistics. Since commit 5f8727f5a6, this patch doesn't apply anymore. Fortunately, it is just a small hunk. I reviewed this patch and it looks good to me.

Re: logical replication worker accesses catalogs in error context callback

2021-03-17 Thread Bharath Rupireddy
On Tue, Mar 16, 2021 at 2:21 AM Tom Lane wrote: > > Thanks for pointing to the changes in the commit message. I corrected > > them. Attaching v4 patch set, consider it for further review. > > I took a quick look at this. I'm quite worried about the potential > performance cost of the v4-0001

Re: fdatasync performance problem with large number of DB files

2021-03-17 Thread Paul Guo
On Wed, Mar 17, 2021 at 11:45 AM Thomas Munro wrote: > > On Tue, Mar 16, 2021 at 9:29 PM Fujii Masao > wrote: > > On 2021/03/16 8:15, Thomas Munro wrote: > > > I don't want to add a hypothetical sync_after_crash=none, because it > > > seems like generally a bad idea. We already have a > > >

RE: Parallel Inserts in CREATE TABLE AS

2021-03-17 Thread houzj.f...@fujitsu.com
> > Seems like v22 patch was failing in cfbot for one of the unstable test > > cases. > > Attaching v23 patch set with modification in 0003 and 0004 patches. No > > changes to 0001 and 0002 patches. Hopefully cfbot will be happy with v23. > > > > Please consider v23 for further review. > Hi, >

Re: pgsql: Add libpq pipeline mode support to pgbench

2021-03-17 Thread Fabien COELHO
Bonjour Daniel, Ola Alvaro, Add libpq pipeline mode support to pgbench New metacommands \startpipeline and \endpipeline allow the user to run queries in libpq pipeline mode. Author: Daniel Vérité Reviewed-by: Álvaro Herrera Discussion:

Re: A new function to wait for the backend exit after termination

2021-03-17 Thread Zhihong Yu
Hi, w.r.t. WaitLatch(), if its return value is WL_TIMEOUT, we know the specified timeout has elapsed. It seems WaitLatch() can be enhanced to also return the actual duration of the wait. This way, the caller can utilize the duration directly. As for other places where WaitLatch() is called,

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-03-17 Thread Bharath Rupireddy
On Wed, Mar 17, 2021 at 8:05 AM torikoshia wrote: > > I have not gone through that thread though. Is there any way we can > > detect those child processes(stats collector, sys logger) that are > > forked by the postmaster from a backend process? Thoughts? > > I couldn't find good ways to do that,

  1   2   >