RE: Confusing comment for function ExecParallelEstimate

2019-06-04 Thread Wu, Fei
Thanks for your reply. From the code below: (https://github.com/postgres/postgres/blob/REL_10_7/src/backend/executor/execParallel.c) ### 443 /* 444 * Give

Re: Parallel Append subplan order instability on aye-aye

2019-06-04 Thread Noah Misch
On Mon, May 20, 2019 at 11:15:47PM -0400, Tom Lane wrote: > I got around to excavating in the buildfarm archives, and found a round > dozen of more-or-less-similar incidents. I went back 18 months, which > by coincidence (i.e., I didn't realize it till just now) is just about > the time since

Re: Confusing comment for function ExecParallelEstimate

2019-06-04 Thread Amit Kapila
On Wed, Jun 5, 2019 at 9:24 AM Wu, Fei wrote: > > Hi, all > > Lately I was researching Parallelism of Postgres 10.7(and it is same in all > version), and I was confused when reading the comment of function > ExecParallelEstimate : > > (in src/backend/executor/execParallel.c) > >

Confusing comment for function ExecParallelEstimate

2019-06-04 Thread Wu, Fei
Hi, all Lately I was researching Parallelism of Postgres 10.7(and it is same in all version), and I was confused when reading the comment of function ExecParallelEstimate : (in src/backend/executor/execParallel.c) -- * While we're at it, count the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-04 Thread Masahiko Sawada
On Fri, May 10, 2019 at 2:42 AM Bruce Momjian wrote: > > On Thu, May 9, 2019 at 05:49:12PM +0900, Masahiko Sawada wrote: > > In terms of keys, one advantage could be that we have less keys with > > per-tablespace keys. > > > > Let me briefly explain the current design I'm thinking. The design >

Re: Sort support for macaddr8

2019-06-04 Thread Peter Geoghegan
On Tue, Jun 4, 2019 at 3:23 PM Andres Freund wrote: > > This is half the reason why I ended up implementing itemptr_encode() > > to accelerate the TID sort used by CREATE INDEX CONCURRENTLY some > > years back -- TID is 6 bytes wide, which doesn't have the necessary > > macro support within

Re: Binary support for pgoutput plugin

2019-06-04 Thread Dave Cramer
On Tue, 4 Jun 2019 at 18:08, Andres Freund wrote: > Hi, > > On 2019-06-05 00:05:02 +0200, David Fetter wrote: > > Would it make sense to work toward a binary format that's not > > architecture-specific? I recall from COPY that our binary format is > > not standardized across, for example, big-

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-04 Thread Tomas Vondra
On Tue, Jun 04, 2019 at 03:08:24PM -0400, Robert Haas wrote: On Tue, Jun 4, 2019 at 2:47 PM Melanie Plageman wrote: Oops! You are totally right. I will amend the idea: For each chunk on the inner side, loop through both the original batch file and the unmatched outer tuples file created for

Re: Sort support for macaddr8

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 15:10:07 -0700, Peter Geoghegan wrote: > On Tue, Jun 4, 2019 at 2:55 PM Andres Freund wrote: > > Yea, I don't immediately see a problem with doing that on a major > > version boundary. Obviously that'd only be possible for sizeof(Datum) == > > 8 == sizeof(macaddr8) platforms,

Re: Sort support for macaddr8

2019-06-04 Thread Peter Geoghegan
On Tue, Jun 4, 2019 at 2:55 PM Andres Freund wrote: > Yea, I don't immediately see a problem with doing that on a major > version boundary. Obviously that'd only be possible for sizeof(Datum) == > 8 == sizeof(macaddr8) platforms, but that's the vast majority these > days. And generally, I think

Re: Binary support for pgoutput plugin

2019-06-04 Thread Andres Freund
Hi, On 2019-06-05 00:05:02 +0200, David Fetter wrote: > Would it make sense to work toward a binary format that's not > architecture-specific? I recall from COPY that our binary format is > not standardized across, for example, big- and little-endian machines. I think you recall wrongly. It's

Re: Binary support for pgoutput plugin

2019-06-04 Thread David Fetter
On Tue, Jun 04, 2019 at 04:55:33PM -0400, Dave Cramer wrote: > On Tue, 4 Jun 2019 at 16:46, Andres Freund wrote: > > > Hi, > > > > On 2019-06-04 16:39:32 -0400, Dave Cramer wrote: > > > On Tue, 4 Jun 2019 at 16:30, Andres Freund < > > andres.fre...@enterprisedb.com> > > > wrote: > > > > >

Re: Sort support for macaddr8

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 14:55:16 -0700, Andres Freund wrote: > On 2019-06-04 17:37:35 -0400, Alvaro Herrera wrote: > I think it might be worthwhile to optimize things so that all typlen > 0 > && typlen <= sizeof(Datum) are allowed for byval datums. Maybe even just deprecate specifying byval at CREATE

Re: Sort support for macaddr8

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 17:37:35 -0400, Alvaro Herrera wrote: > On 2019-Jun-03, Peter Geoghegan wrote: > > As you know, it's a bit weird that we're proposing adding sort support > > with abbreviated keys for a type that is 8 bytes, since you'd expect > > it to also be pass-by-value on most platforms,

Re: Sort support for macaddr8

2019-06-04 Thread Alvaro Herrera
On 2019-Jun-03, Peter Geoghegan wrote: > As you know, it's a bit weird that we're proposing adding sort support > with abbreviated keys for a type that is 8 bytes, since you'd expect > it to also be pass-by-value on most platforms, which largely defeats > the purpose of having abbreviated keys

Re: Binary support for pgoutput plugin

2019-06-04 Thread Chapman Flack
On 6/4/19 4:39 PM, Dave Cramer wrote: > I haven't really thought this through completely but one place JDBC has > problems with binary is with > timestamps with timezone as we don't know which timezone to use. Is it safe > to assume everything is in UTC > since the server stores in UTC ? PL/Java,

Re: Sort support for macaddr8

2019-06-04 Thread Tomas Vondra
On Tue, Jun 04, 2019 at 01:49:23PM -0400, Stephen Frost wrote: Greetings, * Melanie Plageman (melanieplage...@gmail.com) wrote: Peter and I implemented this small (attached) patch to extend abbreviated key compare sort to macaddr8 datatype (currently supported for macaddr). Nice. I think

Update list of combining characters

2019-06-04 Thread Peter Eisentraut
In src/backend/utils/mb/wchar.c, function ucs_wcwidth(), there is a list of Unicode combining characters, so that those can be ignored for computing the display length of a Unicode string. It seems to me that that list is either outdated or plain incorrect. For example, the list starts with

Re: Binary support for pgoutput plugin

2019-06-04 Thread Dave Cramer
On Tue, 4 Jun 2019 at 16:46, Andres Freund wrote: > Hi, > > On 2019-06-04 16:39:32 -0400, Dave Cramer wrote: > > On Tue, 4 Jun 2019 at 16:30, Andres Freund < > andres.fre...@enterprisedb.com> > > wrote: > > > > There's also no reason that I am aware that binary outputs can't be > > > >

Re: Binary support for pgoutput plugin

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 16:39:32 -0400, Dave Cramer wrote: > On Tue, 4 Jun 2019 at 16:30, Andres Freund > wrote: > > > There's also no reason that I am aware that binary outputs can't be > > > supported. > > > > Well, it *does* increase version dependencies, and does make replication > > more

Re: nitpick about poor style in MergeAttributes

2019-06-04 Thread Alvaro Herrera
On 2019-May-23, Mark Dilger wrote: > On Thu, May 23, 2019 at 5:24 PM Michael Paquier wrote: > > Looking closer, this code is not new as of v12. We have that since > > e7b3349 which has introduced CREATE TABLE OF. Yeah, I was not quite understanding why it was being blamed on a commit that

Re: Binary support for pgoutput plugin

2019-06-04 Thread Dave Cramer
Dave Cramer On Tue, 4 Jun 2019 at 16:30, Andres Freund wrote: > Hi, > > On 2019-06-04 15:47:04 -0400, Dave Cramer wrote: > > On Mon, 3 Jun 2019 at 20:54, David Fetter wrote: > > > > > On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote: > > > > Is there a reason why pgoutput sends

Re: Binary support for pgoutput plugin

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 15:47:04 -0400, Dave Cramer wrote: > On Mon, 3 Jun 2019 at 20:54, David Fetter wrote: > > > On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote: > > > Is there a reason why pgoutput sends data in text format? Seems to > > > me that sending data in binary would provide

Re: No mention of no CIC support for partitioned index in docs

2019-06-04 Thread Alvaro Herrera
On 2019-May-24, David Rowley wrote: > It appears there is no mention of lack of support for CREATE INDEX > CONCURRENTLY on partitioned index in the documents. I'll leave this one for you to handle, thanks. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development,

Re: coverage additions

2019-06-04 Thread Alvaro Herrera
On 2019-Jun-04, Michael Paquier wrote: > On Sat, Jun 01, 2019 at 12:55:47AM -0400, Alvaro Herrera wrote: > > Ah, now I remember that I tried this before, but it requires some extra > > packages installed in the machine I think, and those create running > > services. Did you note that

Re: Binary support for pgoutput plugin

2019-06-04 Thread Dave Cramer
Dave Cramer On Mon, 3 Jun 2019 at 20:54, David Fetter wrote: > On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote: > > Is there a reason why pgoutput sends data in text format? Seems to > > me that sending data in binary would provide a considerable > > performance improvement. > >

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-04 Thread Robert Haas
On Tue, Jun 4, 2019 at 3:09 PM Melanie Plageman wrote: > One question I have is, how would the OR'd together bitmap be > propagated to workers after the first chunk? That is, when there are > no tuples left in the outer bunch, for a given inner chunk, would you > load the bitmaps from each worker

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-04 Thread Melanie Plageman
On Tue, Jun 4, 2019 at 6:05 AM Robert Haas wrote: > On Sun, May 19, 2019 at 7:07 PM Thomas Munro > wrote: > > Unfortunately that bits-in-order scheme doesn't work for parallel > > hash, where the SharedTuplestore tuples seen by each worker are > > non-deterministic. So perhaps in that case we

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-04 Thread Robert Haas
On Tue, Jun 4, 2019 at 2:47 PM Melanie Plageman wrote: > Oops! You are totally right. > I will amend the idea: > For each chunk on the inner side, loop through both the original batch > file and the unmatched outer tuples file created for the last chunk. > Emit any matches and write out any

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-04 Thread Melanie Plageman
On Tue, Jun 4, 2019 at 5:43 AM Robert Haas wrote: > On Mon, Jun 3, 2019 at 5:10 PM Melanie Plageman > wrote: > > I was talking to Jeff Davis about this on Saturday, and, he felt that > > there might be a way to solve the problem differently if we thought of > > the left join case as performing

Re: Sort support for macaddr8

2019-06-04 Thread Melanie Plageman
On Mon, Jun 3, 2019 at 2:39 PM Peter Geoghegan wrote: > > As you know, it's a bit weird that we're proposing adding sort support > with abbreviated keys for a type that is 8 bytes, since you'd expect > it to also be pass-by-value on most platforms, which largely defeats > the purpose of having

Re: Confusing error message for REINDEX TABLE CONCURRENTLY

2019-06-04 Thread Ashwin Agrawal
On Mon, Jun 3, 2019 at 6:27 PM Michael Paquier wrote: > On Mon, Jun 03, 2019 at 04:53:48PM -0700, Ashwin Agrawal wrote: > > Please check if the attached patch addresses and satisfies all the points > > discussed so far in this thread. > > It looks to be so, please see below for some comments. >

Re: Sort support for macaddr8

2019-06-04 Thread Stephen Frost
Greetings, * Melanie Plageman (melanieplage...@gmail.com) wrote: > Peter and I implemented this small (attached) patch to extend > abbreviated key compare sort to macaddr8 datatype (currently supported > for macaddr). Nice. > I think that that seems like an improvement. I was thinking of >

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 17:20:42 +0100, Andrew Gierth wrote: > fwiw on FreeBSD with no /etc/localtime and no TZ in the environment (and > hence running on UTC), I get "UCT" on both 11.3 and HEAD. That makes sense. As far as I can tell the reason that 12 sometimes ends up with the proper timezone is

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 08:53:30 -0700, Andres Freund wrote: > If I set the system-wide default, using dpkg-reconfigure -plow tzdata, > to UTC I *do* get Etc/UTC. > > root@alap4:/home/andres/src/postgresql# cat /etc/timezone > Etc/UTC > root@alap4:/home/andres/src/postgresql# ls -l /etc/timezone >

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Andrew Gierth
> "Christoph" == Christoph Berg writes: Christoph> There is something wrong here. On Debian Buster/unstable, Christoph> using system tzdata (2019a-1), if /etc/timezone is Christoph> "Etc/UTC": Christoph> 11.3's initdb adds timezone = 'UCT' to postgresql.conf Christoph> 12beta1's initdb

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 11:27:31 -0400, Tom Lane wrote: > $ TZ=UTC initdb > ... > selecting default timezone ... UTC > ... Btw, if the input is Etc/UTZ, do you also get UTC or Etc/UTZ? Because it seems that debian only configures Etc/UTZ on a system-wide basis now. Which seems not insane, given that's

Re: Minimal logical decoding on standbys

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 15:51:01 +0530, Amit Khandekar wrote: > After giving more thought on this, I think it might make sense to > arrange for the xl_running_xact record to be sent from master to the > standby, when a logical slot is to be created on standby. How about > standby sending a new message

Re: Minimal logical decoding on standbys

2019-06-04 Thread Andres Freund
Hi, On 2019-05-31 17:31:34 +0530, Amit Khandekar wrote: > On Fri, 31 May 2019 at 11:08, Amit Khandekar wrote: > > > > On Thu, 30 May 2019 at 20:13, Andres Freund wrote: > > Yes, it waits until there is a commit record, or (just tried) until a > > checkpoint command. That's fine with me. > Is

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Andrew Gierth
> "Tom" == Tom Lane writes: > Andrew Gierth writes: >> I believe I pointed out a long, long time ago that this tie-breaking >> strategy was insane, and that the rule should be to prefer canonical >> names and use something else only in the case of a strictly better >> match. Tom>

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Andres Freund
Hi, On 2019-06-04 11:27:31 -0400, Tom Lane wrote: > Hm, I don't have a Debian machine at hand, but I'm unable to > reproduce this using macOS or RHEL. I tried things like > > $ TZ=UTC initdb > ... > selecting default timezone ... UTC > ... On debian unstable that's what I get too, both with

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Tom Lane
Andrew Gierth writes: > I believe I pointed out a long, long time ago that this tie-breaking > strategy was insane, and that the rule should be to prefer canonical > names and use something else only in the case of a strictly better > match. This is assuming that the tzdb data has a concept of a

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Tom Lane
Christoph Berg writes: > There is something wrong here. On Debian Buster/unstable, using > system tzdata (2019a-1), if /etc/timezone is "Etc/UTC": > 11.3's initdb adds timezone = 'UCT' to postgresql.conf > 12beta1's initdb add timezone = 'Etc/UCT' to postgresql.conf Hm, I don't have a Debian

Re: Custom table AMs need to include heapam.h because of BulkInsertState

2019-06-04 Thread Robert Haas
On Sat, Jun 1, 2019 at 3:09 PM Michael Paquier wrote: > I am fine to live with the dependency with vacuum.h as it is not that > strange. However for BulkInsertState we get a hard dependency with a > heap-related area and it seems to me that we had better move that part > out of heapam.c, as we

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-04 Thread Robert Haas
On Sun, May 19, 2019 at 7:07 PM Thomas Munro wrote: > Unfortunately that bits-in-order scheme doesn't work for parallel > hash, where the SharedTuplestore tuples seen by each worker are > non-deterministic. So perhaps in that case we could use the > HEAP_TUPLE_HAS_MATCH bit in the outer tuple

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-04 Thread Robert Haas
On Mon, Jun 3, 2019 at 5:10 PM Melanie Plageman wrote: > I was talking to Jeff Davis about this on Saturday, and, he felt that > there might be a way to solve the problem differently if we thought of > the left join case as performing an inner join and an antijoin > instead. > > Riffing on this

Re: hyrax vs. RelationBuildPartitionDesc

2019-06-04 Thread Robert Haas
On Fri, May 17, 2019 at 4:36 PM Tom Lane wrote: > Yeah, I did some additional testing that showed that it's pretty darn > hard to get the leak to amount to anything. The test case that I > originally posted did many DDLs in a single transaction, and it > seems that that's actually essential to

Re: UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Andrew Gierth
> "Christoph" == Christoph Berg writes: >> Etc/UCT is now a backward-compatibility link to Etc/UTC, instead of >> being a separate zone that generates the abbreviation "UCT", which >> nowadays is typically a typo. Postgres will still accept "UCT" as an >> input zone name, but it won't

Re: Minimal logical decoding on standbys

2019-06-04 Thread Amit Khandekar
On Fri, 31 May 2019 at 17:31, Amit Khandekar wrote: > > On Fri, 31 May 2019 at 11:08, Amit Khandekar wrote: > > > > On Thu, 30 May 2019 at 20:13, Andres Freund wrote: > > > > > > Hi, > > > > > > On 2019-05-30 19:46:26 +0530, Amit Khandekar wrote: > > > > @@ -1042,7 +1042,8 @@

UCT (Re: pgsql: Update time zone data files to tzdata release 2019a.)

2019-06-04 Thread Christoph Berg
Re: Tom Lane 2019-04-26 > Update time zone data files to tzdata release 2019a. > > DST law changes in Palestine and Metlakatla. > Historical corrections for Israel. > > Etc/UCT is now a backward-compatibility link to Etc/UTC, instead > of being a separate zone that generates the abbreviation

Re: Pinned files at Windows

2019-06-04 Thread Konstantin Knizhnik
On 04.06.2019 3:18, Michael Paquier wrote: On Mon, Jun 03, 2019 at 11:37:30PM +0300, Konstantin Knizhnik wrote: but without success because ERROR_DELETE_PENDING is never returned by Win32. And moreover, stat() doesn't ever return error in this case. Could it be possible to find a reliable

Re: Add command column to pg_stat_progress_create_index

2019-06-04 Thread Peter Eisentraut
On 2019-05-27 20:18, Peter Eisentraut wrote: > I propose to add a column "command" to pg_stat_progress_create_index. > The sibling view pg_stat_progress_cluster already contains such a > column. This can help distinguish which command is running and thus > which phases to expect. It seems

[PATCH] Improve performance of NOTIFY over many databases (issue blocking on AccessExclusiveLock on object 0 of class 1262 of database 0)

2019-06-04 Thread Martijn van Oosterhout
Hoi hackers, We've been having issues with NOTIFYs blocking over multiple databases (see [1] for more details). That was 9.4 but we've updated the database to 11.3 and still have the same issue. Now however we could use perf to do profiling and got the following profile (useless details elided):