RE: Recovery performance of standby for multiple concurrent truncates on large tables

2018-07-30 Thread Tsunakawa, Takayuki
From: 'Andres Freund' [mailto:and...@anarazel.de] > I'm continuing to work on it, but unfortunately there's a couple > projects that have higher priority atm :(. I'm doubtful I can have a > patchset in a committable shape for v12, but I'm pretty sure I'll have > it in a shape good enough to make p

RE: Recovery performance of standby for multiple concurrent truncates on large tables

2018-07-30 Thread Tsunakawa, Takayuki
From: Robert Haas [mailto:robertmh...@gmail.com] > It's not clear to me whether it would be worth the overhead of doing > something like this. Quite frankly, not really to me, too. > Making relation drops faster at the cost of > making buffer cleaning slower could be a loser. The purpose is not

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-07-30 Thread Kyotaro HORIGUCHI
At Mon, 30 Jul 2018 16:59:16 +0900, Michael Paquier wrote in <20180730075916.gb2...@paquier.xyz> > On Fri, Jul 27, 2018 at 08:27:26AM +, Tsunakawa, Takayuki wrote: > > I don't have a strong opinion, but I wonder which of namespace.c or > > autovacuum.c is suitable, because isTempNamespaceInUs

Re: Reduce amount of WAL generated by CREATE INDEX for gist, gin and sp-gist

2018-07-30 Thread Andrey Lepikhov
With the consent of Anastasia I will improving this patch further. Attachment contains next version of the patch set. 11.07.2018 00:03, Heikki Linnakangas пишет: On 28/02/18 18:03, Anastasia Lubennikova wrote: Implementation is based on generic_xlog. Why? I think we should just add a log_re

Re: Explain buffers wrong counter with parallel plans

2018-07-30 Thread Amit Kapila
On Tue, Jul 31, 2018 at 12:58 AM, Andres Freund wrote: > Hi, > > I'm not an expert in the area of the code, but here's a review anyway. I > did not read through the entire thread. > > > I think we should try to get this fixed soon, to make some progress > towards release-ability. Or just declare i

pg_default_acl missing 'n' case in doc

2018-07-30 Thread Fabien COELHO
While looking at ACL prettyprinting, I noticed that "pg_default_acl" documentation does not say anything about type 'n' for schema (namespace), which seems to be supported according to "\ddp" and the catalog code. Here is a small addition to add that 'n' is allowed for schema. -- Fabien.diff

Re: partition tree inspection functions

2018-07-30 Thread Amit Langote
Hi Jesper, On 2018/07/30 22:21, Jesper Pedersen wrote: > On 07/30/2018 05:21 AM, Amit Langote wrote: >>> As 0 is a valid return value for root nodes I think we should use -1 >>> instead for these cases. >> >> Makes sense, changed to be that way. >> > > Looks good, the documentation for pg_partiti

Re: Expression errors with "FOR UPDATE" and postgres_fdw with partition wise join enabled.

2018-07-30 Thread Etsuro Fujita
(2018/07/31 4:06), Andres Freund wrote: On 2018-07-20 08:38:09 -0400, Robert Haas wrote: I'm going to study this some more now, but I really think this is going in the wrong direction. We're going to have to get somewhere on this topic soon. This thread has been open for nearly half a year, an

Re: documentation about explicit locking

2018-07-30 Thread Amit Langote
On 2018/07/31 5:27, Peter Eisentraut wrote: > On 19/07/2018 06:17, Amit Langote wrote: >> When playing with this, I observed that a less user-friendly error message >> is emitted if multiple sessions race to create the same collation. >> >> Session 1: >> begin; >> create collation collname (...); >

Re: documentation about explicit locking

2018-07-30 Thread Amit Langote
On 2018/07/31 5:25, Peter Eisentraut wrote: > On 20/07/2018 02:30, Amit Langote wrote: >> We don't explicitly mention what locks we take on system catalogs >> elsewhere, but CREATE COLLATION is different from other commands, so I'm >> fine with adding more details as you suggest, so updated the tex

Re: [HACKERS] Parallel Append implementation

2018-07-30 Thread Thomas Munro
On Tue, Jul 31, 2018 at 5:05 AM, Robert Haas wrote: > New version attached. Looks good to me. -- Thomas Munro http://www.enterprisedb.com

Re: Making "COPY partitioned_table FROM" faster

2018-07-30 Thread David Rowley
On 31 July 2018 at 06:21, Peter Eisentraut wrote: > On 30/07/2018 15:26, David Rowley wrote: >>> - Add some tests. The if (nBufferedTuples > 0) that flushes the tuples >>> when the partition changes is not currently exercised. >> >> That seems like a good idea. In fact, it uncovered a bug around

Re: make installcheck-world in a clean environment

2018-07-30 Thread Tom Lane
I wrote: > The original complaint about ecpg remains; I'm not terribly excited > about messing with that. Well, I got curious as to why we were seeing such a weird error, and eventually traced it to this stuff in ecpg/test/Makefile.regress: # Standard way to invoke the ecpg preprocessor ECPG = ..

Re: Recovery performance of standby for multiple concurrent truncates on large tables

2018-07-30 Thread Thomas Munro
On Tue, Jul 31, 2018 at 8:01 AM, Robert Haas wrote: > On Mon, Jul 30, 2018 at 1:22 AM, Jamison, Kirk > wrote: >> 1. Because the multiple scans of the whole shared buffer per concurrent >> truncate/drop table was the cause of the time-consuming behavior, DURING the >> failover process while WAL

Re: FailedAssertion on partprune

2018-07-30 Thread Alvaro Herrera
On 2018-Jul-25, David Rowley wrote: > Thinking again about the patch I submitted upthread; I wonder if it's > actually possible to support pruning with Jamie's query. Without > looking at the code, I don't quite see the reason that the > sub-partitioned table wouldn't be correctly pruned by the ru

Re: Avoid extra Sort nodes between WindowAggs when sorting can be reused

2018-07-30 Thread Daniel Gustafsson
> On 27 Jul 2018, at 21:12, Alexander Kuzmenkov > wrote: > Thanks for the update. Thank you for reviewing and hacking! > On 07/25/2018 01:37 AM, Daniel Gustafsson wrote: >> >>> Hmm, this is missing the eqop fields of SortGroupClause. I haven't >>> tested yet but does the similar degradation h

Re: negative bitmapset member not allowed Error with partition pruning

2018-07-30 Thread Alvaro Herrera
On 2018-Jul-27, David Rowley wrote: > On 27 July 2018 at 15:14, Tom Lane wrote: > > Well, my thinking is that it helps nobody if call sites have to have > > explicit workarounds for a totally-arbitrary refusal to handle edge > > cases in the primitive functions. I do not think that is good soft

Re: make installcheck-world in a clean environment

2018-07-30 Thread Tom Lane
Alexander Lakhin writes: > 14.07.2018 13:57, Peter Eisentraut wrote: >> On 06.07.18 09:45, Alexander Lakhin wrote: >>> ./configure --enable-tap-tests >>> make install >>> make install -C contrib >>> chown -R postgres:postgres /usr/local/pgsql/ >>> /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/da

RE: GSOC 2018 Project - A New Sorting Routine

2018-07-30 Thread Kefan Yang
Hey Tomas! Sorry to bother but it would be great if we can get the test results this week. Regards, Kefan From: Tomas Vondra Sent: July 24, 2018 8:16 AM To: Kefan Yang Cc: Andrey Borodin; Peter Geoghegan; alvhe...@2ndquadrant.com; PostgreSQL Hackers Subject: Re: GSOC 2018 Project - A New Sorti

Re: Segfault logical replication PG 10.4

2018-07-30 Thread Mai Peng
Thank you Alvaro :) > Le 30 juil. 2018 à 22:33, Alvaro Herrera a écrit : > > On 2018-Jul-28, Alvaro Herrera wrote: > >> Aha, I see, thanks. Here's a complete fix with included testcase. In >> an unpatched assert-enabled build, this crashes this >> >> TRAP: FailedAssertion("!(ActiveSnapshotSe

Re: Making "COPY partitioned_table FROM" faster

2018-07-30 Thread Melanie Plageman
On Mon, Jul 30, 2018 at 11:21 AM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 30/07/2018 15:26, David Rowley wrote: > >> - Add some tests. The if (nBufferedTuples > 0) that flushes the tuples > >> when the partition changes is not currently exercised. > > > > That seems like

Re: Segfault logical replication PG 10.4

2018-07-30 Thread Alvaro Herrera
On 2018-Jul-28, Alvaro Herrera wrote: > Aha, I see, thanks. Here's a complete fix with included testcase. In > an unpatched assert-enabled build, this crashes this > > TRAP: FailedAssertion("!(ActiveSnapshotSet())", File: > "/pgsql/source/REL_10_STABLE/src/backend/tcop/postgres.c", Line: 788)

Re: documentation about explicit locking

2018-07-30 Thread Peter Eisentraut
On 19/07/2018 06:17, Amit Langote wrote: > When playing with this, I observed that a less user-friendly error message > is emitted if multiple sessions race to create the same collation. > > Session 1: > begin; > create collation collname (...); > > Session 2: > create collation collname (...); >

Re: documentation about explicit locking

2018-07-30 Thread Peter Eisentraut
On 20/07/2018 02:30, Amit Langote wrote: > We don't explicitly mention what locks we take on system catalogs > elsewhere, but CREATE COLLATION is different from other commands, so I'm > fine with adding more details as you suggest, so updated the text. committed the documentation change -- Peter

Re: Recovery performance of standby for multiple concurrent truncates on large tables

2018-07-30 Thread 'Andres Freund'
Hi, On 2018-07-30 05:22:48 +, Jamison, Kirk wrote: > BTW, are there any updates whether the community will push through > anytime soon regarding the buffer mapping implementation you > mentioned? I'm continuing to work on it, but unfortunately there's a couple projects that have higher priori

Re: Recovery performance of standby for multiple concurrent truncates on large tables

2018-07-30 Thread Andres Freund
Hi, On 2018-07-30 16:01:53 -0400, Robert Haas wrote: > (1) Limit the number of deferred drops to a reasonably small number > (one cache line? 1kB?). Yea, you'd have to, because we'd frequently need to check it, and it'd need to be in shared memory. But that'd still leave us to regress to O(n^2)

Re: Recovery performance of standby for multiple concurrent truncates on large tables

2018-07-30 Thread Robert Haas
On Mon, Jul 30, 2018 at 1:22 AM, Jamison, Kirk wrote: > 1. Because the multiple scans of the whole shared buffer per concurrent > truncate/drop table was the cause of the time-consuming behavior, DURING the > failover process while WAL is being applied, we temporary delay the scanning > and inv

Re: Explain buffers wrong counter with parallel plans

2018-07-30 Thread Andres Freund
Hi, I'm not an expert in the area of the code, but here's a review anyway. I did not read through the entire thread. I think we should try to get this fixed soon, to make some progress towards release-ability. Or just declare it to be entirely unrelated to the release, and remove it from the ope

Re: Documenting that queries can be run over replication protocol

2018-07-30 Thread Peter Eisentraut
On 24/07/2018 09:54, Chris Travers wrote: > In the process of building some tooling based on replication we > discovered that PostgreSQL 10 uses COPY to populate the initial table on > logical replication, see commit 7c4f52409a8c7d85ed169bbbc1f6092274d03920 > > This is not currently documented.  A

Re: [HACKERS] Getting rid of "accept incoming network connections" prompts on OS X

2018-07-30 Thread Peter Eisentraut
On 26/07/2018 23:45, Tom Lane wrote: > This came up again today, and I've confirmed that the issue still exists > in current macOS. Did you get any response to your bug report, and if > so what did they say? There hasn't been any response to the radar. I think our analysis is correct, it's an OS

Re: Expression errors with "FOR UPDATE" and postgres_fdw with partition wise join enabled.

2018-07-30 Thread Andres Freund
Hi, On 2018-07-20 08:38:09 -0400, Robert Haas wrote: > I'm going to study this some more now, but I really think this is > going in the wrong direction. We're going to have to get somewhere on this topic soon. This thread has been open for nearly half a year, and we're late in the beta phase now.

Re: Making "COPY partitioned_table FROM" faster

2018-07-30 Thread Peter Eisentraut
On 30/07/2018 15:26, David Rowley wrote: >> - Add some tests. The if (nBufferedTuples > 0) that flushes the tuples >> when the partition changes is not currently exercised. > > That seems like a good idea. In fact, it uncovered a bug around > ConvertPartitionTupleSlot() freeing the previously sto

Re: Adding a note to protocol.sgml regarding CopyData

2018-07-30 Thread Fabien COELHO
Hello Tatsuo-san, Minor suggestions, although I'm not a native English speaker. In libpq.sgml following is stated: Before PostgreSQL protocol 3.0, it was necessary for the application to explicitly send the two characters \. as a final line to indicate to the server tha

Re: [HACKERS] PoC: full merge join on comparison clause

2018-07-30 Thread Alexander Kuzmenkov
El 18/07/18 a las 16:58, Ashutosh Bapat escribió: Thanks for the commit messages. I would use word "in-equality" instead of "comparison" since equality is also a comparison. Fixed. Comparing this with the original code, I think, is_mj_equality should be true if restrictinfo->mergeopfamilies

Re: request for new parameter for disable promote (slave only mode)

2018-07-30 Thread Robert Haas
On Fri, Jul 27, 2018 at 12:05 PM, Ioseph Kim wrote: > I want to build one master & multi slave environments to use physical > replication. > Slave nodes have low hardware spec, so I changed max_connection server > parameters, and try start slave node. > But I could not start slave nodes, > becau

Re: [HACKERS] Parallel Append implementation

2018-07-30 Thread Robert Haas
On Sun, Jul 29, 2018 at 5:49 PM, Thomas Munro wrote: > On Thu, May 10, 2018 at 7:08 AM, Robert Haas wrote: >> [parallel-append-doc-v2.patch] > > +plans just as they can in any other plan. However, in a parallel plan, > +it is also possible that the planner may choose to substitute a > +

Re: Would like to help with documentation for Postgres 11

2018-07-30 Thread Michael Goldshteyn
Thanks for the replies, I'll investigate further.. On Sun, Jul 29, 2018 at 7:11 PM Tatsuo Ishii wrote: > > Justin Pryzby writes: > >> On Sun, Jul 29, 2018 at 11:50:40AM -0500, Michael Goldshteyn wrote: > >>> I would like to offer some help writing and improving the English > >>> documentation f

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

2018-07-30 Thread Bossart, Nathan
On 7/29/18, 7:35 PM, "Michael Paquier" wrote: > Yeah, I was testing that yesterday night and bumped on this case when > trying do a REINDEX SCHEMA pg_class. The point is that you can simplify > the check and remove pg_database_ownercheck as there is already an ACL > check on the database/system/s

Re: GiST VACUUM

2018-07-30 Thread Heikki Linnakangas
On 29/07/18 14:47, Andrey Borodin wrote: Fixed both problems. PFA v14. Thanks, took a really quick look at this. The text being added to README is outdated for these latest changes. In second step I still use paloc's memory, but only to store two bitmaps: bitmap of internal pages and bitmap

Re: Explain buffers wrong counter with parallel plans

2018-07-30 Thread Jonathan S. Katz
> On Jul 28, 2018, at 2:14 AM, Amit Kapila wrote: > > On Fri, Jul 27, 2018 at 11:12 PM, Jonathan S. Katz > wrote: >> >>> On Jul 27, 2018, at 8:31 AM, Amit Kapila wrote: >>> >>> >>> Yeah, that would be better. Today, I have tried the patch on both >>> Head and PG11 and I am getting same and

Re: Making "COPY partitioned_table FROM" faster

2018-07-30 Thread David Rowley
On 30 July 2018 at 20:33, Peter Eisentraut wrote: > Two more thoughts: > > - Add some tests. The if (nBufferedTuples > 0) that flushes the tuples > when the partition changes is not currently exercised. That seems like a good idea. In fact, it uncovered a bug around ConvertPartitionTupleSlot() f

Re: partition tree inspection functions

2018-07-30 Thread Jesper Pedersen
Hi Amit, On 07/30/2018 05:21 AM, Amit Langote wrote: As 0 is a valid return value for root nodes I think we should use -1 instead for these cases. Makes sense, changed to be that way. Looks good, the documentation for pg_partition_level could be expanded to describe the -1 scenario. New

Re: Usability fail with psql's \dp command

2018-07-30 Thread Fabien COELHO
Hello Kyotaro-san, Note that 'No privileges' could be somehow interpreted as "default privileges" (no "special/given" privileges) or as "no permissions at all", so there is still some ambiguity, at least for me. FWIW "No privileges" seems to me as "The user cannot access it at all" with no a

Re: Tips on committing

2018-07-30 Thread Peter Eisentraut
On 23/07/2018 05:58, Tom Lane wrote: > 013f320dc reminds me of something I check for religiously: look for > alternative output files for any regression test you're updating the > output of. > > Actually updating said files, once you notice you need to, can be tricky > in itself. Most of the time

Re: [PATCH] pg_hba.conf : new auth option : clientcert=verify-full

2018-07-30 Thread Julian Markwort
On 07/19/2018 03:00 AM, Thomas Munro wrote: Some more comments: if (parsedline->auth_method == uaCert) { - parsedline->clientcert = true; + parsedline->clientcert = clientCertOn; } The "cert" method is technically redundant with this patch,

Re: ssl_library parameter

2018-07-30 Thread Peter Eisentraut
On 26/06/2018 11:49, Daniel Gustafsson wrote: >> Extracted from the GnuTLS thread/patch, here is a patch to add a >> server-side read-only parameter ssl_library, which currently reports >> either 'OpenSSL' or an empty string, depending on what SSL library was >> built with. This is analogous to th

Re: Make deparsing of column defaults faster

2018-07-30 Thread Jeff Janes
On Mon, Jul 30, 2018 at 7:03 AM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 07/07/2018 20:07, Jeff Janes wrote: > > One case that your patch doesn't improve (neither does my posted one) is > > check constraints. To fix that, pg_get_constraintdef_worker would also > > need to

Re: Make deparsing of column defaults faster

2018-07-30 Thread Peter Eisentraut
On 07/07/2018 20:07, Jeff Janes wrote: > One case that your patch doesn't improve (neither does my posted one) is > check constraints.  To fix that, pg_get_constraintdef_worker would also > need to grow a cache as well.  I don't know how often people put check > constraints on most of the columns o

Re: [PATCH] Improve geometric types

2018-07-30 Thread Tomas Vondra
On 07/30/2018 11:57 AM, Emre Hasegeli wrote: OK, thanks for confirming. I'll get it committed and we'll see what the animals think soon. Thank you for fixing this. I wanted to preserve this code but wasn't sure about the correct place or whether it is still necessary. There are more places

Re: Fix for documentation of Covering Indexes

2018-07-30 Thread Liudmila Mantrova
On 04/18/2018 12:52 PM, Heikki Linnakangas wrote: On 11/04/18 04:20, Michael Paquier wrote: Hi all, The documentation of covering indexes is incorrect for CREATE and ALTER TABLE: - ALTER TABLE's page is missing the call. - Exclusion constraints can use INCLUDE clauses. In order to simplify the

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

2018-07-30 Thread Michael Paquier
On Mon, Jul 30, 2018 at 05:53:54PM +0900, Kyotaro HORIGUCHI wrote: > I feel that just being a database owner doesn't justify to cause > this problem innocently. Catalog owner is also doubious but we > can carefully configure the ownerships to avoid the problem since > only superuser can change it.

Re: [PATCH] Include application_name in "connection authorized" log message

2018-07-30 Thread Peter Eisentraut
On 13/07/2018 20:20, Don Seiler wrote: > See attached for latest revision. This doesn't compile with SSL enabled because there is a comma missing. This implementation doesn't run the application_name through check_application_name(), so it could end up logging application_name values that are oth

Re: [PATCH] Improve geometric types

2018-07-30 Thread Emre Hasegeli
> OK, thanks for confirming. I'll get it committed and we'll see what the > animals think soon. Thank you for fixing this. I wanted to preserve this code but wasn't sure about the correct place or whether it is still necessary. There are more places we produce -0. The regression tests have alte

Re: patch to allow disable of WAL recycling

2018-07-30 Thread Kyotaro HORIGUCHI
At Mon, 30 Jul 2018 10:43:20 +0200, Peter Eisentraut wrote in > On 19/07/2018 05:59, Kyotaro HORIGUCHI wrote: > > My result is that we cannot disable recycling perfectly just by > > setting min/max_wal_size. > > Maybe the behavior of min_wal_size should be rethought? Elsewhere in > this threa

Re: [PATCH] Improve geometric types

2018-07-30 Thread Emre Hasegeli
> This should fix it I guess, and it's how we deal with unused return > values elsewhere. I've considered using USE_ASSERT_CHECKING here, but it > seems rather ugly with that. I'll wait for Emre's opinion ... Assert() is the wrong thing to do in here. Drawn-perpendicular lines may not intersect b

Re: partition tree inspection functions

2018-07-30 Thread Amit Langote
Hi, On 2018/07/27 21:21, Jesper Pedersen wrote: > Hi Amit, > > On 07/26/2018 10:33 PM, Amit Langote wrote: >> Optional parameter sounds good, so made it get_partition_level(regclass [ >> , regclass ]) in the updated patch.  Although, adding that argument is not >> without possible surprises its r

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

2018-07-30 Thread Kyotaro HORIGUCHI
At Mon, 30 Jul 2018 09:34:22 +0900, Michael Paquier wrote in <20180730003422.ga2...@paquier.xyz> > On Sun, Jul 29, 2018 at 04:11:38PM +, Bossart, Nathan wrote: > > On 7/27/18, 7:10 PM, "Michael Paquier" wrote: > > This is added to ReindexMultipleTables(), which is used for REINDEX > > SCHEMA

Re: patch to allow disable of WAL recycling

2018-07-30 Thread Peter Eisentraut
On 19/07/2018 05:59, Kyotaro HORIGUCHI wrote: > My result is that we cannot disable recycling perfectly just by > setting min/max_wal_size. Maybe the behavior of min_wal_size should be rethought? Elsewhere in this thread, there was also a complaint that max_wal_size isn't actually a max. It seem

Re: Making "COPY partitioned_table FROM" faster

2018-07-30 Thread Amit Langote
On 2018/07/30 17:33, Peter Eisentraut wrote: > - With proute becoming a function-level variable, > cstate->partition_tuple_routing is obsolete and could be removed. (No > point in saving this in cstate if it's only used in one function anyway.) +1. Also seems to apply to transition_capture, whic

Re: Making "COPY partitioned_table FROM" faster

2018-07-30 Thread Peter Eisentraut
Two more thoughts: - Add some tests. The if (nBufferedTuples > 0) that flushes the tuples when the partition changes is not currently exercised. - With proute becoming a function-level variable, cstate->partition_tuple_routing is obsolete and could be removed. (No point in saving this in cstate

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-07-30 Thread Amit Langote
On 2018/07/28 10:54, David Rowley wrote: > On 27 July 2018 at 19:11, Amit Langote wrote: >> I've attached a delta patch to make the above changes. I'm leaving the >> hash table rename up to you though. > > Thanks for the delta patch. I took all of it, just rewrote a comment slightly. > > I also

Re: adding tab completions

2018-07-30 Thread Arthur Zakirov
On Sun, Jul 29, 2018 at 07:42:43PM -0500, Justin Pryzby wrote: > Your suggestion is good, so attached updated patch. The patch is in good shape. It compiles without errors. The patch doesn't need in documentation. I marked the patch as "Ready for Commiter". > > > Actually..another thought: since

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-07-30 Thread Michael Paquier
On Fri, Jul 27, 2018 at 08:27:26AM +, Tsunakawa, Takayuki wrote: > I don't have a strong opinion, but I wonder which of namespace.c or > autovacuum.c is suitable, because isTempNamespaceInUse is specific to > autovacuum. I think that there is also a point in allowing other backends to use it a