Re: [PROPOSAL] Drop orphan temp tables in single-mode

2019-06-07 Thread Arthur Zakirov
Hello Alexander, On Friday, June 7, 2019, Alexander Korotkov wrote: > BTW, does this patch checks that temporary table is really orphan? > AFAICS, user may define some temporary tables in single-user mode > before running VACUUM. As far as I remember, the patch checks it. -- Arthur Zakirov

Bloom Indexes - bit array length and the total number of bits (or hash functions ?? ) !

2019-06-07 Thread Avinash Kumar
Hi All, I was testing bloom indexes today. I understand bloom indexes uses bloom filters. As i understand, a bloom filter is a bit array of m bits and a constant "k" number of hash functions are used to generate hashes for the data. And then the appropriate bits are set to 1. I was doing the

Re: Temp table handling after anti-wraparound shutdown (Was: BUG #15840)

2019-06-07 Thread Michael Paquier
On Fri, Jun 07, 2019 at 05:26:32PM -0700, Andres Freund wrote: > I was more thinking that we'd move the check for orphaned-ness into a > separate function (maybe IsOrphanedRelation()), and move the code to > drop orphan relations into a separate function (maybe > DropOrphanRelations()). That'd

Re: be-gssapi-common.h should be located in src/include/libpq/

2019-06-07 Thread Michael Paquier
On Fri, Jun 07, 2019 at 09:52:26AM +0200, Daniel Gustafsson wrote: > While looking at libpq.h I noticed what seems to be a few nitpicks: the GSS > function prototype isn’t using the common format of having a comment > specifying > the file it belongs to; ssl_loaded_verify_locations is defined as

Re: be-gssapi-common.h should be located in src/include/libpq/

2019-06-07 Thread Michael Paquier
On Fri, Jun 07, 2019 at 08:11:07AM -0400, Stephen Frost wrote: > I'm pretty sure it ended up there just because that's how things are in > src/interfaces/libpq. I don't have any objection to moving it, I had > really just been waiting to see where that thread ended up going. > > On a quick look,

Re: Binary support for pgoutput plugin

2019-06-07 Thread Andres Freund
Hi, On 2019-06-07 21:16:12 -0400, Chapman Flack wrote: > On 06/07/19 21:01, Andres Freund wrote: > > On 2019-06-07 20:52:38 -0400, Chapman Flack wrote: > > That'd be a *lot* of additional complexity, and pretty much prohibitive > > from a performance POV. We'd have to not continue decoding on the

Re: Binary support for pgoutput plugin

2019-06-07 Thread Chapman Flack
On 06/07/19 21:01, Andres Freund wrote: > On 2019-06-07 20:52:38 -0400, Chapman Flack wrote: >> It seems they had ended up designing a whole 'nother "protocol level" >> involving queries wrapping their results as JSON and an app layer that >> unwraps again, after trying a simpler first approach

Re: Binary support for pgoutput plugin

2019-06-07 Thread Andres Freund
Hi, On 2019-06-07 20:52:38 -0400, Chapman Flack wrote: > It seems they had ended up designing a whole 'nother "protocol level" > involving queries wrapping their results as JSON and an app layer that > unwraps again, after trying a simpler first approach that was foiled by the > inability to see

Re: Binary support for pgoutput plugin

2019-06-07 Thread Chapman Flack
On 06/07/19 19:27, Andres Freund wrote: > The problem is that I don't recognize a limiting principle: > > If we want NOT NULL information for clients, why don't we include the > underlying types for arrays, and the fields in composite types? What > about foreign keys? And unique keys? This

Re: Temp table handling after anti-wraparound shutdown (Was: BUG #15840)

2019-06-07 Thread Andres Freund
Hi, On 2019-06-08 08:59:37 +0900, Michael Paquier wrote: > On Fri, Jun 07, 2019 at 03:58:43PM -0700, Andres Freund wrote: > > Do we need to move the orphan temp cleanup code into database vacuums or > > such? > > When entering into the vacuum() code path for an autovacuum, only one > relation at

Re: Custom table AMs need to include heapam.h because of BulkInsertState

2019-06-07 Thread Michael Paquier
On Fri, Jun 07, 2019 at 08:55:36AM -0400, Robert Haas wrote: > Are you talking about the call to ReleaseBulkInsertStatePin, or something > else? Yes, I was referring to ReleaseBulkInsertStatePin() -- Michael signature.asc Description: PGP signature

Re: Temp table handling after anti-wraparound shutdown (Was: BUG #15840)

2019-06-07 Thread Michael Paquier
On Fri, Jun 07, 2019 at 03:58:43PM -0700, Andres Freund wrote: > Do we need to move the orphan temp cleanup code into database vacuums or > such? When entering into the vacuum() code path for an autovacuum, only one relation at a time is processed, and we have prior that extra processing related

Re: Binary support for pgoutput plugin

2019-06-07 Thread Andres Freund
Hi, On 2019-06-05 19:05:05 -0400, Dave Cramer wrote: > I am curious why you are "strongly" opposed however. We already have the > information. Adding doesn't seem onerous. (thought I'd already replied with this) The problem is that I don't recognize a limiting principle: If we want NOT NULL

Temp table handling after anti-wraparound shutdown (Was: BUG #15840)

2019-06-07 Thread Andres Freund
Hi, (Moving a part of this discussion to hackers) In #15840 Thierry had the situation that autovacuum apparently could not keep up, and he ended up with a wraparound situation. Following the hints and shutting down the cluster and vacuuming the relevant DB in single user mode did not resolve the

Re: tableam: abstracting relation sizing code

2019-06-07 Thread Daniel Gustafsson
> On 7 Jun 2019, at 14:43, Robert Haas wrote: > I think that's probably going in the wrong direction. It's arguable, > of course. However, it seems to me that there's nothing heap-specific > about the number 10. It's not computed based on the format of a heap > page or a heap tuple. It's

Re: heapam_index_build_range_scan's anyvisible

2019-06-07 Thread Alvaro Herrera
On 2019-Jun-07, Robert Haas wrote: > Yeah, I wondered whether SnapshotNonVacuumable might've been added > later, but I was too lazy to check the commit log. I'll try coding up > that approach and see how it looks. Thanks. > But do you have any comment on the question of whether this function >

Re: heapam_index_build_range_scan's anyvisible

2019-06-07 Thread Robert Haas
On Fri, Jun 7, 2019 at 4:30 PM Alvaro Herrera wrote: > On 2019-Jun-07, Robert Haas wrote: > > I spent some time today studying heapam_index_build_range_scan and > > quickly reached the conclusion that it's kind of a mess. At heart > > it's pretty simple: loop over all the table, check each tuple

Re: heapam_index_build_range_scan's anyvisible

2019-06-07 Thread Alvaro Herrera
On 2019-Jun-07, Robert Haas wrote: > I spent some time today studying heapam_index_build_range_scan and > quickly reached the conclusion that it's kind of a mess. At heart > it's pretty simple: loop over all the table, check each tuple against > any qual, and pass the visible ones to the

heapam_index_build_range_scan's anyvisible

2019-06-07 Thread Robert Haas
I spent some time today studying heapam_index_build_range_scan and quickly reached the conclusion that it's kind of a mess. At heart it's pretty simple: loop over all the table, check each tuple against any qual, and pass the visible ones to the callback. However, in an attempt to make it cater

Re: pg_dump: fail to restore partition table with serial type

2019-06-07 Thread Alvaro Herrera
On 2019-May-06, Alvaro Herrera wrote: > On 2019-May-06, Rushabh Lathia wrote: > > > Found another scenario where check constraint is not getting > > dump for the child table. > > You're right, the patched code is bogus; I'm reverting it all for > today's minors. Thanks for reporting. Here's

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2019-06-07 Thread Alexander Korotkov
On Fri, Dec 28, 2018 at 11:32 AM Masahiko Sawada wrote: > On Thu, Dec 27, 2018 at 10:24 PM Alvaro Herrera > wrote: > > > > On 2018-Dec-27, Alexey Kondratov wrote: > > > > > To summarize: > > > > > > 1) Alvaro and Michael agreed, that REINDEX with tablespace move may be > > > useful. This is done

Re: [PROPOSAL] Drop orphan temp tables in single-mode

2019-06-07 Thread Alexander Korotkov
On Fri, Mar 8, 2019 at 9:28 AM Michael Paquier wrote: > On Thu, Mar 07, 2019 at 10:49:29AM -0500, Robert Haas wrote: > > On Thu, Mar 7, 2019 at 10:24 AM Tom Lane wrote: > > > So if we think we can invent a "MAGICALLY FIX MY DATABASE" command, > > > let's do that. But please let's not turn a

Re: tableam: inconsistent parameter name

2019-06-07 Thread Robert Haas
On Fri, Jun 7, 2019 at 1:19 PM Andres Freund wrote: > On 2019-06-07 13:11:21 -0400, Robert Haas wrote: > > I found what appears to be another typo very nearby. Attached please > > find v2, fixing both issues. > > Hm, I thinks that's fixed already? Oops, you're right. -- Robert Haas

Re: tableam: inconsistent parameter name

2019-06-07 Thread Andres Freund
On 2019-06-07 13:11:21 -0400, Robert Haas wrote: > I found what appears to be another typo very nearby. Attached please > find v2, fixing both issues. Hm, I thinks that's fixed already?

Re: tableam: inconsistent parameter name

2019-06-07 Thread Robert Haas
On Fri, Jun 7, 2019 at 12:52 PM Andres Freund wrote: > On 2019-06-07 12:37:33 -0400, Robert Haas wrote: > > TableAmRoutine's index_build_range_scan thinks that parameter #8 is > > called end_blockno, but table_index_build_range_scan and > > heapam_index_build_range_scan and BRIN's summarize_range

Re: tableam: inconsistent parameter name

2019-06-07 Thread Andres Freund
Hi, On 2019-06-07 12:37:33 -0400, Robert Haas wrote: > TableAmRoutine's index_build_range_scan thinks that parameter #8 is > called end_blockno, but table_index_build_range_scan and > heapam_index_build_range_scan and BRIN's summarize_range all agree > that it's the number of blocks to be

Re: Custom table AMs need to include heapam.h because of BulkInsertState

2019-06-07 Thread Andres Freund
Hi, (David, see bottom if you're otherwise not interested). On 2019-06-07 09:48:29 -0400, Robert Haas wrote: > On Sat, Jun 1, 2019 at 3:20 PM Andres Freund wrote: > > > I'd like to think that the best way to deal with that and reduce the > > > confusion would be to move anything related to bulk

tableam: inconsistent parameter name

2019-06-07 Thread Robert Haas
TableAmRoutine's index_build_range_scan thinks that parameter #8 is called end_blockno, but table_index_build_range_scan and heapam_index_build_range_scan and BRIN's summarize_range all agree that it's the number of blocks to be scanned. I assume that this got changed at some point while Andres

Re: tableam: abstracting relation sizing code

2019-06-07 Thread Alvaro Herrera
On 2019-Jun-07, Robert Haas wrote: > On Fri, Jun 7, 2019 at 8:43 AM Robert Haas wrote: > > Good catch, and now I notice that the callback is not called > > estimate_rel_size but relation_estimate_size. Updated patch attached; > > thanks for the review. > > Let's try that one more time, and

Re: tableam: abstracting relation sizing code

2019-06-07 Thread Andres Freund
Hi, On 2019-06-07 08:32:37 -0400, Robert Haas wrote: > On Thu, Jun 6, 2019 at 10:08 PM Michael Paquier wrote: > > "allvisfrac", "pages" and "tuples" had better be documented about > > which result they represent. > > A lot of the table AM stuff (and the related slot stuff) lacks > function

Re: tableam: abstracting relation sizing code

2019-06-07 Thread Robert Haas
On Fri, Jun 7, 2019 at 8:43 AM Robert Haas wrote: > Good catch, and now I notice that the callback is not called > estimate_rel_size but relation_estimate_size. Updated patch attached; > thanks for the review. Let's try that one more time, and this time perhaps I'll make it compile. -- Robert

Re: doc: pg_trgm missing description for GUC "pg_trgm.strict_word_similarity_threshold"

2019-06-07 Thread Ian Barwick
On 6/7/19 9:00 PM, Michael Paquier wrote: On Fri, Jun 07, 2019 at 03:44:14PM +0900, Masahiko Sawada wrote: BTW while looking GUC variables defined in trgm_op.c the operators in each short description seems not correct; there is an extra percent sign. Should we also fix them? Both of you are

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-07 Thread Robert Haas
On Fri, Jun 7, 2019 at 10:17 AM Tomas Vondra wrote: > Yes, they could get quite big, and I think you're right we need to > keep that in mind, because it's on the outer (often quite large) side of > the join. And if we're aiming to restrict memory usage, it'd be weird to > just ignore this. > >

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-07 Thread Robert Haas
On Thu, Jun 6, 2019 at 7:31 PM Melanie Plageman wrote: > I'm not sure I understand why you would need to compare the original > tuples to the unmatched tuples file. I think I was confused. Actually, I'm still not sure I understand this part: > Then, you iterate again through the outer side a

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-07 Thread Tomas Vondra
On Thu, Jun 06, 2019 at 04:33:31PM -0700, Melanie Plageman wrote: On Tue, Jun 4, 2019 at 12:15 PM Robert Haas wrote: On Tue, Jun 4, 2019 at 3:09 PM Melanie Plageman wrote: > One question I have is, how would the OR'd together bitmap be > propagated to workers after the first chunk? That is,

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-07 Thread Tomas Vondra
On Thu, Jun 06, 2019 at 04:37:19PM -0700, Melanie Plageman wrote: On Thu, May 16, 2019 at 3:22 PM Thomas Munro wrote: Admittedly I don't have a patch, just a bunch of handwaving. One reason I haven't attempted to write it is because although I know how to do the non-parallel version using a

Re: Custom table AMs need to include heapam.h because of BulkInsertState

2019-06-07 Thread Robert Haas
On Sat, Jun 1, 2019 at 3:20 PM Andres Freund wrote: > > I'd like to think that the best way to deal with that and reduce the > > confusion would be to move anything related to bulk inserts into their > > own header/file, meaning the following set: > > - ReleaseBulkInsertStatePin > > -

Re: Custom table AMs need to include heapam.h because of BulkInsertState

2019-06-07 Thread Robert Haas
On Thu, Jun 6, 2019 at 10:29 PM Michael Paquier wrote: > One thing which is a bit tricky is that for example with COPY FROM we > have a code path which is able to release a buffer held by the bulk > insert state. Are you talking about the call to ReleaseBulkInsertStatePin, or something else? --

Re: hyrax vs. RelationBuildPartitionDesc

2019-06-07 Thread Robert Haas
On Thu, Jun 6, 2019 at 2:48 AM Amit Langote wrote: > Attached is a patch that applies on top of Robert's pdoldcxt-v1.patch, > which seems to fix this issue for me. Yeah, that looks right. I think my patch was full of fuzzy thinking and inadequate testing; thanks for checking it over and coming

Re: tableam: abstracting relation sizing code

2019-06-07 Thread Robert Haas
On Fri, Jun 7, 2019 at 4:11 AM Daniel Gustafsson wrote: > Makes sense. Regarding one of the hacks: > > +* HACK: if the relation has never yet been vacuumed, use a minimum > size > +* estimate of 10 pages. The idea here is to avoid assuming a > +* newly-created table is

Re: tableam: abstracting relation sizing code

2019-06-07 Thread Robert Haas
On Thu, Jun 6, 2019 at 10:08 PM Michael Paquier wrote: > Looks like a neat split. Thanks. > "allvisfrac", "pages" and "tuples" had better be documented about > which result they represent. A lot of the table AM stuff (and the related slot stuff) lacks function header comments; I don't like

Re: Fix inconsistencies for v12

2019-06-07 Thread Alexander Lakhin
Hello Amit, 07.06.2019 7:30, Amit Kapila wrote: > Leaving the changes related to the above two points, I have combined > all the changes and fixed the things as per my review in the attached > patch. Alexander, see if you can verify once the combined patch. I > am planning to commit the attached

Re: be-gssapi-common.h should be located in src/include/libpq/

2019-06-07 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > As mentioned on another thread about test coverage, I have noticed > that be-gssapi-common.h is not placed at the correct location, even > its its identication path at the top points to where the file should > be: >

Re: doc: pg_trgm missing description for GUC "pg_trgm.strict_word_similarity_threshold"

2019-06-07 Thread Michael Paquier
On Fri, Jun 07, 2019 at 03:44:14PM +0900, Masahiko Sawada wrote: > BTW while looking GUC variables defined in trgm_op.c the operators in > each short description seems not correct; there is an extra percent > sign. Should we also fix them? Both of you are right here, and the addition

Re: Small review comment on pg_checksums

2019-06-07 Thread Michael Paquier
On Fri, Jun 07, 2019 at 03:30:35PM +0900, Masahiko Sawada wrote: > Agreed. Please find an attached patch. Thanks, committed. -- Michael signature.asc Description: PGP signature

Re: PG 12 draft release notes

2019-06-07 Thread Adrien Nayrat
On 5/11/19 10:33 PM, Bruce Momjian wrote: > I have posted a draft copy of the PG 12 release notes here: > > http://momjian.us/pgsql_docs/release-12.html > > They are committed to git. It includes links to the main docs, where > appropriate. Our official developer docs will rebuild in a

Missing generated column in ALTER TABLE ADD COLUMN doc

2019-06-07 Thread Masahiko Sawada
Hi, We support ALTER TABLE ADD COLUMN .. GENERATED ALWAYS AS .. but the doc is missing it. Attached small patch fixes this. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center doc_alter_column_add_generated_column.patch Description: Binary

Re: tableam: abstracting relation sizing code

2019-06-07 Thread Daniel Gustafsson
> On 6 Jun 2019, at 22:40, Robert Haas wrote: > It looks to me as though any table AM that uses the relation forks > supported by PostgreSQL in a more or less normal manner is likely to > require an implementation of the relation_size callback that is > identical to the one for heap, and an

Wording variations in descriptions of gucs.

2019-06-07 Thread Kyotaro Horiguchi
Hello. In guc.c many of the variables are described as "Set_s_ something" as if the variable name is the omitted subject. A few seem being wrongly written as "Set something" with the same intention. Is it useful to unify them to the majority? wal_level > gettext_noop("Set the level of

Re: be-gssapi-common.h should be located in src/include/libpq/

2019-06-07 Thread Daniel Gustafsson
> On 7 Jun 2019, at 06:34, Michael Paquier wrote: > Any objections to something like the attached? No objections to moving the file per the patch. While looking at libpq.h I noticed what seems to be a few nitpicks: the GSS function prototype isn’t using the common format of having a comment

Re: Should we warn against using too many partitions?

2019-06-07 Thread Amit Langote
Hi, Thanks for the updated patches. On Fri, Jun 7, 2019 at 2:34 PM David Rowley wrote: > Anyway comments welcome. If I had a few more minutes to spare I'd > have wrapped OLTP in tags, but out of time for now. Some rewording suggestions. 1. +...Removal of unwanted data is also a

Re: doc: pg_trgm missing description for GUC "pg_trgm.strict_word_similarity_threshold"

2019-06-07 Thread Masahiko Sawada
On Thu, Jun 6, 2019 at 10:19 PM Ian Barwick wrote: > > Hi > > Commit be8a7a68662 added custom GUC > "pg_trgm.strict_word_similarity_threshold", > but omitted to document this in the section "GUC Parameters"; Indeed. BTW while looking GUC variables defined in trgm_op.c the operators in each

Re: Small review comment on pg_checksums

2019-06-07 Thread Masahiko Sawada
On Thu, Jun 6, 2019 at 10:21 PM Michael Paquier wrote: > > On Thu, Jun 06, 2019 at 05:16:30PM +0900, Masahiko Sawada wrote: > > So I'd like to propose a small fix for that; move the comment to the > > right place and add another comment. Please find an attached small > > patch. > > No objections

Re: [HACKERS] Block level parallel vacuum

2019-06-07 Thread Masahiko Sawada
On Wed, Apr 10, 2019 at 2:19 PM Masahiko Sawada wrote: > > On Mon, Apr 8, 2019 at 7:25 PM Kyotaro HORIGUCHI > wrote: > > > > Hello. > > > > # Is this still living? I changed the status to "needs review" > > > > At Sat, 6 Apr 2019 06:47:32 +0900, Masahiko Sawada > > wrote in > > > > > >