Re: \describe*

2019-06-21 Thread David Fetter
On Fri, Jun 21, 2019 at 05:49:43PM -0400, Alvaro Herrera wrote: > On 2019-Jun-21, David Fetter wrote: > > > On Fri, Jun 21, 2019 at 05:20:54PM -0400, Alvaro Herrera wrote: > > > On 2018-Jan-29, David Fetter wrote: > > > > > > > We could certainly have \d call DESCRIBE for later versions of the >

Re: Usage of epoch in txid_current

2019-06-21 Thread Alexander Korotkov
On Thu, Mar 28, 2019 at 8:30 AM Thomas Munro wrote: > Thanks for the reviews! Pushed. Any ideas we should move towards 64-bit xids in more places? That has been discussed several times already. I think last time it was discussed in person during FOSDEM PGDay 2018 Developer Meeting [1]. There

Re: POC: Cleaning up orphaned files using undo logs

2019-06-21 Thread Tom Lane
Robert Haas writes: > On Thu, Jun 20, 2019 at 4:54 AM Dilip Kumar wrote: >> The idea behind having the loop inside the undo machinery was that >> while traversing the blkprev chain, we can read all the undo records >> on the same undo page under one buffer lock. > That's not a bad goal,

Re: \describe*

2019-06-21 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Jan-29, David Fetter wrote: >> We could certainly have \d call DESCRIBE for later versions of the >> server. \ commands which call different SQL depending on server >> version have long been a standard practice. > So what is the uptake on implementing this at

Re: \describe*

2019-06-21 Thread Alvaro Herrera
On 2019-Jun-21, David Fetter wrote: > On Fri, Jun 21, 2019 at 05:20:54PM -0400, Alvaro Herrera wrote: > > On 2018-Jan-29, David Fetter wrote: > > > > > We could certainly have \d call DESCRIBE for later versions of the > > > server. \ commands which call different SQL depending on server > > >

Re: \describe*

2019-06-21 Thread David Fetter
On Fri, Jun 21, 2019 at 05:20:54PM -0400, Alvaro Herrera wrote: > On 2018-Jan-29, David Fetter wrote: > > > We could certainly have \d call DESCRIBE for later versions of the > > server. \ commands which call different SQL depending on server > > version have long been a standard practice. > >

Re: POC: Cleaning up orphaned files using undo logs

2019-06-21 Thread Robert Haas
On Thu, Jun 20, 2019 at 4:54 AM Dilip Kumar wrote: > The idea behind having the loop inside the undo machinery was that > while traversing the blkprev chain, we can read all the undo records > on the same undo page under one buffer lock. That's not a bad goal, although invoking a user-supplied

Re: \describe*

2019-06-21 Thread Alvaro Herrera
On 2018-Jan-29, David Fetter wrote: > We could certainly have \d call DESCRIBE for later versions of the > server. \ commands which call different SQL depending on server > version have long been a standard practice. So what is the uptake on implementing this at the server side, ie. DESCRIBE?

Re: allow_system_table_mods stuff

2019-06-21 Thread Andres Freund
Hi, On 2019-06-21 16:37:16 -0400, Tom Lane wrote: > We do have to get past the compatibility issue though. My thought was > that for a period of N years we could force allow_system_table_dml = on > while running extension scripts, and then cease doing so. This would > give extension authors a

Re: allow_system_table_mods stuff

2019-06-21 Thread Chapman Flack
On 6/21/19 4:37 PM, Tom Lane wrote: > We do have to get past the compatibility issue though. My thought was > that for a period of N years we could force allow_system_table_dml = on > while running extension scripts, and then cease doing so. This would > give extension authors a reasonable

Re: allow_system_table_mods stuff

2019-06-21 Thread Tom Lane
Chapman Flack writes: > I'd be leery of collateral damage from that to extension update scripts > in extension releases currently in the wild. Yeah, that's my primary concern here. > Maybe there should be a new extension control file setting > needs_system_table_mods = (boolean) > which means

Re: allow_system_table_mods stuff

2019-06-21 Thread Chapman Flack
On 6/21/19 3:07 PM, Stephen Frost wrote: > When it comes to cases that fundamentally are one-off's and that we > don't think really deserve a proper DDL command, then I'd say we make > the extensions set the flag. At least then it's clear "hey, we had to > do something really grotty here, maybe

Re: File descriptors inherited by restore_command

2019-06-21 Thread David Steele
On 6/21/19 10:26 AM, Stephen Frost wrote: >> >>> Another possible issue is that if we allow a child process to inherit >>> all these fds it might accidentally write to them, which would be bad. >>> I know the child process can go and maliciously open and trash files if >>> it wants, but it doesn't

Re: Problem with default partition pruning

2019-06-21 Thread Alvaro Herrera
On 2019-Jun-17, Shawn Wang wrote: > I tested different types of key values, and multi-level partitioned tables, > and found no problems. > Only the SQL in the file of src/test/regress/results/partition_prune.out has > a space that caused the regression test to fail. It's not clear to me what

Re: allow_system_table_mods stuff

2019-06-21 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > So there's certainly some fraction of these cases where we could have > avoided doing manual catalog updates by expending work on some ALTER > command instead. But I don't see much reason to think that we could, > or should try to, insist that

Re: allow_system_table_mods stuff

2019-06-21 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Keep in mind that DML-on-system-catalogs is unfortunately a really >> standard hack in extension upgrade scripts. (If memory serves, >> some of our contrib scripts do that, and we've certainly told third >> parties that it's the

progress report for ANALYZE

2019-06-21 Thread Alvaro Herrera
Hello Here's a patch that implements progress reporting for ANALYZE. Thanks -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 572fea555bc1c9d3512f7806aed6b97a374199eb Mon Sep 17 00:00:00 2001 From: Alvaro

Re: allow_system_table_mods stuff

2019-06-21 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2019-06-21 12:28:43 -0400, Tom Lane wrote: > > Robert Haas writes: > > > A related issue is that alter_system_table_mods prohibits both stuff > > > that's probably not going to cause any big problem and stuff that is > > > almost

Re: allow_system_table_mods stuff

2019-06-21 Thread Andres Freund
Hi, On 2019-06-21 12:28:43 -0400, Tom Lane wrote: > Robert Haas writes: > > I kinda feel like we should prohibit DML on system catalogs, even by > > superusers, unless you press the big red button that says "I am > > definitely sure that I know what I'm doing." > > Keep in mind that

Re: allow_system_table_mods stuff

2019-06-21 Thread Andres Freund
On 2019-06-21 11:12:38 +0200, Peter Eisentraut wrote: > After the earlier thread [0] that dealt with ALTER TABLE on system > catalogs, I took a closer look at the allow_system_table_mods setting. > I found a few oddities, and it seems there is some room for improvement. I complained about this

Re: SQL/JSON path issues/questions

2019-06-21 Thread Alexander Korotkov
On Wed, Jun 19, 2019 at 10:14 PM Alexander Korotkov wrote: > > While I have no objections to the proposed fixes, I think we can further > > improve patch 0003 and the text it refers to. > > In attempt to clarify jsonpath docs and address the concern that ? is > > hard to trace in the current

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-21 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> I haven't been paying too close attention to this thread, but isn't > >> that exactly what it does now and always has? guc.c, at least, certainly > >> is going to interpret

Re: allow_system_table_mods stuff

2019-06-21 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Robert Haas writes: > > I kinda feel like we should prohibit DML on system catalogs, even by > > superusers, unless you press the big red button that says "I am > > definitely sure that I know what I'm doing." > > Keep in mind that

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-21 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> I haven't been paying too close attention to this thread, but isn't >> that exactly what it does now and always has? guc.c, at least, certainly >> is going to interpret duplicate entries that way. > The issue isn't with reading

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-21 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Robert Haas writes: > > To me, forcing every tools author to use postgresql.conf parsing tools > > rather than just appending to the file is a needless burden on tool > > authors. I'd vote for just having ALTER SYSTEM silently drop all but > >

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-21 Thread Tom Lane
Robert Haas writes: > To me, forcing every tools author to use postgresql.conf parsing tools > rather than just appending to the file is a needless burden on tool > authors. I'd vote for just having ALTER SYSTEM silently drop all but > the last of duplicated entries. I haven't been paying too

Re: allow_system_table_mods stuff

2019-06-21 Thread Tom Lane
Robert Haas writes: > I kinda feel like we should prohibit DML on system catalogs, even by > superusers, unless you press the big red button that says "I am > definitely sure that I know what I'm doing." Keep in mind that DML-on-system-catalogs is unfortunately a really standard hack in

Re: Minimal logical decoding on standbys

2019-06-21 Thread Amit Khandekar
On Thu, 20 Jun 2019 at 00:31, Andres Freund wrote: > On 2019-06-12 17:30:02 +0530, Amit Khandekar wrote: > > In the attached v6 version of the patch, I did the above. That is, I > > used XLogFindNextRecord() to bump up the restart_lsn of the slot to > > the first valid record. But since

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-21 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Jun 17, 2019 at 10:50 AM Ian Barwick > wrote: > > In Pg12, the code in pg_basebackup implies the correct thing to do is > > append to .auto.conf, > > but as demonstrated that can cause problems with duplicate entries. > > Yeah.

Re: allow_system_table_mods stuff

2019-06-21 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Fri, Jun 21, 2019 at 5:12 AM Peter Eisentraut > wrote: > > Any other thoughts? > > I kinda feel like we should prohibit DML on system catalogs, even by > superusers, unless you press the big red button that says "I am > definitely

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-21 Thread Robert Haas
On Mon, Jun 17, 2019 at 10:50 AM Ian Barwick wrote: > In Pg12, the code in pg_basebackup implies the correct thing to do is append > to .auto.conf, > but as demonstrated that can cause problems with duplicate entries. Yeah. To me, forcing every tools author to use postgresql.conf parsing tools

Re: allow_system_table_mods stuff

2019-06-21 Thread Robert Haas
On Fri, Jun 21, 2019 at 5:12 AM Peter Eisentraut wrote: > Attached are some patches to get the discussion rolling: One patch makes > allow_system_table_mods settable at run time by superuser, the second > one is a test suite that documents the current behavior that I gathered > after analyzing

Re: File descriptors inherited by restore_command

2019-06-21 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > David Steele writes: > > On 6/21/19 9:45 AM, Tom Lane wrote: > >> +1 for using O_CLOEXEC on machines that have it. I don't think I want to > >> jump through hoops for machines that don't have it --- POSIX has required > >> it for some time, so

Re: File descriptors inherited by restore_command

2019-06-21 Thread Tom Lane
I wrote: > In practice, as long as we use O_CLOEXEC > for files opened by fd.c, that would eliminate the actual too-many-fds > hazard. I don't object to desultorily looking around for other places > where we might want to add it, but personally I'd be satisfied with a > patch that CLOEXEC-ifies

Re: File descriptors inherited by restore_command

2019-06-21 Thread Tom Lane
David Steele writes: > On 6/21/19 9:45 AM, Tom Lane wrote: >> +1 for using O_CLOEXEC on machines that have it. I don't think I want to >> jump through hoops for machines that don't have it --- POSIX has required >> it for some time, so there should be few machines in that category. > Another

Re: File descriptors inherited by restore_command

2019-06-21 Thread David Steele
On 6/21/19 9:45 AM, Tom Lane wrote: > David Steele writes: >> While investigating "Too many open files" errors reported in our >> parallel restore_command I noticed that the restore_command can inherit >> quite a lot of fds from the recovery process. This limits the number of >> fds available in

Google Season of Docs

2019-06-21 Thread Mahesh S
Hello, I am Mahesh S Nair from India. I am a GSoC 2018 student at KDE. I am very much interested in working with the organization for Google Season of Docs 2019. I think I am capable for, an open-source technical writing work as I have done GSoC and was a GCI mentor also. I am actually new to this

Re: using explicit_bzero

2019-06-21 Thread Dagfinn Ilmari Mannsåker
Tom Lane writes: > Peter Eisentraut writes: >> +#ifndef HAVE_EXPLICIT_BZERO >> +#define explicit_bzero(b, len) bzero(b, len) >> +#endif > > This presumes that every platform has bzero, which is unsafe (POSIX > doesn't specify it) and is an assumption we kicked to the curb a dozen > years ago

Re: File descriptors inherited by restore_command

2019-06-21 Thread Tom Lane
David Steele writes: > While investigating "Too many open files" errors reported in our > parallel restore_command I noticed that the restore_command can inherit > quite a lot of fds from the recovery process. This limits the number of > fds available in the restore_command depending on the

File descriptors inherited by restore_command

2019-06-21 Thread David Steele
Hackers, While investigating "Too many open files" errors reported in our parallel restore_command I noticed that the restore_command can inherit quite a lot of fds from the recovery process. This limits the number of fds available in the restore_command depending on the setting of system nofile

Re: using explicit_bzero

2019-06-21 Thread Tom Lane
Peter Eisentraut writes: > +#ifndef HAVE_EXPLICIT_BZERO > +#define explicit_bzero(b, len) bzero(b, len) > +#endif This presumes that every platform has bzero, which is unsafe (POSIX doesn't specify it) and is an assumption we kicked to the curb a dozen years ago (067a5cdb3). Please use memset()

Re: ldapbindpasswdfile

2019-06-21 Thread Stephen Frost
Greetings, * Thomas Munro (thomas.mu...@gmail.com) wrote: > I also know that a motivated user could also use GSSAPI instead of > LDAP. Do you think we should update the manual to say so, perhaps in > a "tip" box on the LDAP auth page? Hrm, not sure how I missed this before, but, yes, I'm all

Re: O(N^2) when building multi-column MCV lists

2019-06-21 Thread Tomas Vondra
Hi, Attached is a WIP/PoC fix addressing the O(N^2) behavior in ANALYZE with high statistic target values. It needs more work, but it's good enough to show some measurements. For benchmark, I've created a simple 2-column table, with MCV list on those two columns: CREATE TABLE t (a int, b

allow_system_table_mods stuff

2019-06-21 Thread Peter Eisentraut
After the earlier thread [0] that dealt with ALTER TABLE on system catalogs, I took a closer look at the allow_system_table_mods setting. I found a few oddities, and it seems there is some room for improvement. Attached are some patches to get the discussion rolling: One patch makes

Re: Choosing values for multivariate MCV lists

2019-06-21 Thread Dean Rasheed
On Thu, 20 Jun 2019 at 23:35, Tomas Vondra wrote: > > On Thu, Jun 20, 2019 at 06:55:41AM +0100, Dean Rasheed wrote: > > >I'm not sure it's easy to justify ordering by Abs(freq-base_freq)/freq > >though, because that would seem likely to put too much weight on the > >least commonly occurring

Re: Disconnect from SPI manager on error

2019-06-21 Thread RekGRpth
>It is not plpgsql's job to clean up after other backend subsystems during a transaction abort. But plpgsql do clean up on success! I suggest only do cleanup and on exception. чт, 20 июн. 2019 г. в 20:33, Tom Lane : > RekGRpth writes: > > A patch fixing this bug > > >

using explicit_bzero

2019-06-21 Thread Peter Eisentraut
In a recent thread[0], the existence of explicit_bzero() was mentioned. I went to look where we could use that to clear sensitive information from memory and found a few candidates: - In be-secure-common.c, clear the entered SSL passphrase in the error path. (In the non-error path, the buffer

Re: Multivariate MCV list vs. statistics target

2019-06-21 Thread Dean Rasheed
On Thu, 20 Jun 2019 at 23:12, Tomas Vondra wrote: > > On Thu, Jun 20, 2019 at 08:08:44AM +0100, Dean Rasheed wrote: > >On Tue, 18 Jun 2019 at 22:34, Tomas Vondra > >wrote: > >> > >> So I'm thinking we should allow tweaking the statistics for extended > >> stats, and serialize it in the