Re: Ought to use heap_multi_insert() for pg_attribute/depend insertions?

2020-06-28 Thread Michael Paquier
On Fri, Jun 26, 2020 at 02:26:50PM +0200, Daniel Gustafsson wrote: > On 26 Jun 2020, at 10:11, Michael Paquier wrote: >> + /* TODO is nreferenced a reasonable allocation of slots? */ >> + slot = palloc(sizeof(TupleTableSlot *) * nreferenced); >> It seems to me that we could just apply the same

Re: Creating a function for exposing memory usage of backend process

2020-06-28 Thread Bharath Rupireddy
> > If there is no such way to know the cache sizes and other info such as > > statistics, number of entries, cache misses, hits etc. can the > > approach discussed here be applied? > I think it's partially yes. > > > If the user knows the cache statistics and other information, may be > > we can

Re: Creating a function for exposing memory usage of backend process

2020-06-28 Thread Fujii Masao
On 2020/06/29 12:01, torikoshia wrote: On 2020-06-20 03:11, Robert Haas wrote: On Wed, Jun 17, 2020 at 11:56 PM Fujii Masao wrote: > As a first step, to deal with (3) I'd like to add > pg_stat_get_backend_memory_context() which target is limited to the > local backend process. +1 +1 from

Re: [PATCH] COPY command's data format option allows only lowercase csv, text or binary

2020-06-28 Thread Bharath Rupireddy
> As that shows, there's already a round of lowercasing done by the parser. > The only way that strcasecmp in copy.c would be useful is if you wanted to > accept things like > copy foo from stdin (format "CSV"); > I don't find that to be a terribly good idea. The normal implication > of qu

[PATCH] Performance Improvement For Copy From Binary Files

2020-06-28 Thread Bharath Rupireddy
Hi Hackers, For Copy From Binary files, there exists below information for each tuple/row. 1. field count(number of columns) 2. for every field, field size(column data length) 3. field data of field size(actual column data) Currently, all the above data required at each step is read directly from

Re: POC: postgres_fdw insert batching

2020-06-28 Thread Amit Langote
Hi Tomas, On Mon, Jun 29, 2020 at 12:10 AM Tomas Vondra wrote: > > Hi, > > One of the issues I'm fairly regularly reminded by users/customers is > that inserting into tables sharded using FDWs are rather slow. We do > even get it reported on pgsql-bugs from time to time [1]. > > Some of the slown

Re: Resetting spilled txn statistics in pg_stat_replication

2020-06-28 Thread Masahiko Sawada
On Fri, 26 Jun 2020 at 17:53, Amit Kapila wrote: > > On Fri, Jun 26, 2020 at 11:31 AM Masahiko Sawada > wrote: > > > > On Thu, 25 Jun 2020 at 19:35, Amit Kapila wrote: > > > > > > On Tue, Jun 23, 2020 at 6:39 PM Amit Kapila > > > wrote: > > > > > > > > On Tue, Jun 23, 2020 at 3:48 PM Tomas Von

[Bug fix]There is the case archive_timeout parameter is ignored after recovery works.

2020-06-28 Thread higuchi.dais...@fujitsu.com
Hi, I found the bug about archive_timeout parameter. There is the case archive_timeout parameter is ignored after recovery works. [Problem] When the value of archive_timeout is smaller than that of checkpoint_timeout and recovery works, archive_timeout is ignored in the first WAL archiving. Once

Re: Fwd: PostgreSQL: WolfSSL support

2020-06-28 Thread Felix Lechner
Hi Jonah, On Sat, Jun 27, 2020 at 12:35 PM Jonah H. Harris wrote: > > Somewhere, I recall seeing an open-source OpenSSL compatibility wrapper for > WolfSSL. Assuming that still exists, this patch seems entirely unnecessary. The patch uses the OpenSSL compatibility layer. Kind regards Felix Lec

Re: Fwd: PostgreSQL: WolfSSL support

2020-06-28 Thread Felix Lechner
Hi Tom, On Sat, Jun 27, 2020 at 11:52 AM Tom Lane wrote: > > The configure > script could add -I/usr/include/wolfssl (or wherever those files > are) to CPPFLAGS instead of touching all those #includes. That does not work well when OpenSSL's development files are installed. I did not think a segm

Re: Fwd: PostgreSQL: WolfSSL support

2020-06-28 Thread Felix Lechner
Hi Tom, On Sat, Jun 27, 2020 at 7:56 AM Tom Lane wrote: > > However, judging from the caveats mentioned in the initial message, > my inclination would be to wait awhile for wolfSSL to mature. Please have a closer look. The library has been around since 2004 and is popular in embedded systems. (I

Re: PostgreSQL: WolfSSL support

2020-06-28 Thread Felix Lechner
Hi Greg, On Sun, Jun 28, 2020 at 6:40 AM Greg Stark wrote: > > I'm more interested in a library like Amazon's Does S2N support TLS 1.3? https://github.com/awslabs/s2n/issues/388 Kind regards Felix Lechner

Re: pgsql: Enable Unix-domain sockets support on Windows

2020-06-28 Thread Amit Kapila
On Sun, Jun 28, 2020 at 2:03 PM Peter Eisentraut wrote: > > On 2020-06-27 13:57, Amit Kapila wrote: > > BTW, in which > > format the path needs to be specified for unix_socket_directories? I > > tried with '/c/tmp', 'c:/tmp', 'tmp' but nothing seems to be working, > > it gives me errors like: "co

estimation problems for DISTINCT ON with FDW

2020-06-28 Thread Jeff Janes
If I use the attached sql file to set up the database with loop-back postgres_fdw, and then turn on use_remote_estimate for this query: distinct on (id) id, z from fgn.priority order by id, priority desc,z It issues two queries for the foreign estimate, one with a sort and one without: EXPLAIN S

Re: Creating a function for exposing memory usage of backend process

2020-06-28 Thread torikoshia
On 2020-06-20 03:11, Robert Haas wrote: On Wed, Jun 17, 2020 at 11:56 PM Fujii Masao wrote: > As a first step, to deal with (3) I'd like to add > pg_stat_get_backend_memory_context() which target is limited to the > local backend process. +1 +1 from me, too. Attached a patch that adds a fu

Re: Default setting for enable_hashagg_disk

2020-06-28 Thread Peter Geoghegan
On Sat, Jun 27, 2020 at 3:41 AM Tomas Vondra wrote: > Well, there are multiple ideas discussed in this (sub)thread, one of > them being a per-query memory limit. That requires decisions how much > memory should different nodes get, which I think would need to be > cost-based. A design like that p

Re: ModifyTable overheads in generic plans

2020-06-28 Thread David Rowley
On Sat, 27 Jun 2020 at 00:36, Amit Langote wrote: > 2. ExecCheckRTPerms(): checks permissions of *all* partitions before > executing the plan tree, but maybe it's okay to check only the ones > that will be accessed I don't think it needs to be quite as complex as that. expand_single_inheritance_c

Re: Fwd: PostgreSQL: WolfSSL support

2020-06-28 Thread Michael Paquier
On Sun, Jun 28, 2020 at 10:18:12AM +0200, Peter Eisentraut wrote: > We have added support for allegedly-OpenSSL compatible libraries such as > LibreSSL before, so some tweaks for wolfSSL would seem acceptable. However, > I doubt we are going to backpatch them, so unless you want to take > responsib

Re: ModifyTable overheads in generic plans

2020-06-28 Thread Amit Langote
On Fri, Jun 26, 2020 at 9:36 PM Amit Langote wrote: > I would like to discuss a refactoring patch that builds on top of the > patches at [1] to address $subject. I've added this to the next CF: https://commitfest.postgresql.org/28/2621/ -- Amit Langote EnterpriseDB: http://www.enterprisedb.com

Re: Default setting for enable_hashagg_disk

2020-06-28 Thread Peter Geoghegan
On Sat, Jun 27, 2020 at 3:00 AM Amit Kapila wrote: > I think the advantage of delaying it is that we > might see some real problems (like where hash aggregate is not a good > choice) which can be fixed via the costing model. I think any problem that might come up with the costing is best thought

Re: Commitfest 2020-07

2020-06-28 Thread Michael Paquier
On Mon, Jun 29, 2020 at 08:04:35AM +0800, Andy Fan wrote: > Thanks for the volunteering! +1. -- Michael signature.asc Description: PGP signature

Re: Commitfest 2020-07

2020-06-28 Thread Andy Fan
On Mon, Jun 29, 2020 at 2:50 AM Magnus Hagander wrote: > On Sun, Jun 28, 2020 at 4:49 PM Tom Lane wrote: > >> Daniel Gustafsson writes: >> > In a few days, the first commitfest of the 14 cycle - 2020-07 - will >> start. >> > Unless anyone has already spoken up that I've missed, I'm happy to >>

Re: pg_read_file() with virtual files returns empty string

2020-06-28 Thread Tom Lane
Joe Conway writes: > All good stuff -- I believe the attached checks all the boxes. Looks okay to me, except I think you want ! if (bytes_to_read > 0) to be ! if (bytes_to_read >= 0) As it stands, a zero request will be treated like -1 (read all the rest of the file) while ISTM it

Revert workarounds for unportability of printf %.*s format?

2020-06-28 Thread Tom Lane
In connection with the discussion at [1], I realized that we could unwind the hacks we've introduced --- mostly in commit 54cd4f045 --- to avoid depending on the behavior of %.*s format in printf. Now that we always use our own snprintf.c code, we know that it measures field widths in bytes not ch

Re: Commitfest 2020-07

2020-06-28 Thread Magnus Hagander
On Sun, Jun 28, 2020 at 4:49 PM Tom Lane wrote: > Daniel Gustafsson writes: > > In a few days, the first commitfest of the 14 cycle - 2020-07 - will > start. > > Unless anyone has already spoken up that I've missed, I'm happy to > volunteer to > > run CFM for this one. > > No one has volunteered

Re: new heapcheck contrib module

2020-06-28 Thread Mark Dilger
> On Jun 28, 2020, at 9:05 AM, Dilip Kumar wrote: > > Some more comments on v9_0001. > 1. > + LWLockAcquire(XidGenLock, LW_SHARED); > + nextFullXid = ShmemVariableCache->nextFullXid; > + ctx.oldestValidXid = ShmemVariableCache->oldestXid; > + LWLockRelease(XidGenLock); > + ctx.nextKnownValidXi

Re: Fwd: PostgreSQL: WolfSSL support

2020-06-28 Thread Tom Lane
Felix Lechner writes: > On Sat, Jun 27, 2020 at 7:56 AM Tom Lane wrote: >> However, judging from the caveats mentioned in the initial message, >> my inclination would be to wait awhile for wolfSSL to mature. > Please have a closer look. The library has been around since 2004 and > is popular in

Re: bugfix: invalid bit/varbit input causes the log file to be unreadable

2020-06-28 Thread Tom Lane
I wrote: > Even granting the premise, the proposed patch seems like a significant > decrease in user-friendliness for typical cases. I'd rather see us > make an effort to print one valid-per-the-DB-encoding character. > Now that we can rely on snprintf to count %s restrictions in bytes, > I think

Re: pg_read_file() with virtual files returns empty string

2020-06-28 Thread Joe Conway
On 6/27/20 3:43 PM, Tom Lane wrote: > Joe Conway writes: >> The attached patch fixes this for me. I think it ought to be backpatched >> through >> pg11. > >> Comments? > > 1. Doesn't seem to be accounting for the possibility of an error in fread(). > > 2. Don't we want to remove the stat() cal

Re: new heapcheck contrib module

2020-06-28 Thread Dilip Kumar
On Sun, Jun 28, 2020 at 8:59 PM Dilip Kumar wrote: > > On Mon, Jun 22, 2020 at 5:44 AM Mark Dilger > wrote: > > > > > > > > > On Jun 21, 2020, at 2:54 AM, Dilip Kumar wrote: > > > > > > I have looked into 0001 patch and I have a few comments. > > > > > > 1. > > > + > > > + /* Skip over unused/de

Re: new heapcheck contrib module

2020-06-28 Thread Dilip Kumar
On Mon, Jun 22, 2020 at 5:44 AM Mark Dilger wrote: > > > > > On Jun 21, 2020, at 2:54 AM, Dilip Kumar wrote: > > > > I have looked into 0001 patch and I have a few comments. > > > > 1. > > + > > + /* Skip over unused/dead/redirected line pointers */ > > + if (!ItemIdIsUsed(ctx.itemid) || > > + It

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-06-28 Thread Dilip Kumar
On Fri, Jun 26, 2020 at 11:47 AM Amit Kapila wrote: > > On Thu, Jun 25, 2020 at 7:11 PM Dilip Kumar wrote: > > > > On Wed, Jun 24, 2020 at 4:04 PM Amit Kapila wrote: > > > > > > > > > Review comments on various patches. > > > > > > poc_shared_fileset_cleanup_on_procexit > > > ===

Re: tar-related code in PostgreSQL

2020-06-28 Thread Hamid Akhtar
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested The patch works perfectly. The new status of this patch is: Ready fo

POC: postgres_fdw insert batching

2020-06-28 Thread Tomas Vondra
Hi, One of the issues I'm fairly regularly reminded by users/customers is that inserting into tables sharded using FDWs are rather slow. We do even get it reported on pgsql-bugs from time to time [1]. Some of the slowness / overhead is expected, doe to the latency between machines in the sharded

Re: More tzdb fun: POSIXRULES is being deprecated upstream

2020-06-28 Thread Tom Lane
Seems like I'm not getting any traction in convincing people that back-patching this change is wise. To get this closed out before the CF starts, I'm just going to put it into HEAD/v13 and call it a day. I remain of the opinion that we'll probably regret not doing anything in the back branches, s

Re: Commitfest 2020-07

2020-06-28 Thread Tom Lane
Daniel Gustafsson writes: > In a few days, the first commitfest of the 14 cycle - 2020-07 - will start. > Unless anyone has already spoken up that I've missed, I'm happy to volunteer > to > run CFM for this one. No one has volunteered that I recall, so the baton is yours.

Re: Fwd: PostgreSQL: WolfSSL support

2020-06-28 Thread Bruce Momjian
On Sun, Jun 28, 2020 at 10:18:12AM +0200, Peter Eisentraut wrote: > On 2020-06-27 14:50, Christoph Berg wrote: > > Re: Peter Eisentraut > > > What would be the advantage of using wolfSSL over OpenSSL? > > > > Avoiding the OpenSSL-vs-GPL linkage problem with readline. > > We have added support for

Re: PostgreSQL: WolfSSL support

2020-06-28 Thread Greg Stark
Personally I'm more interested in a library like Amazon's which is trying to do less rather than more. I would rather a simpler, better tested, easier to audit code-base than one with more features and more complications. https://github.com/awslabs/s2n

Re: [HACKERS] Custom compression methods

2020-06-28 Thread Dilip Kumar
On Wed, Jun 24, 2020 at 5:30 PM Robert Haas wrote: > > On Tue, Jun 23, 2020 at 4:00 PM Andres Freund wrote: > > https://postgr.es/m/20130621000900.GA12425%40alap2.anarazel.de is a > > thread with more information / patches further along. > > > > I confused this patch with the approach in > > http

TLS checking in pgstat

2020-06-28 Thread Daniel Gustafsson
As I mentioned in [1], checking (struct Port)->ssl for NULL to determine whether TLS is used for connection is a bit of a leaky abstraction, as that's an OpenSSL specific struct member. This sets the requirement that all TLS implementations use a pointer named SSL, and that the pointer is set to N

Commitfest 2020-07

2020-06-28 Thread Daniel Gustafsson
In a few days, the first commitfest of the 14 cycle - 2020-07 - will start. Unless anyone has already spoken up that I've missed, I'm happy to volunteer to run CFM for this one. cheers ./daniel

Re: PostgreSQL: WolfSSL support

2020-06-28 Thread Daniel Gustafsson
> On 27 Jun 2020, at 21:40, Bruce Momjian wrote: > On Sat, Jun 27, 2020 at 04:22:51PM -0300, Ranier Vilela wrote: >> WolfSSL, will provide a commercial license for PostgreSQL? >> Isn't LIbreSSL a better alternative? > > Seems it might be. That's not really an apples/apples comparison as the pro

Re: pgsql: Enable Unix-domain sockets support on Windows

2020-06-28 Thread Peter Eisentraut
On 2020-06-27 13:57, Amit Kapila wrote: Fair enough, but what should be the behavior in the Windows versions (<10) where Unix-domain sockets are not supported? You get an error about an unsupported address family, similar to trying to use IPv6 on a system that doesn't support it. BTW, in wh

Re: Fwd: PostgreSQL: WolfSSL support

2020-06-28 Thread Peter Eisentraut
On 2020-06-27 14:50, Christoph Berg wrote: Re: Peter Eisentraut What would be the advantage of using wolfSSL over OpenSSL? Avoiding the OpenSSL-vs-GPL linkage problem with readline. We have added support for allegedly-OpenSSL compatible libraries such as LibreSSL before, so some tweaks for