Re: logical decoding : exceeded maxAllocatedDescs for .spill files

2020-01-11 Thread Tom Lane
Tomas Vondra writes: > On Sat, Jan 11, 2020 at 10:53:57PM -0500, Tom Lane wrote: >> remind me where the win came from, exactly? > Well, the problem is that in 10 we allocate tuple data in the main > memory ReorderBuffer context, and when the transaction gets decoded we > pfree() it. But in AllocS

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

2020-01-11 Thread Tomas Vondra
On Sat, Jan 11, 2020 at 10:53:57PM -0500, Tom Lane wrote: Tomas Vondra writes: On Thu, Jan 09, 2020 at 07:40:12PM -0500, Tom Lane wrote: It seems reasonably likely to me that this result is telling us about an actual bug, ie, faulty back-patching of one or more of those fixes into v10 and perh

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

2020-01-11 Thread Tom Lane
Tomas Vondra writes: > On Thu, Jan 09, 2020 at 07:40:12PM -0500, Tom Lane wrote: >> It seems reasonably likely to me that this result is telling us about >> an actual bug, ie, faulty back-patching of one or more of those fixes >> into v10 and perhaps earlier branches. > Well, one thing we did in

Re: ECPG: proposal for new DECLARE STATEMENT

2020-01-11 Thread Tomas Vondra
On Sun, Jan 12, 2020 at 03:52:48AM +0100, Tomas Vondra wrote: ... I'm not an ecpg expert (in fact I've never even used it), so my review is pretty superficial, but I only found a couple of minor whitespace issues (adding/removing a line/tab) - see the attached file. Meh, forgot to attach the

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

2020-01-11 Thread Tomas Vondra
On Thu, Jan 09, 2020 at 07:40:12PM -0500, Tom Lane wrote: I wrote: ReorderBuffer: 223302560 total in 26995 blocks; 7056 free (3 chunks); 223295504 used The test case is only inserting 50K fairly-short rows, so this seems like an unreasonable amount of memory to be consuming for tha

Re: Why is pq_begintypsend so slow?

2020-01-11 Thread Tom Lane
I wrote: > I saw at this point that the remaining top spots were > enlargeStringInfo and appendBinaryStringInfo, so I experimented > with inlining them (again, see patch below). That *did* move > the needle: down to 72691 ms, or 20% better than HEAD. Oh ... marking the test in the inline part of

Re: [proposal] de-TOAST'ing using a iterator

2020-01-11 Thread John Naylor
On Mon, Sep 23, 2019 at 9:55 PM Binguo Bao wrote: > > Alvaro Herrera 于2019年9月17日周二 上午5:51写道: >> In broad terms this patch looks pretty good to me. I only have a small >> quibble with this API definition in fmgr.h -- namely that it forces us >> to export the definition of all the structs (that co

Re: ECPG: proposal for new DECLARE STATEMENT

2020-01-11 Thread Tomas Vondra
Hi, On Thu, Oct 31, 2019 at 12:29:30PM +, kuroda.hay...@fujitsu.com wrote: Dear hackers, As declared last month, I propose again the new ECPG grammar, DECLARE STATEMENT. This had been committed once, but it removed from PG12 because of some problems. In this mail, I want to report some prob

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

2020-01-11 Thread Amit Kapila
On Sat, Jan 11, 2020 at 11:16 AM Tom Lane wrote: > > Amit Kapila writes: > > On Fri, Jan 10, 2020 at 9:31 AM Amit Kapila wrote: > >> ... So, we have the below > >> options: > >> (a) remove this test entirely from all branches and once we found the > >> memory leak problem in back-branches, then

Re: Why is pq_begintypsend so slow?

2020-01-11 Thread Tom Lane
David Fetter writes: > On Sat, Jan 11, 2020 at 03:19:37PM -0500, Tom Lane wrote: >> Jeff Janes writes: >>> I saw that the hotspot was pq_begintypsend at 20%, which was twice the >>> percentage as the next place winner (AllocSetAlloc). >> I'd be inclined to replace the appendStringInfoCharMacro c

Re: [Proposal] Global temporary tables

2020-01-11 Thread Tomas Vondra
On Fri, Jan 10, 2020 at 11:47:42AM +0300, Konstantin Knizhnik wrote: On 09.01.2020 19:48, Tomas Vondra wrote: The most complex and challenged task is to support GTT for all kind of indexes. Unfortunately I can not proposed some good universal solution for it. Just patching all existed index

Re: Why is pq_begintypsend so slow?

2020-01-11 Thread David Fetter
On Sat, Jan 11, 2020 at 03:19:37PM -0500, Tom Lane wrote: > Jeff Janes writes: > > I saw that the hotspot was pq_begintypsend at 20%, which was twice the > > percentage as the next place winner (AllocSetAlloc). > > Weird. > > > Why is this such a bottleneck? > > Not sure, but it seems like a pr

Re: [Proposal] Global temporary tables

2020-01-11 Thread Tomas Vondra
On Fri, Jan 10, 2020 at 03:24:34PM +0300, Konstantin Knizhnik wrote: On 09.01.2020 19:30, Tomas Vondra wrote: 3 Still no one commented on GTT's transaction information processing, they include 3.1 Should gtt's frozenxid need to be care? 3.2 gtt’s clog clean 3.3 How to deal with "too o

Re: Windows UTF-8, non-ICU collation trouble

2020-01-11 Thread Noah Misch
On Wed, Dec 11, 2019 at 01:54:47PM +1300, Thomas Munro wrote: > On Tue, Dec 10, 2019 at 10:29 PM Noah Misch wrote: > > This does suggest some set of CompareString* parameters is free from the > > problem. If that's right, we could offer collations based on that. (I'm > > not > > sure it would b

Re: [PATCH] Atomic pgrename on Windows

2020-01-11 Thread Alexander Korotkov
On Tue, Jan 7, 2020 at 11:04 PM Tom Lane wrote: > Alexander Korotkov writes: > > I'm not sure issue we faced is really about single platform. TBH, the > > assumptions we place to rename function is very strict. We assume > > rename works atomically on system crash. And we indirectly assume it

Re: Improve checking for pg_index.xmin

2020-01-11 Thread Alexander Korotkov
On Wed, Jan 8, 2020 at 4:37 PM Tom Lane wrote: > Heikki Linnakangas writes: > > On 01/11/2019 01:50, Alexander Korotkov wrote: > >> This happens so, because we're checking that there is no broken HOT > >> chains after index creation by comparison pg_index.xmin and > >> TransactionXmin. So, we c

Re: 12.1 not useable: clientlib fails after a dozen queries (GSSAPI ?)

2020-01-11 Thread Tom Lane
I wrote: > (Still doesn't touch the static-buffer issue, though.) And here's a delta patch for that. This fixes an additional portability hazard, which is that there were various places doing stuff like input.length = ntohl(*(uint32 *) PqGSSRecvBuffer); That's a SIGBUS hazard on

Re: Avoid full GIN index scan when possible

2020-01-11 Thread Alexander Korotkov
Hi! Thank you for feedback! On Sat, Jan 11, 2020 at 3:19 PM Julien Rouhaud wrote: > On Sat, Jan 11, 2020 at 1:10 AM Alexander Korotkov > wrote: > > > > On Fri, Jan 10, 2020 at 7:36 PM Alexander Korotkov > > wrote: > > > > > > On Fri, Jan 10, 2020 at 6:31 PM Tom Lane wrote: > > >> > > >> Alexa

Re: 12.1 not useable: clientlib fails after a dozen queries (GSSAPI ?)

2020-01-11 Thread Tom Lane
I wrote: > So last night I was assuming that this problem just requires more careful > attention to what to return in the error exit paths. In the light of > morning, though, I realize that the algorithms involved in > be-secure-gssapi.c and fe-secure-gssapi.c are just fundamentally wrong: Here's

Re: [Proposal] Global temporary tables

2020-01-11 Thread Pavel Stehule
Hi so 11. 1. 2020 v 15:00 odesílatel 曾文旌(义从) napsal: > Hi all > > This is the latest patch > > The updates are as follows: > 1. Support global temp Inherit table global temp partition table > 2. Support serial column in GTT > 3. Provide views pg_gtt_relstats pg_gtt_stats for GTT’s statistics > 4

Re: Why is pq_begintypsend so slow?

2020-01-11 Thread Tom Lane
Jeff Janes writes: > I saw that the hotspot was pq_begintypsend at 20%, which was twice the > percentage as the next place winner (AllocSetAlloc). Weird. > Why is this such a bottleneck? Not sure, but it seems like a pretty dumb way to push the stringinfo's len forward. We're reading/updating

Why is pq_begintypsend so slow?

2020-01-11 Thread Jeff Janes
I was using COPY recently and was wondering why BINARY format is not much (if any) faster than the default format. Once I switched from mostly exporting ints to mostly exporting double precisions (7e6 rows of 100 columns, randomly generated), it was faster, but not by as much as I intuitively thou

Re: Make autovacuum sort tables in descending order of xid_age

2020-01-11 Thread David Fetter
On Thu, Jan 09, 2020 at 12:23:46PM -0500, Robert Haas wrote: > On Thu, Dec 12, 2019 at 2:26 PM David Fetter wrote: > > > I wonder if you might add information about table size, table changes, > > > and bloat to your RelFrozenXidAge struct and modify rfxa_comparator to > > > use a heuristic to cost

Re: pg_basebackup fails on databases with high OIDs

2020-01-11 Thread Magnus Hagander
On Sat, Jan 11, 2020 at 5:44 PM Julien Rouhaud wrote: > > On Sat, Jan 11, 2020 at 08:21:11AM +0100, Peter Eisentraut wrote: > > On 2020-01-06 21:00, Magnus Hagander wrote: > > > > +0.5 to avoid calling OidInputFunctionCall() > > > > > > Or just directly using atol() instead of atoi()? Well maybe n

Re: pg_basebackup fails on databases with high OIDs

2020-01-11 Thread Julien Rouhaud
On Sat, Jan 11, 2020 at 08:21:11AM +0100, Peter Eisentraut wrote: > On 2020-01-06 21:00, Magnus Hagander wrote: > > > +0.5 to avoid calling OidInputFunctionCall() > > > > Or just directly using atol() instead of atoi()? Well maybe not > > directly but in a small wrapper that verifies it's not bigge

Re: [HACKERS] Block level parallel vacuum

2020-01-11 Thread Mahendra Singh Thalor
On Sat, 11 Jan 2020 at 19:48, Masahiko Sawada < masahiko.saw...@2ndquadrant.com> wrote: > > On Sat, 11 Jan 2020 at 13:18, Amit Kapila wrote: > > > > On Sat, Jan 11, 2020 at 9:23 AM Masahiko Sawada > > wrote: > > > > > > On Fri, 10 Jan 2020 at 20:54, Mahendra Singh Thalor < mahi6...@gmail.com> wro

Re: Allow to_date() and to_timestamp() to accept localized names

2020-01-11 Thread Tomas Vondra
On Thu, Sep 26, 2019 at 08:36:25PM +0200, Juan José Santamaría Flecha wrote: On Wed, Sep 25, 2019 at 9:57 PM Alvaro Herrera wrote: This patch no longer applies. Can you please rebase? Thank you for the notification. The patch rot after commit 1a950f3, a rebased version is attached. Than

Re: 12.1 not useable: clientlib fails after a dozen queries (GSSAPI ?)

2020-01-11 Thread Tom Lane
I wrote: > Here's a draft patch that cleans up all the logic errors I could find. So last night I was assuming that this problem just requires more careful attention to what to return in the error exit paths. In the light of morning, though, I realize that the algorithms involved in be-secure-gss

Re: [HACKERS] Block level parallel vacuum

2020-01-11 Thread Masahiko Sawada
On Sat, 11 Jan 2020 at 13:18, Amit Kapila wrote: > > On Sat, Jan 11, 2020 at 9:23 AM Masahiko Sawada > wrote: > > > > On Fri, 10 Jan 2020 at 20:54, Mahendra Singh Thalor > > wrote: > > > > > > On Fri, 10 Jan 2020 at 15:51, Sergei Kornilov wrote: > > > > > > > > Hi > > > > Thank you for update!

Re: Efficient output for integer types

2020-01-11 Thread Tomas Vondra
Hi, Any plans regarding committing this patch? I see the thread is silent since September 24, when the last patch version was posted. The patch is already marked as RFC since December, when David changed the status. I don't have any opinion whether the patch is RFC or not (it might well be), but

Re: How to retain lesser paths at add_path()?

2020-01-11 Thread Tomas Vondra
On Sat, Jan 11, 2020 at 05:07:11PM +0900, Kohei KaiGai wrote: Hi, The proposition I posted at 10th-Oct proposed to have a separate list to retain lesser paths not to expand the path_list length, but here are no comments by others at that time. Indeed, the latest patch has not been updated yet. P

Re: Amcheck: do rightlink verification with lock coupling

2020-01-11 Thread Tomas Vondra
On Fri, Jan 10, 2020 at 06:49:33PM -0800, Peter Geoghegan wrote: On Fri, Jan 10, 2020 at 5:45 PM Tomas Vondra wrote: Peter, any opinion on this proposed amcheck patch? In the other thread [1] you seemed to agree this is worth checking, and Alvaro's proposal to make this check optional seems lik

Re: Avoid full GIN index scan when possible

2020-01-11 Thread Julien Rouhaud
Hi, On Sat, Jan 11, 2020 at 1:10 AM Alexander Korotkov wrote: > > On Fri, Jan 10, 2020 at 7:36 PM Alexander Korotkov > wrote: > > > > On Fri, Jan 10, 2020 at 6:31 PM Tom Lane wrote: > >> > >> Alexander Korotkov writes: > >> > So, I think v10 is a version of patch, which can be committed after

Re: [WIP] UNNEST(REFCURSOR): allowing SELECT to consume data from a REFCURSOR

2020-01-11 Thread Dent John
> On 10 Jan 2020, at 15:45, Daniel Verite wrote: > > At a quick glance, I don't see it called in the select-list in any > of the regression tests. […] Yep. I didn’t test it because I figured it wasn’t particularly useful in that context. I’ll add some tests for that too once I get to the root

Re: Add pg_file_sync() to adminpack

2020-01-11 Thread Artur Zakirov
Hello, On Sat, Jan 11, 2020 at 2:12 AM Fujii Masao wrote: > > + > > + pg_file_sync fsyncs the specified file or directory > > + named by filename. Returns true on success, > > + an error is thrown otherwise (e.g., the specified file is not present). > > + > > What's the point of having a fu

Re: base backup client as auxiliary backend process

2020-01-11 Thread Peter Eisentraut
On 2020-01-10 04:32, Masahiko Sawada wrote: I agreed that these patches are useful on its own and 0001 patch and committed 0001 0002 patch look good to me. For 0003 patch, + linkend="guc-primary-slot-name"/>. Otherwise, the WAL receiver may use + a temporary replication slot (dete

Re: How to retain lesser paths at add_path()?

2020-01-11 Thread Kohei KaiGai
Hi, The proposition I posted at 10th-Oct proposed to have a separate list to retain lesser paths not to expand the path_list length, but here are no comments by others at that time. Indeed, the latest patch has not been updated yet. Please wait for a few days. I'll refresh the patch again. Thanks