Re: reducing isolation tests runtime

2019-02-12 Thread Andres Freund
Hi, On 2018-01-25 17:34:15 -0300, Alvaro Herrera wrote: > Tom Lane wrote: > > Alvaro Herrera writes: > > > > I think we could solve this by putting in the same parallel group only > > > slow tests that mostly sleeps, ie. nothing that would monopolize CPU for > > > long enough to cause a

Re: Planning counters in pg_stat_statements (using pgss_store)

2019-02-12 Thread Sergei Kornilov
Hi > I think it is related to https://commitfest.postgresql.org/16/1373/ > but I don't know how to link it with. You can just add new entry in open commitfest and then attach previous thread. This is usual way for pick up old patches. For example, as i did here:

Re: pg_basebackup ignores the existing data directory permissions

2019-02-12 Thread Haribabu Kommi
On Wed, Feb 13, 2019 at 12:42 PM Michael Paquier wrote: > On Tue, Feb 12, 2019 at 06:03:37PM +1100, Haribabu Kommi wrote: > > During the testing allow group access permissions on the standby database > > directory, one of my colleague found the issue, that pg_basebackup > > doesn't verify

Re: obsolete comment above index_pages_fetched

2019-02-12 Thread Amit Langote
On 2019/02/13 16:33, Michael Paquier wrote: > On Wed, Feb 13, 2019 at 01:57:09PM +0900, Amit Langote wrote: >> total_table_pages is computed by make_one_rel as of the aforementioned >> commit. Attached fixes this. > > Thanks, fixed. Thank you. Regards, Amit

Re: obsolete comment above index_pages_fetched

2019-02-12 Thread Michael Paquier
On Wed, Feb 13, 2019 at 01:57:09PM +0900, Amit Langote wrote: > total_table_pages is computed by make_one_rel as of the aforementioned > commit. Attached fixes this. Thanks, fixed. -- Michael signature.asc Description: PGP signature

Re: pg_dump multi VALUES INSERT

2019-02-12 Thread Surafel Temesgen
On Mon, Feb 11, 2019 at 10:20 AM David Rowley wrote: > Reviewing pg_dump-rows-per-insert-option-v14. > Also, maybe one for Fabien (because he seems keen on keeping the > --rows-per-insert validation code) > > strtol() returns a long. dump_inserts is an int, so on machines where > sizeof(long) ==

Re: Protect syscache from bloating with negative cache entries

2019-02-12 Thread Kyotaro HORIGUCHI
At Tue, 12 Feb 2019 20:36:28 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20190212.203628.118792892.horiguchi.kyot...@lab.ntt.co.jp> > > (4) > > + hash_size = cp->cc_nbuckets * sizeof(dlist_head); > > + tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len; > > +

RE: Cache relation sizes?

2019-02-12 Thread Jamison, Kirk
On February 6, 2019, 8:57 AM +, Andres Freund wrote: > Maybe I'm missing something here, but why is it actually necessary to > have the sizes in shared memory, if we're just talking about caching > sizes? It's pretty darn cheap to determine the filesize of a file that > has been recently

Re: Refactoring the checkpointer's fsync request queue

2019-02-12 Thread Thomas Munro
On Wed, Feb 13, 2019 at 3:58 PM Shawn Debnath wrote: > On Wed, Jan 30, 2019 at 09:59:38PM -0800, Shawn Debnath wrote: > > I wonder if it might be better to introduce two different functions > > catering to the two different use cases for forcing an immediate sync: > > > > - sync a relation > >

Better error messages when lacking connection slots for autovacuum workers and bgworkers

2019-02-12 Thread Michael Paquier
Hi all, When lacking connection slots, the backend returns a simple "sorry, too many clients", which is something that has been tweaked by recent commit ea92368 for WAL senders. However, the same message would show up for autovacuum workers and bgworkers. Based on the way autovacuum workers are

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-02-12 Thread Tatsuro Yamada
On 2019/02/13 4:16, Alvaro Herrera wrote: I added metrics for the validate_index phases; patch attached. This is still a bit raw, but it looks much better now. Here's a sample concurrent index build on a 100M tuple table. There are no concurrent Great! + s.pid AS pid, S.datid

obsolete comment above index_pages_fetched

2019-02-12 Thread Amit Langote
Hi, I think the following commit: commit c6e4133fae1fde93769197379ffcc2b379845113 Author: Tom Lane Date: Wed Nov 7 12:12:56 2018 -0500 Postpone calculating total_table_pages until after pruning/exclusion. ... obsoleted a sentence in the comment above index_pages_fetched(), which says:

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-02-12 Thread Amit Langote
Hi Alvaro, On 2019/02/12 12:18, Alvaro Herrera wrote: > I ended up defining phases for the index_build phase in the AM itself; > the code reports a phase number using the regular API, and then the > pgstat_progress view obtains the name of each phase using a support > method. diff --git

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-02-12 Thread Amit Langote
On 2019/02/13 11:59, Simon Riggs wrote: > On Wed, 13 Feb 2019 at 00:46, Alvaro Herrera > wrote: > >> Here's a sample >> concurrent index build on a 100M tuple table. > > > Cool +1 Looking at the "([phase] x of x)" in the sample output, I thought pg_stat_progress_vacuum's output should've had

Re: Fast path for empty relids in check_outerjoin_delay()

2019-02-12 Thread Richard Guo
Yes sure. Thanks Tom. On Fri, Feb 1, 2019 at 1:04 AM Tom Lane wrote: > Since we've reached the end of the January commitfest, and it's pretty > clear that this patch isn't going to get committed in anything like > its current form, I'm going to close the CF entry as returned with > feedback.

Re: 2019-02-14 Press Release Draft

2019-02-12 Thread Jonathan S. Katz
On 2/13/19 4:13 AM, Chapman Flack wrote: > On 02/12/19 22:00, Jonathan S. Katz wrote: >> Attached is a draft of the press release for the upcoming 2019-02-14 >> release. Feedback & suggestions are welcome. > > > Users on PostgreSQL 9.4 should plan to upgrade to a supported version of >

Re: 2019-02-14 Press Release Draft

2019-02-12 Thread Chapman Flack
On 02/12/19 22:00, Jonathan S. Katz wrote: > Attached is a draft of the press release for the upcoming 2019-02-14 > release. Feedback & suggestions are welcome. Users on PostgreSQL 9.4 should plan to upgrade to a supported version of PostgreSQL as the community will stop releasing fixes for

Re: Should we still have old release notes in docs?

2019-02-12 Thread Jonathan S. Katz
On 2/12/19 5:45 PM, Jonathan S. Katz wrote: > >> On Feb 12, 2019, at 5:12 PM, Tom Lane wrote: >> >> The part about having a unified release-note archive somewhere else is >> still WIP. The ball is in the web team's court on that, I think. > > Yes - we are going to try one thing with the

2019-02-14 Press Release Draft

2019-02-12 Thread Jonathan S. Katz
Hi, Attached is a draft of the press release for the upcoming 2019-02-14 release. Feedback & suggestions are welcome. Thanks! Jonathan 2019-02-14 Cumulative Update Release The PostgreSQL Global Development Group has released an update to all supported

Re: monitoring CREATE INDEX [CONCURRENTLY]

2019-02-12 Thread Simon Riggs
On Wed, 13 Feb 2019 at 00:46, Alvaro Herrera wrote: > Here's a sample > concurrent index build on a 100M tuple table. Cool > There are no concurrent > transactions, so phases that wait for lockers don't appear. Would prefer to see them, so we know they are zero. -- Simon Riggs

RE: Protect syscache from bloating with negative cache entries

2019-02-12 Thread Tsunakawa, Takayuki
From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > > I didin't consider planning that happen within a function. If > > 5min is the default for catalog_cache_prune_min_age, 10% of it > > (30s) seems enough and gettieofday() with such intervals wouldn't > > affect forground jobs. I'd choose

RE: Reaping Temp tables to avoid XID wraparound

2019-02-12 Thread Tsunakawa, Takayuki
From: Andrew Gierth [mailto:and...@tao11.riddles.org.uk] > Tsunakawa> SELECT pg_stat_get_backend_pid(backendid); > > Doesn't work - that function's idea of "backend id" doesn't match the > real one, since it's looking at a local copy of the stats from which > unused slots have been removed.

Re: Reaping Temp tables to avoid XID wraparound

2019-02-12 Thread Michael Paquier
On Wed, Feb 13, 2019 at 12:38:51AM +, Andrew Gierth wrote: > Doesn't work - that function's idea of "backend id" doesn't match the > real one, since it's looking at a local copy of the stats from which > unused slots have been removed. The temporary namespace OID is added to PGPROC since v11,

Re: Statement-level rollback

2019-02-12 Thread Michael Paquier
On Tue, Feb 12, 2019 at 07:13:45PM -0300, Alvaro Herrera wrote: > It was reasonable to close this patch as returned-with-feedback in > January commitfest, but what you did here was first move it to the March > commitfest and then close it as returned-with-feedback in the March > commitfest, which

Re: dsa_allocate() faliure

2019-02-12 Thread Thomas Munro
On Mon, Feb 11, 2019 at 10:33 AM Tom Lane wrote: > Thomas Munro writes: > > So I'll wait until after the release, and we'll have > > to live with the bug for another 3 months. > > Check. Please hold off committing until you see the release tags > appear, probably late Tuesday my time /

Re: Reaping Temp tables to avoid XID wraparound

2019-02-12 Thread Andrew Gierth
> "Tsunakawa" == Tsunakawa, Takayuki > writes: >> From the temp table namespace I can get the backend ID using a regex >> - but I have no idea how I can map that to a PID - any thoughts? Tsunakawa> SELECT pg_stat_get_backend_pid(backendid); Doesn't work - that function's idea of

RE: Reaping Temp tables to avoid XID wraparound

2019-02-12 Thread Tsunakawa, Takayuki
From: James Sewell [mailto:james.sew...@jirotech.com] > From the temp table namespace I can get the backend ID using a regex - but > I have no idea how I can map that to a PID - any thoughts? > SELECT pg_stat_get_backend_pid(backendid);

RE: Protect syscache from bloating with negative cache entries

2019-02-12 Thread Tsunakawa, Takayuki
From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > I'm at a loss how call syscache for users. I think it is "catalog > cache". The most basic component is called catcache, which is > covered by the syscache layer, both of then are not revealed to > users, and it is shown to user as

Reaping Temp tables to avoid XID wraparound

2019-02-12 Thread James Sewell
Hi all, I hit an issue yesterday where I was quickly nearing XID wraparound on a database due to a temp table being created in a session which was then left IDLE out of transaction for 6 days. I solved the issue by tracing the owner of the temp table back to a session and terminating it - in my

Re: Statement-level rollback

2019-02-12 Thread Alvaro Herrera
On 2019-Feb-04, Michael Paquier wrote: > On Thu, Jan 31, 2019 at 04:38:27AM -0800, Andres Freund wrote: > > Is this still in development? Or should we mark this as returned with > > feedback? > > Marked as returned with feedback, as it has already been two months. > If you have an updated patch

Re: Too rigorous assert in reorderbuffer.c

2019-02-12 Thread Alvaro Herrera
On 2019-Feb-12, Arseny Sher wrote: > > Alvaro Herrera writes: > > >> I thought the blanket removal of the assert() was excessive, and we can > >> relax it instead; what do you think of the attached? > > > > More precisely, my question was: with this change, does the code still > > work

Re: Planning counters in pg_stat_statements (using pgss_store)

2019-02-12 Thread legrand legrand
Hi Sergei, Thank you for this review ! >Did you register patch in CF app? I did not found entry. I think it is related to https://commitfest.postgresql.org/16/1373/ but I don't know how to link it with. Yes, there are many things to improve, but before to go deeper, I would like to know if

Re: [PATCH v20] GSSAPI encryption support

2019-02-12 Thread Robbie Harwood
Stephen Frost writes: > * Robbie Harwood (rharw...@redhat.com) wrote: >> Stephen Frost writes: >>> * Robbie Harwood (rharw...@redhat.com) wrote: Stephen Frost writes: > * Robbie Harwood (rharw...@redhat.com) wrote: > >> Attached please find version 20 of the GSSAPI encryption

Re: [PATCH v20] GSSAPI encryption support

2019-02-12 Thread Stephen Frost
Greetings, * Robbie Harwood (rharw...@redhat.com) wrote: > Stephen Frost writes: > > * Robbie Harwood (rharw...@redhat.com) wrote: > >> Stephen Frost writes: > >>> * Robbie Harwood (rharw...@redhat.com) wrote: > >>> > Attached please find version 20 of the GSSAPI encryption support. >

Re: [PATCH v20] GSSAPI encryption support

2019-02-12 Thread Robbie Harwood
Stephen Frost writes: > * Robbie Harwood (rharw...@redhat.com) wrote: >> Stephen Frost writes: >>> * Robbie Harwood (rharw...@redhat.com) wrote: >>> Attached please find version 20 of the GSSAPI encryption support. This has been rebased onto master (thanks Stephen for calling out

Re: [PATCH] xlogreader: do not read a file block twice

2019-02-12 Thread Andrey Lepikhov
On 11.02.2019 21:25, Arthur Zakirov wrote: Hello hackers, Grigory noticed that one of our utilities has very slow performance when xlogreader reads zlib archives. We found out that xlogreader sometimes reads a WAL file block twice. zlib has slow performance when you read an archive not

Re: Protect syscache from bloating with negative cache entries

2019-02-12 Thread Tomas Vondra
On 2/12/19 12:35 PM, Kyotaro HORIGUCHI wrote: > Thank you for testing and the commits, Tomas. > > At Sat, 9 Feb 2019 19:09:59 +0100, Tomas Vondra > wrote in > <74386116-0bc5-84f2-e614-0cff19aca...@2ndquadrant.com> >> On 2/7/19 1:18 PM, Kyotaro HORIGUCHI wrote: >>> At Thu, 07 Feb 2019

Re: Should we still have old release notes in docs?

2019-02-12 Thread Thomas Kellerer
Tom Lane schrieb am 12.02.2019 um 17:12: Yeah, see 527b5ed1a et al. The part about having a unified release-note archive somewhere else is still WIP. The ball is in the web team's court on that, I think. The Bucardo team has already done that: https://bucardo.org/postgres_all_versions.html

Re: Should we still have old release notes in docs?

2019-02-12 Thread Jonathan S. Katz
> On Feb 12, 2019, at 5:12 PM, Tom Lane wrote: > > The part about having a unified release-note archive somewhere else is > still WIP. The ball is in the web team's court on that, I think. Yes - we are going to try one thing with the existing way we load docs to try to avoid additional

Re: Should we still have old release notes in docs?

2019-02-12 Thread Dave Page
> On 12 Feb 2019, at 16:00, Greg Stark wrote: > > I was just perusing our PDF docs for the first time in ages and > realized that of the 3,400+ pages of docs there's about 1,000 pages of > release notes in it That seems like a bit overkill. > > I love having the old release notes online

Re: BUG #15548: Unaccent does not remove combining diacritical characters

2019-02-12 Thread Hugh Ranalli
On Tue, 12 Feb 2019 at 08:54, Ramanarayana wrote: > Hi Michael, > The issue was that the python script was working in python 2 but not in > python 3 in Windows. This is because the python script writes the final > output to stdout and stdout encoding is set to utf-8 only for python 2 but > not

Re: use Getopt::Long for catalog scripts

2019-02-12 Thread John Naylor
On 2/12/19, Alvaro Herrera wrote: > Pushed, thanks. Thank you. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Should we still have old release notes in docs?

2019-02-12 Thread Tom Lane
Andres Freund writes: > On 2019-02-12 11:00:51 -0500, Greg Stark wrote: >> I love having the old release notes online but perhaps they can be >> somewhere other than the main docs? We could limit the current docs to >> including the release notes for just the supported versions -- after >> all

Re: Too rigorous assert in reorderbuffer.c

2019-02-12 Thread Arseny Sher
Alvaro Herrera writes: >> I thought the blanket removal of the assert() was excessive, and we can >> relax it instead; what do you think of the attached? > > More precisely, my question was: with this change, does the code still > work correctly in your non-toy case? Yes, it works. I thought

Re: Should we still have old release notes in docs?

2019-02-12 Thread Andres Freund
Hi, On 2019-02-12 11:00:51 -0500, Greg Stark wrote: > I was just perusing our PDF docs for the first time in ages and > realized that of the 3,400+ pages of docs there's about 1,000 pages of > release notes in it That seems like a bit overkill. > > I love having the old release notes online

Should we still have old release notes in docs?

2019-02-12 Thread Greg Stark
I was just perusing our PDF docs for the first time in ages and realized that of the 3,400+ pages of docs there's about 1,000 pages of release notes in it That seems like a bit overkill. I love having the old release notes online but perhaps they can be somewhere other than the main docs? We

Re: use Getopt::Long for catalog scripts

2019-02-12 Thread Alvaro Herrera
On 2019-Feb-12, Alvaro Herrera wrote: > On 2019-Feb-08, John Naylor wrote: > > > On Thu, Feb 7, 2019 at 6:53 PM David Fetter wrote: > > > [some style suggestions] > > > > I've included these in v2, and also some additional tweaks for consistency. > > I noticed that because we have this line

Re: use Getopt::Long for catalog scripts

2019-02-12 Thread Alvaro Herrera
On 2019-Feb-08, John Naylor wrote: > On Thu, Feb 7, 2019 at 6:53 PM David Fetter wrote: > > [some style suggestions] > > I've included these in v2, and also some additional tweaks for consistency. I noticed that because we have this line in the scripts, BEGIN { use lib

Re: [PROPOSAL]a new data type 'bytea' for ECPG

2019-02-12 Thread Michael Meskes
Matsumura-san, > I try to explain as follows. I would like to receive your comment. > ... I'm afraid I don't really understand your explanation. Why is handling a bytea so different from handling a varchar? I can see some differences due to its binary nature, but I do not understand why it needs

Re: libpq compression

2019-02-12 Thread Andreas Karlsson
On 2/11/19 5:28 PM, Peter Eisentraut wrote: One mitigation is to not write code like that, that is, don't put secret parameters and user-supplied content into the same to-be-compressed chunk, or at least don't let the end user run that code at will in a tight loop. The difference in CRIME is

Re: BUG #15548: Unaccent does not remove combining diacritical characters

2019-02-12 Thread Ramanarayana
Hi Michael, The issue was that the python script was working in python 2 but not in python 3 in Windows. This is because the python script writes the final output to stdout and stdout encoding is set to utf-8 only for python 2 but not python 3.If no encoding is set for stdout it takes the encoding

Re: Too rigorous assert in reorderbuffer.c

2019-02-12 Thread Alvaro Herrera
On 2019-Feb-11, Alvaro Herrera wrote: > On 2019-Feb-07, Arseny Sher wrote: > > > Alvaro Herrera writes: > > > > Ah, okay. Does the test still fail when run without the code fix? > > > > Yes. The problem here is overriding cmax of catalog (pg_attribute in the > > test) tuples, so it fails

Re: Planning counters in pg_stat_statements (using pgss_store)

2019-02-12 Thread Sergei Kornilov
Hello Thank you for picking this up! Did you register patch in CF app? I did not found entry. Currently we have pg_stat_statements 1.7 version and this patch does not apply... My fast and small view: > - errmsg("could not read file \"%s\": %m", > +

Re: Connection slots reserved for replication

2019-02-12 Thread Oleksii Kliukin
> On 12. Feb 2019, at 05:12, Michael Paquier wrote: > > On Mon, Feb 11, 2019 at 08:57:41PM -0700, Kevin Hale Boyes wrote: >> I think there's a small problem with the commit. >> The position of xlrec.max_wal_senders (line 117) should be below >> max_worker_processes. > > Fixed, thanks!

Re: Logical replication and restore from pg_basebackup

2019-02-12 Thread Alexey Kondratov
Hi Dmitry, On 11.02.2019 17:39, Dmitry Vasiliev wrote: What is the scope of logical replication if I cannot make recovery from pg_basebackup? No, you can, but there are some things to keep in mind: 1) I could be wrong, but usage of pgbench in such a test seems to be a bad idea, since it

Re: Problems with plan estimates in postgres_fdw

2019-02-12 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/08 21:35), Etsuro Fujita wrote: > > (2019/02/08 2:04), Antonin Houska wrote: > >> * add_foreign_ordered_paths() > >> > >> Exprssion root->upper_targets[UPPERREL_FINAL] is used to access the > >> target. > >> > >> I think create_ordered_paths() should actually set

Re: Protect syscache from bloating with negative cache entries

2019-02-12 Thread Kyotaro HORIGUCHI
At Tue, 12 Feb 2019 01:02:39 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FB972A6@G01JPEXMBYT05> > From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > > Recuded frequency of dlist_move_tail by taking 1ms interval between two > > succesive updates on the

Re: Protect syscache from bloating with negative cache entries

2019-02-12 Thread Kyotaro HORIGUCHI
Thank you for testing and the commits, Tomas. At Sat, 9 Feb 2019 19:09:59 +0100, Tomas Vondra wrote in <74386116-0bc5-84f2-e614-0cff19aca...@2ndquadrant.com> > On 2/7/19 1:18 PM, Kyotaro HORIGUCHI wrote: > > At Thu, 07 Feb 2019 15:24:18 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > > wrote

RE: [PROPOSAL]a new data type 'bytea' for ECPG

2019-02-12 Thread Matsumura, Ryo
Meskes-san Thank you for your review. > There is one thing that I don't understand right now. YOu > change ecpg_store_input() to handle the bytea data type, yet you also > change ECPGset_desc() to not use ecpg_store_input() in case of an > bytea. This looks odd to me. Can you please explain that

Re: pg11.1: dsa_area could not attach to segment

2019-02-12 Thread Sergei Kornilov
Hi > I think this is tentatively confirmed..I ran 20 loops for over 90 minutes with > no crash when parallel_leader_participation=off. > > On enabling parallel_leader_participation, crash within 10min. > > Sergei, could you confirm ? I still have error with parallel_leader_participation = off.

Re: Protect syscache from bloating with negative cache entries

2019-02-12 Thread Kyotaro HORIGUCHI
At Fri, 8 Feb 2019 09:42:20 +, "Ideriha, Takeshi" wrote in <4E72940DA2BF16479384A86D54D0988A6F41EDD1@G01JPEXMBKW04> > >From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > >I made a rerun of benchmark using "-S -T 30" on the server build with no > >assertion and > >-O2. The

Re: Problems with plan estimates in postgres_fdw

2019-02-12 Thread Antonin Houska
Etsuro Fujita wrote: > (2019/02/08 2:04), Antonin Houska wrote: > > > First, I wonder if the information on LIMIT needs to be passed to the > > FDW. Cannot the FDW routines use root->tuple_fraction? > > I think it's OK for the FDW to use the tuple_fraction, but I'm not sure the >

Re: [PATCH] xlogreader: do not read a file block twice

2019-02-12 Thread Arthur Zakirov
On 12.02.2019 07:23, Michael Paquier wrote: On 02/11/2019 07:25 PM, Arthur Zakirov wrote: Grigory noticed that one of our utilities has very slow performance when xlogreader reads zlib archives. We found out that xlogreader sometimes reads a WAL file block twice. What do you think? I think

Re: proposal: variadic argument support for least, greatest function

2019-02-12 Thread Pavel Stehule
Hi Ășt 12. 2. 2019 v 2:00 odesĂ­latel Chapman Flack napsal: > The following review has been posted through the commitfest application: > make installcheck-world: tested, failed > Implements feature: not tested > Spec compliant: not tested > Documentation:not tested >