Augment every test postgresql.conf

2018-12-28 Thread Noah Misch
"env EXTRA_REGRESS_OPTS=--temp-config=SOMEFILE make check" appends the contents of SOMEFILE to the test cluster's postgresql.conf. I want a similar feature for TAP suites and other non-pg_regress suites. (My immediate use case is to raise authentication_timeout and wal_sender_timeout on my buildf

Re: add_partial_path() may remove dominated path but still in use

2018-12-28 Thread Kohei KaiGai
2018年12月29日(土) 1:44 Tom Lane : > > Kohei KaiGai writes: > > I've investigated a crash report of PG-Strom for a few days, then I doubt > > add_partial_path() can unexpectedly release dominated old partial path > > but still referenced by other Gather node, and it leads unexpected system > > crash.

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Tom Lane
Thomas Munro writes: > I was going to suggest that we might be able to use a single > not-visible-to-users number that is mixed into the existing recipe, so > that we only ever read urandom once for the cluster. Yeah, I was thinking along similar lines, but there's a problem: InitProcessGlobals r

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Thomas Munro
On Sat, Dec 29, 2018 at 1:37 PM Tom Lane wrote: > I wrote: > > Looking at this, I seem to remember that we considered doing exactly this > > awhile ago, but refrained because there was concern about depleting the > > system's reserve of entropy if we have a high backend spawn rate, and it > > didn

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Tom Lane
I wrote: > Looking at this, I seem to remember that we considered doing exactly this > awhile ago, but refrained because there was concern about depleting the > system's reserve of entropy if we have a high backend spawn rate, and it > didn't seem like there was a security reason to insist on unpre

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2018-12-28 Thread Peter Geoghegan
On Fri, Dec 28, 2018 at 3:20 PM Heikki Linnakangas wrote: > Right. You'll need to do the free space computations from left to right, > but once you have done that, you can compute the penalties in any order. > > I'm envisioning that you have an array, with one element for each item > on the page (

Re: Prepare Transaction support for ON COMMIT DROP temporary tables

2018-12-28 Thread Michael Paquier
On Fri, Dec 28, 2018 at 08:32:15PM +0100, Dimitri Fontaine wrote: > Alvaro Herrera writes: >> I suggest to add in your regression tests a case where the prepared >> transaction commits, and ensuring that the temp table is gone from >> catalogs. > > Please find such a revision attached. Being abl

Re: Minor comment fix for pg_config_manual.h

2018-12-28 Thread Michael Paquier
On Fri, Dec 28, 2018 at 12:37:41AM -0300, Alvaro Herrera wrote: > Looks good to me. Thanks for the lookup. I have committed and back-patched to v11 for consistency. -- Michael signature.asc Description: PGP signature

Re: Offline enabling/disabling of data checksums

2018-12-28 Thread Michael Paquier
On Fri, Dec 28, 2018 at 10:12:24AM +0100, Magnus Hagander wrote: > I think a heads- up in the way of "planning to change it, now's the time to > yell" is the reasonable thing. And done. -- Michael signature.asc Description: PGP signature

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2018-12-28 Thread Heikki Linnakangas
On 29/12/2018 01:04, Peter Geoghegan wrote: However, naively computing the penalty upfront for every offset would be a bit wasteful. Instead, start from the middle of the page, and walk "outwards" towards both ends, until you find a "good enough" penalty. You can't start at the middle of the pa

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Tom Lane
I wrote: > Thomas Munro writes: >> +1, but I wonder if just separating them is enough. Is our seeding >> algorithm good enough for this new purpose? The initial seed is 100% >> predictable to a logged in user (it's made from the backend PID and >> backend start time, which we tell you), and not

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2018-12-28 Thread Peter Geoghegan
On Fri, Dec 28, 2018 at 10:04 AM Heikki Linnakangas wrote: > I spent some time reviewing this. I skipped the first patch, to add a > column to pg_depend, and I got through patches 2, 3 and 4. Impressive > results, and the code looks sane. Thanks! I really appreciate your taking the time to do suc

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Tom Lane
I wrote: > On further reflection, it seems likely that in most installations a lot > of processes never invoke drandom()/setseed() at all, making such work > in InitProcessGlobals a waste of cycles. Probably a better idea is to > have drandom() initialize the seed on first use, if it wasn't alread

Re: Regression tests using multiple sessions

2018-12-28 Thread Alvaro Herrera
Hi, On 2018-Dec-28, Mitar wrote: > Hi! > > Thanks. You're welcome. Please don't top-post. Regards -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Regression tests using multiple sessions

2018-12-28 Thread Mitar
Hi! Thanks. Mitar On Fri, Dec 28, 2018 at 1:01 PM Alvaro Herrera wrote: > > Hi, > > On 2018-Dec-28, Mitar wrote: > > > How can I define regression tests which should use multiple client > > sessions to test interaction between them? > > See src/test/isolation/README. > > -- > Álvaro Herrera

Re: Regression tests using multiple sessions

2018-12-28 Thread Alvaro Herrera
Hi, On 2018-Dec-28, Mitar wrote: > How can I define regression tests which should use multiple client > sessions to test interaction between them? See src/test/isolation/README. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Trai

Regression tests using multiple sessions

2018-12-28 Thread Mitar
Hi! How can I define regression tests which should use multiple client sessions to test interaction between them? Mitar -- http://mitar.tnode.com/ https://twitter.com/mitar_m

Re: [Patch] Create a new session in postmaster by calling setsid()

2018-12-28 Thread Tom Lane
Heikki Linnakangas writes: > We talked about adding a flag to postmaster, to tell it to call > setsid(). But I'm not sure what would be the appropriate time to do it. Yeah, there's no ideal time in the postmaster. > Another idea would be to call setsid() in pg_ctl, after forking > postmaster,

Re: Prepare Transaction support for ON COMMIT DROP temporary tables

2018-12-28 Thread Dimitri Fontaine
Alvaro Herrera writes: > Glad to see you submitting patches again. Thanks! > I suggest to add in your regression tests a case where the prepared > transaction commits, and ensuring that the temp table is gone from > catalogs. Please find such a revision attached. Regards, -- dim commit 0b2b2

Re: [Patch] Create a new session in postmaster by calling setsid()

2018-12-28 Thread Heikki Linnakangas
On 30/09/2018 15:47, Michael Paquier wrote: On Thu, Sep 13, 2018 at 12:24:38PM -0700, Andres Freund wrote: On 2018-09-13 15:27:58 +0800, Paul Guo wrote: From the respective of users instead of developers, I think for such important service, tty should be dissociated, i.e. postmaster should be

Re: Prepare Transaction support for ON COMMIT DROP temporary tables

2018-12-28 Thread Alvaro Herrera
Hi Dimitri On 2018-Dec-28, Dimitri Fontaine wrote: > Please find attached a patch to enable support for temporary tables in > prepared transactions when ON COMMIT DROP has been specified. Glad to see you submitting patches again. I suggest to add in your regression tests a case where the prepar

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2018-12-28 Thread Heikki Linnakangas
On 04/12/2018 05:10, Peter Geoghegan wrote: Attached is v9, ... I spent some time reviewing this. I skipped the first patch, to add a column to pg_depend, and I got through patches 2, 3 and 4. Impressive results, and the code looks sane. I wrote a laundry list of little comments on minor th

Re: rewrite ExecPartitionCheckEmitError

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-07, Robert Haas wrote: > On Thu, Dec 6, 2018 at 5:22 PM Alvaro Herrera > wrote: > > Just on cleanliness grounds, I propose to rewrite the function in > > $SUBJECT. I came across this while reviewing some already-committed > > patch for partition pruning, and it's been sitting in my

Re: removal of dangling temp tables

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-28, Tom Lane wrote: > I wrote: > > Alvaro Herrera writes: > >> I looked at > >> https://github.com/postgrespro/pg_wait_sampling/blob/master/pg_wait_sampling.c > >> https://github.com/citusdata/citus/search?q=pgproc&unscoped_q=pgproc > >> and skimmed a few others can't find any instan

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Tom Lane
Fabien COELHO writes: >> but I don't feel a need for replacing the algorithm. > Hmmm. Does it mean that you would veto any change, even if the speed > concern is addressed (i.e. faster/not slower with better quality)? Well, not veto exactly, but I'd be suspicious of it. First, erand48 has been

Re: removal of dangling temp tables

2018-12-28 Thread Tom Lane
I wrote: > Alvaro Herrera writes: >> I looked at >> https://github.com/postgrespro/pg_wait_sampling/blob/master/pg_wait_sampling.c >> https://github.com/citusdata/citus/search?q=pgproc&unscoped_q=pgproc >> and skimmed a few others can't find any instance where the full struct >> is used, as oppos

Re: add_partial_path() may remove dominated path but still in use

2018-12-28 Thread Tom Lane
Kohei KaiGai writes: > I've investigated a crash report of PG-Strom for a few days, then I doubt > add_partial_path() can unexpectedly release dominated old partial path > but still referenced by other Gather node, and it leads unexpected system > crash. Hm. This seems comparable to the special

Re: removal of dangling temp tables

2018-12-28 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Dec-28, Alvaro Herrera wrote: >> On 2018-Dec-28, Michael Paquier wrote: >>> There are many exotic extensions which could be using sizeof(PGPROC) >>> as that's a popular structure, >> Can you show one instance of this? > I looked at > https://github.com/postgresp

Re: removal of dangling temp tables

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-28, Alvaro Herrera wrote: > On 2018-Dec-28, Michael Paquier wrote: > > > On Fri, Dec 28, 2018 at 12:05:34AM -0300, Alvaro Herrera wrote: > > > The server code allocates arrays, but that's fine because that code is > > > recompiled. Extensions only pass pointers around -- they don't c

Re: pg_dump multi VALUES INSERT

2018-12-28 Thread Fabien COELHO
At first i also try to do it like that but it seems the function will became long and more complex to me Probably. But calling it with size 100 should result in the same behavior, so it is really just an extension of the preceeding one? Or am I missing something? -- Fabien.

Re: Poor buildfarm coverage of strong-random alternatives

2018-12-28 Thread Tom Lane
Heikki Linnakangas writes: > Yeah, there probably isn't anyone needing --disable-strong-random in > practice. The situation is slightly different between the frontend and > backend, though. It's more likely that someone might need to build libpq > on a very ancient system, but not the server. A

Re: pg_dump multi VALUES INSERT

2018-12-28 Thread Surafel Temesgen
On Tue, Dec 25, 2018 at 2:47 PM Fabien COELHO wrote: Thank you for looking into it > > Hello Surafel, > > > Thank you for informing, Here is an updated patch against current master > > Patch applies cleanly, compiles, "make check" is okay, but given that the > feature is not tested... > > Featur

Re: Offline enabling/disabling of data checksums

2018-12-28 Thread Fabien COELHO
[...] I'm not sure data checksums are particularly great evidence. For example with the recent fsync issues, we might have ended with partial writes (and thus invalid checksums). The OS migh have even told us about the failure, but we've gracefully ignored it. So I'm afraid data checksums are

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Fabien COELHO
Hello again, - lrand48 (48 bits state as 3 uint16)is 29 ops (10 =, 8 *, 7 +, 4 >>) - xoshiro256** (256 bits states as 4 uint64) is 24 ops (18 if rot in hw) 8 =, 2 *, 2 +, 5 <<, 5 ^, 2 | Small benchmark on my laptop with gcc-7.3 -O3: - pg_lrand48 takes 4.0 seconds to generate 1 b

Re: Poor buildfarm coverage of strong-random alternatives

2018-12-28 Thread Heikki Linnakangas
On 28/12/2018 01:14, Tom Lane wrote: Michael Paquier writes: On Thu, Dec 27, 2018 at 03:56:52PM -0500, Tom Lane wrote: More urgently, what about the lack of --disable-strong-random coverage? I feel like we should either have a buildfarm critter or two testing that code, or decide that it's no

Re: removal of dangling temp tables

2018-12-28 Thread Alvaro Herrera
On 2018-Dec-28, Michael Paquier wrote: > On Fri, Dec 28, 2018 at 12:05:34AM -0300, Alvaro Herrera wrote: > > The server code allocates arrays, but that's fine because that code is > > recompiled. Extensions only pass pointers around -- they don't create > > any additional arrays. > > There are m

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Adrien NAYRAT
On 12/26/18 7:45 PM, Tom Lane wrote: I wonder whether we should establish a project policy to avoid use of random() for internal purposes, ie try to get to a point where drandom() is the only caller in the backend. FYI I picked the idea from auto_explain. Maybe we will have to change auto_expl

Re: Prepare Transaction support for ON COMMIT DROP temporary tables

2018-12-28 Thread Vik Fearing
On 28/12/2018 12:46, Dimitri Fontaine wrote: > Hi, > > Please find attached a patch to enable support for temporary tables in > prepared transactions when ON COMMIT DROP has been specified. The comments I made on IRC have been addressed in this version of the patch, so it looks good to me. -- Vi

Prepare Transaction support for ON COMMIT DROP temporary tables

2018-12-28 Thread Dimitri Fontaine
Hi, Please find attached a patch to enable support for temporary tables in prepared transactions when ON COMMIT DROP has been specified. The comment in the existing code around this idea reads: * Don't allow PREPARE TRANSACTION if we've accessed a temporary table in * this tra

Re: could recovery_target_timeline=latest be the default in standby mode?

2018-12-28 Thread Peter Eisentraut
On 28/12/2018 00:15, Michael Paquier wrote: > Yes, I was also thinking something among those lines, and the patch is > a bit confusing by linking standby mode with latest timeline. It > seems to me that switching the default value to "latest" at GUC level > would be the way to go, instead of picki

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Fabien COELHO
- lrand48 (48 bits state as 3 uint16)is 29 ops (10 =, 8 *, 7 +, 4 >>) - xoshiro256** (256 bits states as 4 uint64) is 24 ops (18 if rot in hw) 8 =, 2 *, 2 +, 5 <<, 5 ^, 2 | See http://vigna.di.unimi.it/xorshift/ Small benchmark on my laptop with gcc-7.3 -O3: - pg_lrand48 takes

Re: Offline enabling/disabling of data checksums

2018-12-28 Thread Magnus Hagander
On Fri, Dec 28, 2018 at 1:14 AM Tomas Vondra wrote: > > > On 12/28/18 12:25 AM, Michael Paquier wrote: > > On Thu, Dec 27, 2018 at 03:46:48PM +0100, Tomas Vondra wrote: > >> On 12/27/18 11:43 AM, Magnus Hagander wrote: > >>> Should we double-check with packagers that this won't cause a problem? >

Re: insensitive collations

2018-12-28 Thread Peter Eisentraut
Here is an updated patch. I have updated the naming to "deterministic", as discussed. I have fixed up support for the "name" type, added foreign key support, psql, pg_dump support, more tests. There are a couple of TODOs in bpchar support that I need to look into a bit more. But other than that

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-28 Thread Andres Freund
On 2018-12-27 14:22:11 -0500, Tom Lane wrote: > Andrew Dunstan writes: > > A smaller project might be to see if we can replace the binary keyword > > search in ScanKeyword with a perfect hashing function generated by > > gperf, or something similar. I had a quick look at that, too. > > Yeah, we'v

Re: Feature: triggers on materialized views

2018-12-28 Thread Mitar
Hi! False alarm. It just looks like updating patches takes longer than updating e-mails. Mitar On Fri, Dec 28, 2018 at 12:11 AM Mitar wrote: > > Hi! > > Hm, why in commitfest it does not display the latest patch? > > https://commitfest.postgresql.org/21/1953/ > > It does display correctly the

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Fabien COELHO
About costs, not counting array accesses: - lrand48 (48 bits state as 3 uint16)is 29 ops (10 =, 8 *, 7 +, 4 >>) - xorshift+ (128 bits state as 2 uint64) is 13 ops ( 5 =, 0 *, 1 +, 3 >>, 4 ^) - xororshift128+ (idem) is 17 ops ( 6 =, 0 *, 1 +, 5 >>, 3 ^, 2

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2018-12-28 Thread Masahiko Sawada
On Thu, Dec 27, 2018 at 10:24 PM Alvaro Herrera wrote: > > On 2018-Dec-27, Alexey Kondratov wrote: > > > To summarize: > > > > 1) Alvaro and Michael agreed, that REINDEX with tablespace move may be > > useful. This is done in the patch attached to my initial email. Adding > > REINDEX to ALTER TABL

Re: postgres_fdw: oddity in costing aggregate pushdown paths

2018-12-28 Thread Etsuro Fujita
(2018/12/04 17:24), Etsuro Fujita wrote: > (2018/12/03 20:20), Etsuro Fujita wrote: >> (2018/11/30 18:51), Etsuro Fujita wrote: >>> (2018/11/28 13:38), Etsuro Fujita wrote: BTW another thing I noticed is this comment on costing aggregate pushdown paths using local statistics in estimate_p

Re: [HACKERS] CLUSTER command progress monitor

2018-12-28 Thread Tatsuro Yamada
Hi, Do you have a new version of this patch? If not, do you think you'll have something in time for the upcoming commitfest? Not yet, I'll be able to send only a rebased patch by the end of this month. I mean it has no design change because I can't catch up on how to get a progress from sort

Re: random() (was Re: New GUC to sample log queries)

2018-12-28 Thread Fabien COELHO
Hello Tom, Another idea, which would be a lot less prone to breakage by add-on code, is to change drandom() and setseed() to themselves use pg_erand48() with a private seed. The pg_erand48 code looks like crumbs from the 70's optimized for 16 bits architectures (which it is probably not, bu

Re: Feature: triggers on materialized views

2018-12-28 Thread Mitar
Hi! Hm, why in commitfest it does not display the latest patch? https://commitfest.postgresql.org/21/1953/ It does display correctly the latest e-mail, but not the link to the patch. :-( Mitar On Thu, Dec 27, 2018 at 11:51 PM Mitar wrote: > > Hi! > > One more version of the patch with slight