Re: pgsql: Add parallel-aware hash joins.

2018-01-03 Thread Thomas Munro
On Wed, Jan 3, 2018 at 2:38 PM, Tom Lane wrote: > Thomas Munro writes: >> I mean that ExecChooseHashTableSize() estimates the hash table size like >> this: >> inner_rel_bytes = ntuples * tupsize; > >> ... but then at execution time, in the Parallel Hash case, we do >> memory accounting not i

Re: pgsql: Fix deadlock hazard in CREATE INDEX CONCURRENTLY

2018-01-03 Thread Alvaro Herrera
Hello, Andres Freund wrote: > On 2018-01-03 02:20:14 +, Alvaro Herrera wrote: > > Include an isolationtester spec that 8 out of 10 times reproduces the > > deadlock with the unpatched code for me (Álvaro). > > The isolation test doesn't appear to be fully stable: > > https://buildfarm.post

Re: pgsql: Fix deadlock hazard in CREATE INDEX CONCURRENTLY

2018-01-03 Thread Alvaro Herrera
Alvaro Herrera wrote: > However this means that the test will get removed in 9.4 and 9.5 because > isolationtester is not smart enough there. > > I suppose the other option would be to add an alternate expected file > for the test. Actually, so far only 9.6 and up have failed. Maybe the old iso

Re: pgsql: Fix deadlock hazard in CREATE INDEX CONCURRENTLY

2018-01-03 Thread Tom Lane
Alvaro Herrera writes: > Alvaro Herrera wrote: >> However this means that the test will get removed in 9.4 and 9.5 because >> isolationtester is not smart enough there. >> >> I suppose the other option would be to add an alternate expected file >> for the test. > Actually, so far only 9.6 and up

pgsql: Fix isolation test to be less timing-dependent

2018-01-03 Thread Alvaro Herrera
Fix isolation test to be less timing-dependent I did this by adding another locking process, which makes the other two wait. This way the output should be stable enough. Per buildfarm and Andres Freund Discussion: https://postgr.es/m/20180103034445.t3utrtrnrevfs...@alap3.anarazel.de Branch ---

pgsql: Fix isolation test to be less timing-dependent

2018-01-03 Thread Alvaro Herrera
Fix isolation test to be less timing-dependent I did this by adding another locking process, which makes the other two wait. This way the output should be stable enough. Per buildfarm and Andres Freund Discussion: https://postgr.es/m/20180103034445.t3utrtrnrevfs...@alap3.anarazel.de Branch ---

pgsql: Fix isolation test to be less timing-dependent

2018-01-03 Thread Alvaro Herrera
Fix isolation test to be less timing-dependent I did this by adding another locking process, which makes the other two wait. This way the output should be stable enough. Per buildfarm and Andres Freund Discussion: https://postgr.es/m/20180103034445.t3utrtrnrevfs...@alap3.anarazel.de Branch ---

Re: pgsql: Fix deadlock hazard in CREATE INDEX CONCURRENTLY

2018-01-03 Thread Alvaro Herrera
Tom Lane wrote: > Alvaro Herrera writes: > > Actually, so far only 9.6 and up have failed. Maybe the old > > isolationtester is different enough that the other thing doesn't happen. > > > I'm more inclined now to add the alternate file instead of the other > > patch. > > Meh. I'd rather have

pgsql: Allow ldaps when using ldap authentication

2018-01-03 Thread Peter Eisentraut
Allow ldaps when using ldap authentication While ldaptls=1 provides an RFC 4513 conforming way to do LDAP authentication with TLS encryption, there was an earlier de facto standard way to do LDAP over SSL called LDAPS. Even though it's not enshrined in a standard, it's still widely used and somet

pgsql: Teach eval_const_expressions() to handle some more cases.

2018-01-03 Thread Tom Lane
Teach eval_const_expressions() to handle some more cases. Add some infrastructure (mostly macros) to make it easier to write typical cases for constant-expression simplification. Add simplification processing for ArrayRef, RowExpr, and ScalarArrayOpExpr node types, which formerly went unsimplifie

pgsql: Fix some minor errors in new PHJ code.

2018-01-03 Thread Tom Lane
Fix some minor errors in new PHJ code. Correct ExecParallelHashTuplePrealloc's estimate of whether the space_allowed limit is exceeded. Be more consistent about tuples that are exactly HASH_CHUNK_THRESHOLD in size (they're "small", not "large"). Neither of these things explain the current buildfa

Re: pgsql: Add parallel-aware hash joins.

2018-01-03 Thread Tom Lane
Thomas Munro writes: > On Wed, Jan 3, 2018 at 2:38 PM, Tom Lane wrote: >> Hm. That could do it, except it doesn't really account for the observed >> result that slower single-processor machines seem more prone to the >> bug. Surely they should be less likely to get multiple workers activated.

Re: pgsql: Fix deadlock hazard in CREATE INDEX CONCURRENTLY

2018-01-03 Thread Tom Lane
Alvaro Herrera writes: > Tom Lane wrote: >> Meh. I'd rather have the more stable test going forward; I think >> alternate expected-files too easily hide unexpected behavior. We could >> try leaving 9.4/9.5 alone and see if it's true that it doesn't fail >> there. If not, I wouldn't mind losing

Re: pgsql: Fix deadlock hazard in CREATE INDEX CONCURRENTLY

2018-01-03 Thread Alvaro Herrera
Tom Lane wrote: > Alvaro Herrera writes: > > Tom Lane wrote: > >> Meh. I'd rather have the more stable test going forward; I think > >> alternate expected-files too easily hide unexpected behavior. We could > >> try leaving 9.4/9.5 alone and see if it's true that it doesn't fail > >> there. If

Re: pgsql: Fix deadlock hazard in CREATE INDEX CONCURRENTLY

2018-01-03 Thread Tom Lane
Alvaro Herrera writes: > Tom Lane wrote: >> Some of the buildfarm machines still don't like this. > It proves me wrong about the ordering in which the steps return > completion being consistent: > *** > *** 20,24 > unlck > t > - step s1i: <... comp

Re: pgsql: Allow ldaps when using ldap authentication

2018-01-03 Thread Thomas Munro
On Thu, Jan 4, 2018 at 4:14 AM, Peter Eisentraut wrote: > Allow ldaps when using ldap authentication Other Windows systems seem OK with this, but baiji said: .\src\backend\libpq\auth.c(2538): error C2065: 'LDAPS_PORT' : undeclared identifier Checking where that's supposed to come from... -- T

pgsql: Make XactLockTableWait work for transactions that are not yet se

2018-01-03 Thread Alvaro Herrera
Make XactLockTableWait work for transactions that are not yet self-locked XactLockTableWait assumed that its xid argument has already added itself to the lock table. That assumption led to another assumption that if locking the xid has succeeded but the xid is reported as still in progress, then

pgsql: Make XactLockTableWait work for transactions that are not yet se

2018-01-03 Thread Alvaro Herrera
Make XactLockTableWait work for transactions that are not yet self-locked XactLockTableWait assumed that its xid argument has already added itself to the lock table. That assumption led to another assumption that if locking the xid has succeeded but the xid is reported as still in progress, then

pgsql: Make XactLockTableWait work for transactions that are not yet se

2018-01-03 Thread Alvaro Herrera
Make XactLockTableWait work for transactions that are not yet self-locked XactLockTableWait assumed that its xid argument has already added itself to the lock table. That assumption led to another assumption that if locking the xid has succeeded but the xid is reported as still in progress, then

pgsql: Make XactLockTableWait work for transactions that are not yet se

2018-01-03 Thread Alvaro Herrera
Make XactLockTableWait work for transactions that are not yet self-locked XactLockTableWait assumed that its xid argument has already added itself to the lock table. That assumption led to another assumption that if locking the xid has succeeded but the xid is reported as still in progress, then

pgsql: Make XactLockTableWait work for transactions that are not yet se

2018-01-03 Thread Alvaro Herrera
Make XactLockTableWait work for transactions that are not yet self-locked XactLockTableWait assumed that its xid argument has already added itself to the lock table. That assumption led to another assumption that if locking the xid has succeeded but the xid is reported as still in progress, then

pgsql: Fix use of config-specific libraries for Windows OpenSSL

2018-01-03 Thread Andrew Dunstan
Fix use of config-specific libraries for Windows OpenSSL Commit 614350a3 allowed for an different builds of OpenSSL libraries on Windows, but ignored the fact that the alternative builds don't have config-specific libraries. This patch fixes the Solution file to ask for the correct libraries. per

pgsql: Fix use of config-specific libraries for Windows OpenSSL

2018-01-03 Thread Andrew Dunstan
Fix use of config-specific libraries for Windows OpenSSL Commit 614350a3 allowed for an different builds of OpenSSL libraries on Windows, but ignored the fact that the alternative builds don't have config-specific libraries. This patch fixes the Solution file to ask for the correct libraries. per

pgsql: Fix use of config-specific libraries for Windows OpenSSL

2018-01-03 Thread Andrew Dunstan
Fix use of config-specific libraries for Windows OpenSSL Commit 614350a3 allowed for an different builds of OpenSSL libraries on Windows, but ignored the fact that the alternative builds don't have config-specific libraries. This patch fixes the Solution file to ask for the correct libraries. per

pgsql: Fix use of config-specific libraries for Windows OpenSSL

2018-01-03 Thread Andrew Dunstan
Fix use of config-specific libraries for Windows OpenSSL Commit 614350a3 allowed for an different builds of OpenSSL libraries on Windows, but ignored the fact that the alternative builds don't have config-specific libraries. This patch fixes the Solution file to ask for the correct libraries. per

pgsql: Fix use of config-specific libraries for Windows OpenSSL

2018-01-03 Thread Andrew Dunstan
Fix use of config-specific libraries for Windows OpenSSL Commit 614350a3 allowed for an different builds of OpenSSL libraries on Windows, but ignored the fact that the alternative builds don't have config-specific libraries. This patch fixes the Solution file to ask for the correct libraries. per

pgsql: Fix use of config-specific libraries for Windows OpenSSL

2018-01-03 Thread Andrew Dunstan
Fix use of config-specific libraries for Windows OpenSSL Commit 614350a3 allowed for an different builds of OpenSSL libraries on Windows, but ignored the fact that the alternative builds don't have config-specific libraries. This patch fixes the Solution file to ask for the correct libraries. per

pgsql: Rename pg_rewind's copy_file_range() to avoid conflict with new

2018-01-03 Thread Andres Freund
Rename pg_rewind's copy_file_range() to avoid conflict with new linux syscall. Upcoming versions of glibc will contain copy_file_range(2), a wrapper around a new linux syscall for in-kernel copying of data ranges. This conflicts with pg_rewinds function of the same name. Therefore rename pg_rewin

pgsql: Rename pg_rewind's copy_file_range() to avoid conflict with new

2018-01-03 Thread Andres Freund
Rename pg_rewind's copy_file_range() to avoid conflict with new linux syscall. Upcoming versions of glibc will contain copy_file_range(2), a wrapper around a new linux syscall for in-kernel copying of data ranges. This conflicts with pg_rewinds function of the same name. Therefore rename pg_rewin

pgsql: Rename pg_rewind's copy_file_range() to avoid conflict with new

2018-01-03 Thread Andres Freund
Rename pg_rewind's copy_file_range() to avoid conflict with new linux syscall. Upcoming versions of glibc will contain copy_file_range(2), a wrapper around a new linux syscall for in-kernel copying of data ranges. This conflicts with pg_rewinds function of the same name. Therefore rename pg_rewin

pgsql: Rename pg_rewind's copy_file_range() to avoid conflict with new

2018-01-03 Thread Andres Freund
Rename pg_rewind's copy_file_range() to avoid conflict with new linux syscall. Upcoming versions of glibc will contain copy_file_range(2), a wrapper around a new linux syscall for in-kernel copying of data ranges. This conflicts with pg_rewinds function of the same name. Therefore rename pg_rewin

pgsql: Revert "Fix isolation test to be less timing-dependent"

2018-01-03 Thread Alvaro Herrera
Revert "Fix isolation test to be less timing-dependent" This reverts commit 2268e6afd596. It turned out that inconsistency in the report is still possible, so go back to the simpler formulation of the test and instead add an alternate expected output. Discussion: https://postgr.es/m/201801031937

pgsql: Revert "Fix isolation test to be less timing-dependent"

2018-01-03 Thread Alvaro Herrera
Revert "Fix isolation test to be less timing-dependent" This reverts commit 2268e6afd596. It turned out that inconsistency in the report is still possible, so go back to the simpler formulation of the test and instead add an alternate expected output. Discussion: https://postgr.es/m/201801031937

pgsql: Revert "Fix isolation test to be less timing-dependent"

2018-01-03 Thread Alvaro Herrera
Revert "Fix isolation test to be less timing-dependent" This reverts commit 2268e6afd596. It turned out that inconsistency in the report is still possible, so go back to the simpler formulation of the test and instead add an alternate expected output. Discussion: https://postgr.es/m/201801031937

Re: pgsql: Allow ldaps when using ldap authentication

2018-01-03 Thread Thomas Munro
On Thu, Jan 4, 2018 at 9:07 AM, Thomas Munro wrote: > On Thu, Jan 4, 2018 at 4:14 AM, Peter Eisentraut wrote: >> Allow ldaps when using ldap authentication > > Other Windows systems seem OK with this, but baiji said: > > .\src\backend\libpq\auth.c(2538): error C2065: 'LDAPS_PORT' : > undeclared i

Re: pgsql: Fix deadlock hazard in CREATE INDEX CONCURRENTLY

2018-01-03 Thread Alvaro Herrera
Tom Lane wrote: > No, I think that's probably a bad idea, because it would mean that in > cases where you do care about the finishing order (which is all of > them up to now), the test output would fail to prove that you got the > expected behavior. Hmm .. I was thinking that we would check all t

pgsql: Fix typo

2018-01-03 Thread Alvaro Herrera
Fix typo Author: Dagfinn Ilmari Mannsåker Discussion: https://postgr.es/m/d8jefpk4jtd@dalvik.ping.uio.no Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/bab2969867fbba6a6d12730f36a20d13542aea5a Modified Files -- src/backend/utils/cache/lsyscache.c |

pgsql: Clean up tupdesc.c for recent changes.

2018-01-03 Thread Tom Lane
Clean up tupdesc.c for recent changes. TupleDescCopy needs to have the same effects as CreateTupleDescCopy in that, since it doesn't copy constraints, it should clear the per-attribute fields associated with them. Oversight in commit cc5f81366. Since TupleDescCopy has already established the pre

Re: pgsql: Add parallel-aware hash joins.

2018-01-03 Thread Tom Lane
So this patch has been in place for two weeks, the buildfarm has still got the measles, and we're entering the January commitfest with a lot of other work to get done. I realize that the two weeks were mostly holiday time, but it's time to have some urgency about clearing the buildfarm failures.

Re: pgsql: Add parallel-aware hash joins.

2018-01-03 Thread Andres Freund
Hi, On 2018-01-03 20:17:04 -0500, Tom Lane wrote: > So this patch has been in place for two weeks, the buildfarm has still > got the measles, and we're entering the January commitfest with a lot > of other work to get done. I realize that the two weeks were mostly > holiday time, but it's time to

Re: pgsql: Add parallel-aware hash joins.

2018-01-03 Thread Thomas Munro
On Thu, Jan 4, 2018 at 2:17 PM, Tom Lane wrote: > So this patch has been in place for two weeks, the buildfarm has still > got the measles, and we're entering the January commitfest with a lot > of other work to get done. I realize that the two weeks were mostly > holiday time, but it's time to h

Re: pgsql: Add parallel-aware hash joins.

2018-01-03 Thread Tom Lane
Thomas Munro writes: > I spent a lot of time trying and failing to get the world's slowest 32 > bit powerpc emulation to reproduce this. Bleugh. Before we rip that > test out, would you mind checking if this passes repeatedly on gaur or > pademelon? Will do, but that machine is none too fast it

Re: pgsql: Fix deadlock hazard in CREATE INDEX CONCURRENTLY

2018-01-03 Thread Robert Haas
On Wed, Jan 3, 2018 at 4:31 PM, Alvaro Herrera wrote: > step s8a1: LOCK TABLE a1; > step s8a1: <... completed> > step s7a8: <... completed> > error in steps s8a1 s7a8: ERROR: deadlock detected > > TBH I'm surprised that this is never reported in the other order but > that this doesn't hold for t

pgsql: Tweak parallel hash join test case in hopes of improving stabili

2018-01-03 Thread Tom Lane
Tweak parallel hash join test case in hopes of improving stability. This seems to make things better on gaur, let's see what the rest of the buildfarm thinks. Thomas Munro Discussion: https://postgr.es/m/CAEepm=1uuT8iJxMEsR=jl+3zei87db2v0+0h9o_ruxxczpz...@mail.gmail.com Branch -- master D

Re: pgsql: Add parallel-aware hash joins.

2018-01-03 Thread Tom Lane
Thomas Munro writes: > I spent a lot of time trying and failing to get the world's slowest 32 > bit powerpc emulation to reproduce this. Bleugh. Before we rip that > test out, would you mind checking if this passes repeatedly on gaur or > pademelon? I ran a couple dozen test cycles on gaur with