Re: [HACKERS] checkpointer continuous flushing

2015-06-19 Thread Fabien COELHO
Hello Andres, - Move fsync as early as possible, suggested by Andres Freund? My opinion is that this should be left out for the nonce. "for the nonce" - what does that mean? Nonce \Nonce\ (n[o^]ns), n. [For the nonce, OE. for the nones, ... {for the nonce}, i. e. for the present time

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Feng Tian
On Wed, Jun 17, 2015 at 10:52 AM, Tomas Vondra wrote: > Hi, > > I'm currently running some tests on a 3TB TPC-H data set, and I tripped > over a pretty bad n_distinct underestimate, causing OOM in HashAgg (which > somehow illustrates the importance of the memory-bounded hashagg patch Jeff > Davis

Re: [HACKERS] anole: assorted stability problems

2015-06-19 Thread Alvaro Herrera
Alvaro Herrera wrote: > Tom Lane wrote: > > Andres Freund writes: > > > > Uh. I'm pretty sure there were some back when that patch went in. And > > > there definitely used to be a couple earlier. I guess itanium really is > > > dying (mixed bad: It's a horrible architecture, but more coverage wou

[HACKERS] castoroides spinlock failure on test_shm_mq

2015-06-19 Thread Alvaro Herrera
Has anybody noticed the way castoroides is randomly failing? SELECT test_shm_mq_pipelined(16384, (select string_agg(chr(32+(random()*95)::int), '') from generate_series(1,27)), 200, 3); ! PANIC: stuck spinlock (100cb92f4) detected at atomics.c:30 ! server closed the connection unexpectedl

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Jeff Janes
On Fri, Jun 19, 2015 at 1:39 PM, Tomas Vondra wrote: > On 06/19/2015 09:48 PM, Jeff Janes wrote: > >> On Fri, Jun 19, 2015 at 12:27 PM, Tomas Vondra >> mailto:tomas.von...@2ndquadrant.com>> >> wrote: >> >> But I think you might be on to something, because I manually >> collected a random

[HACKERS] Insufficient locking for ALTER DEFAULT PRIVILEGES

2015-06-19 Thread Vik Fearing
I came across the following bug this week: Session 0: begin; create schema bug; alter default privileges in schema bug grant all on tables to postgres; commit; Session 1: begin; alter default privileges in schema bug grant all on tables to postgres; Session 2: alter default privileges in schema

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Michael Paquier
On Fri, Jun 19, 2015 at 11:45 PM, Tom Lane wrote: > Michael Paquier writes: >> Attached is a patch fixing those problems and improving the log >> facility as it really helped me out with those issues. The simplest >> fix would be to include the -w switch missing in the tests of >> pg_rewind and p

Re: [HACKERS] Need Multixact Freezing Docs

2015-06-19 Thread Alvaro Herrera
Jim Nasby wrote: > On 6/14/15 9:50 AM, Alvaro Herrera wrote: > >+values[0] = MultiXactState->oldestMultiXactId; > > What about oldestOffset and offsetStopLimit? Seems those would be useful > too. Looks good other than that. Yeah, that's what I was trying to say. How about this? I realized t

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Michael Paquier
On Sat, Jun 20, 2015 at 12:44 AM, Tom Lane wrote: > Andres Freund writes: >> On 2015-06-19 11:16:18 -0400, Robert Haas wrote: >>> On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane wrote: I wonder whether it's such a good idea for the postmaster to give up waiting before all children are gone

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Michael Paquier
On Sat, Jun 20, 2015 at 12:07 AM, Tom Lane wrote: > Michael Paquier writes: >> Now if we look at RewindTest.pm, there is the following code: >> if ($test_master_datadir) >> { >> system >> "pg_ctl -D $test_master_datadir -s -m immediate stop >> 2>

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Tomas Vondra
On 06/19/2015 09:48 PM, Jeff Janes wrote: On Fri, Jun 19, 2015 at 12:27 PM, Tomas Vondra mailto:tomas.von...@2ndquadrant.com>> wrote: But I think you might be on to something, because I manually collected a random sample with 30k rows (by explicitly generating 30k random TIDs), and I

Re: [HACKERS] Tab completion for TABLESAMPLE

2015-06-19 Thread Brendan Jurd
On Fri, 19 Jun 2015 at 21:05 Petr Jelinek wrote: > On 2015-06-19 09:08, Brendan Jurd wrote: > > I > > think it would be convenient and user-friendly to complete the opening > > bracket -- it would make it perfectly clear that an argument is required > > for the syntax to be valid. > > > > Agreed,

Re: [HACKERS] 9.5 release notes

2015-06-19 Thread Andres Freund
Hi, On 2015-06-11 00:15:21 -0400, Bruce Momjian wrote: > I have committed the first draft of the 9.5 release notes. You can view > the output here: I'm looking through all the commits, checking which I think should possibly be mentioned additionally: - 9f03ca91 - Speed up CREATE INDEX by avoidin

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Jeff Janes
On Fri, Jun 19, 2015 at 12:27 PM, Tomas Vondra wrote: > But I think you might be on to something, because I manually collected a > random sample with 30k rows (by explicitly generating 30k random TIDs), and > I get this: > > tpch=# select cnt, count(*) from (select l_orderkey, count(*) AS cnt fr

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Tomas Vondra
On 06/19/2015 08:32 PM, Jeff Janes wrote: On Wed, Jun 17, 2015 at 10:52 AM, Tomas Vondra mailto:tomas.von...@2ndquadrant.com>> wrote: Hi, I'm currently running some tests on a 3TB TPC-H data set, and I tripped over a pretty bad n_distinct underestimate, causing OOM in HashAgg

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2015-06-19 Thread Josh Berkus
On 06/19/2015 02:51 PM, Tom Lane wrote: > Josh Berkus writes: >> On 05/14/2015 12:10 PM, Fabien COELHO wrote: >>> Add backslash continuations to pgbench custom scripts. > >> I don't personally agree. I believe that it it worth breaking backwards >> compatibility to support line breaks in pgbench

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Alvaro Herrera
Tom Lane wrote: > Andres Freund writes: > > On 2015-06-19 13:56:21 -0300, Alvaro Herrera wrote: > >> We discussed this when that patch got in (82233ce7ea42d6b). The reason > >> for not waiting, it was argued, is that the most likely reason for those > >> processes not to have already gone away by

Re: [HACKERS] pg_regress not waiting for postmaster to stop

2015-06-19 Thread Andrew Dunstan
On 06/19/2015 02:33 AM, Michael Paquier wrote: Hi all, In pg_regress.c, there is the following code: static void stop_postmaster(void) { if (postmaster_running) { /* We use pg_ctl to issue the kill and wait for stop */ charbuf[MAXP

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2015-06-19 Thread Tom Lane
Josh Berkus writes: > On 05/14/2015 12:10 PM, Fabien COELHO wrote: >> Add backslash continuations to pgbench custom scripts. > I don't personally agree. I believe that it it worth breaking backwards > compatibility to support line breaks in pgbench statements, and that if > we're not going to do

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Jeff Janes
On Wed, Jun 17, 2015 at 10:52 AM, Tomas Vondra wrote: > Hi, > > I'm currently running some tests on a 3TB TPC-H data set, and I tripped > over a pretty bad n_distinct underestimate, causing OOM in HashAgg (which > somehow illustrates the importance of the memory-bounded hashagg patch Jeff > Davis

Re: [HACKERS] pgbench - allow backslash-continuations in custom scripts

2015-06-19 Thread Josh Berkus
On 05/14/2015 12:10 PM, Fabien COELHO wrote: > > Add backslash continuations to pgbench custom scripts. > > The benefit of this approach is that it is upward compatible, and it is > also pretty simple to implement. The downside is that backslash > continuation is not the best syntax ever invented

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Tom Lane
Andres Freund writes: > On 2015-06-19 13:56:21 -0300, Alvaro Herrera wrote: >> We discussed this when that patch got in (82233ce7ea42d6b). The reason >> for not waiting, it was argued, is that the most likely reason for those >> processes not to have already gone away by the time we send SIGKILL

Re: [HACKERS] 9.5 release notes

2015-06-19 Thread Andres Freund
Hi, On 2015-06-11 00:15:21 -0400, Bruce Momjian wrote: > I have committed the first draft of the 9.5 release notes. You can view > the output here: So, I did a pass through master's state: > > >Add Block Range Indexes (BRIN) >(Álvaro Herrera, Heikki Linnakangas, Emr

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Alvaro Herrera
Alvaro Herrera wrote: > We discussed this when that patch got in (82233ce7ea42d6b). The reason > for not waiting, it was argued, is that the most likely reason for those > processes not to have already gone away by the time we send SIGKILL was > that they are stuck somewhere in the kernel, and so

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Andres Freund
On 2015-06-19 13:56:21 -0300, Alvaro Herrera wrote: > We discussed this when that patch got in (82233ce7ea42d6b). The reason > for not waiting, it was argued, is that the most likely reason for those > processes not to have already gone away by the time we send SIGKILL was > that they are stuck so

Re: [HACKERS] Auto-vacuum is not running in 9.1.12

2015-06-19 Thread Andres Freund
On 2015-06-17 18:10:42 -0300, Alvaro Herrera wrote: > Yeah, the case is pretty weird and I'm not really sure that the server > ought to be expected to behave. But if this is actually the only part > of the server that misbehaves because of sudden gigantic time jumps, I > think it's fair to patch i

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Alvaro Herrera
Tom Lane wrote: > Andres Freund writes: > > On 2015-06-19 11:16:18 -0400, Robert Haas wrote: > >> On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane wrote: > >>> I wonder whether it's such a good idea for the postmaster to give > >>> up waiting before all children are gone (postmaster.c:1722 in HEAD). >

[HACKERS] [PATCH] pg_upgrade fails when postgres/template1 isn't in default tablespace

2015-06-19 Thread Marti Raudsepp
Hi list One of my databases failed to upgrade successfully and produced this error in the copying phase: error while copying relation "pg_catalog.pg_largeobject" ("/srv/ssd/PG_9.3_201306121/1/12023" to "/PG_9.4_201409291/1/12130"): No such file or directory Turns out this happens when either the

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Cédric Villemain
Le 19/06/2015 17:21, Andres Freund a écrit : > On 2015-06-19 11:19:23 -0400, Magnus Hagander wrote: >> On Fri, Jun 19, 2015 at 11:17 AM, Stephen Frost wrote: >> >>> * Andres Freund (and...@anarazel.de) wrote: To make slot usage in pg_receivexlog easier, should we add --create-slot-if-n

Re: [HACKERS] Auto-vacuum is not running in 9.1.12

2015-06-19 Thread Alvaro Herrera
Tom Lane wrote: > Alvaro Herrera writes: > > Yeah, the case is pretty weird and I'm not really sure that the server > > ought to be expected to behave. But if this is actually the only part > > of the server that misbehaves because of sudden gigantic time jumps, I > > think it's fair to patch it.

[HACKERS] backend depends on libpgport but Make doesn't know

2015-06-19 Thread Alvaro Herrera
I think we're missing a dependency declaration from the backend to libpgport in the makefiles somewhere. I just git updated on 9.1 on a tree that was already built, ran make, and got this: ../../src/port/libpgport_srv.a(dirmod_srv.o): In function `rmtree': /pgsql/source/REL9_1_STABLE/src/port/dir

[HACKERS] outstanding multixact bugs

2015-06-19 Thread Robert Haas
During the Developer Unconference at PGCon, we made this wiki page to track the MultiXact bugs that are outstanding: https://wiki.postgresql.org/wiki/MultiXact_Bugs Let's update this page as we find or fix things, and let's release when we've fixed enough stuff. Even if you are not involved in t

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Tom Lane
Andres Freund writes: > On 2015-06-19 11:16:18 -0400, Robert Haas wrote: >> On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane wrote: >>> I wonder whether it's such a good idea for the postmaster to give >>> up waiting before all children are gone (postmaster.c:1722 in HEAD). >> I doubt it. > Seconded.

Re: [HACKERS] Inheritance planner CPU and memory usage change since 9.3.2

2015-06-19 Thread Tom Lane
Petr Jelinek writes: > On 2015-06-19 01:04, Petr Jelinek wrote: >> On 2015-06-19 00:38, Petr Jelinek wrote: >>> On 2015-06-18 22:04, Tom Lane wrote: By the by, the tablesample additions to range_table_mutator are obviously broken. > Apparently it's not a good idea to do this at 1AM afte

Re: [HACKERS] checkpointer continuous flushing

2015-06-19 Thread Andres Freund
Hi, On 2015-06-17 08:24:38 +0200, Fabien COELHO wrote: > Here is version 3, including many performance tests with various settings, > representing about 100 hours of pgbench run. This patch aims at improving > checkpoint I/O behavior so that tps throughput is improved, late > transactions are less

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Andres Freund
On 2015-06-19 11:19:23 -0400, Magnus Hagander wrote: > On Fri, Jun 19, 2015 at 11:17 AM, Stephen Frost wrote: > > > * Andres Freund (and...@anarazel.de) wrote: > > > To make slot usage in pg_receivexlog easier, should we add > > > --create-slot-if-not-exists? That'd mean you could run the same com

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Cédric Villemain
> To make slot usage in pg_receivexlog easier, should we add > --create-slot-if-not-exists? That'd mean you could run the same command > the first and later invocation. +1 (with a shorter name please, if you can find one... ) -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ Postgr

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Andres Freund
On 2015-06-19 11:16:18 -0400, Robert Haas wrote: > On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane wrote: > > I wonder whether it's such a good idea for the postmaster to give > > up waiting before all children are gone (postmaster.c:1722 in HEAD). > > I doubt it. Seconded. It's pretty bad to possibl

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Magnus Hagander
On Fri, Jun 19, 2015 at 11:17 AM, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > To make slot usage in pg_receivexlog easier, should we add > > --create-slot-if-not-exists? That'd mean you could run the same command > > the first and later invocation. > > Yes, please. >

Re: [HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Stephen Frost
* Andres Freund (and...@anarazel.de) wrote: > To make slot usage in pg_receivexlog easier, should we add > --create-slot-if-not-exists? That'd mean you could run the same command > the first and later invocation. Yes, please. Thanks! Stephen signature.asc Description: D

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 11:07 AM, Tom Lane wrote: > I wonder whether it's such a good idea for the postmaster to give > up waiting before all children are gone (postmaster.c:1722 in HEAD). I doubt it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -

Re: [HACKERS] Missing tab-complete for PASSWORD word in CREATE ROLE syntax

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 5:52 AM, Jeevan Chalke wrote: > I have observed that we are not tab-completing word PASSWORD in the > following > syntaxes: > > 1. > CREATE|ALTER ROLE|USER rolname > > 2. > CREATE|ALTER ROLE|USER rolname WITH > > PASSWORD is used many times and should be in the tab-complete

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Tom Lane
Michael Paquier writes: > Now if we look at RewindTest.pm, there is the following code: > if ($test_master_datadir) > { > system > "pg_ctl -D $test_master_datadir -s -m immediate stop > 2> /dev/null"; > } > if ($test_standby_datadir

Re: [HACKERS] pg_regress not waiting for postmaster to stop

2015-06-19 Thread Tom Lane
Michael Paquier writes: > Except if I am missing something, the comment mentions that we should > wait for the stop but there is no -w switch in the command of pg_ctl. > Doesn't the patch attached make sense? Per pg_ctl's help output: (The default is to wait for shutdown, but not for start or re

Re: [HACKERS] pg_regress not waiting for postmaster to stop

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 2:33 AM, Michael Paquier wrote: > In pg_regress.c, there is the following code: > static void > stop_postmaster(void) > { > if (postmaster_running) > { > /* We use pg_ctl to issue the kill and wait for stop */ > char

[HACKERS] pg_receivexlog --create-slot-if-not-exists

2015-06-19 Thread Andres Freund
Hi, To make slot usage in pg_receivexlog easier, should we add --create-slot-if-not-exists? That'd mean you could run the same command the first and later invocation. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscri

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Tom Lane
Michael Paquier writes: > Attached is a patch fixing those problems and improving the log > facility as it really helped me out with those issues. The simplest > fix would be to include the -w switch missing in the tests of > pg_rewind and pg_ctl though. I agree with adding the -w switch and will

Re: [HACKERS] Time to get rid of PQnoPasswordSupplied?

2015-06-19 Thread Tom Lane
Robert Haas writes: > On Sun, Jun 14, 2015 at 11:34 PM, Craig Ringer wrote: >> I frequently see users confused by one of our more common and less >> clear error messages: >> >> fe_sendauth: no password supplied > I've never seen this error message, but I'm not opposed to improving > it in some

Re: [HACKERS] Time to get rid of PQnoPasswordSupplied?

2015-06-19 Thread Robert Haas
On Sun, Jun 14, 2015 at 11:34 PM, Craig Ringer wrote: > I frequently see users confused by one of our more common and less > clear error messages: > > fe_sendauth: no password supplied I've never seen this error message, but I'm not opposed to improving it in some way. -- Robert Haas Enterp

Re: [HACKERS] GIN function of pageinspect has inconsistency data type.

2015-06-19 Thread Sawada Masahiko
On Wed, Jun 17, 2015 at 4:11 PM, Jim Nasby wrote: > On 6/16/15 8:26 AM, Sawada Masahiko wrote: >> >> I noticed while using gin function of pageinspect that there are some >> inconsistency data types. >> For example, data type of GinMetaPageData.head, and tail is >> BlockNumber, i.g, uint32. >> Bu

Re: [HACKERS] Is it possible to have a "fast-write" Index?

2015-06-19 Thread Merlin Moncure
On Thu, Jun 11, 2015 at 9:32 PM, Qingqing Zhou wrote: > On Fri, Jun 5, 2015 at 10:59 AM, Tom Lane wrote: >> So I really doubt that anyone would have any enthusiasm for saddling btree >> with a similar mechanism. It's complicated (and has been the cause of >> multiple bugs); it's hard to figure o

Re: [HACKERS] Is it possible to have a "fast-write" Index?

2015-06-19 Thread Simon Riggs
On 19 June 2015 at 14:30, Robert Haas wrote: > > So I really doubt that anyone would have any enthusiasm for saddling > btree > > with a similar mechanism. It's complicated (and has been the cause of > > multiple bugs); it's hard to figure out when is the optimal time to flush > > the pending in

Re: [HACKERS] Is it possible to have a "fast-write" Index?

2015-06-19 Thread Robert Haas
> So I really doubt that anyone would have any enthusiasm for saddling btree > with a similar mechanism. It's complicated (and has been the cause of > multiple bugs); it's hard to figure out when is the optimal time to flush > the pending insertions; and it slows down searches in favor of making >

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 8:18 AM, Robert Haas wrote: > On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier > wrote: >>> Listing the directories with pg_ls_dir() has the same problem. >> >> (After some discussion on IM with Heikki on this one). >> This is actually more tricky because pg_ls_dir() does

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-19 Thread Robert Haas
On Fri, Jun 19, 2015 at 12:14 AM, Michael Paquier wrote: >> Listing the directories with pg_ls_dir() has the same problem. > > (After some discussion on IM with Heikki on this one). > This is actually more tricky because pg_ls_dir() does not return '.' > or '..' that we could use to identify that

Re: [HACKERS] Tab completion for TABLESAMPLE

2015-06-19 Thread Petr Jelinek
On 2015-06-19 09:08, Brendan Jurd wrote: On Sun, 14 Jun 2015 at 20:44 Petr Jelinek mailto:p...@2ndquadrant.com>> wrote: looks like I omitted psql tab completion from the TABLESAMPLE patch. The attached patch adds it. Hi Petr, I'm doing an initial review of this patch. Thanks With

Re: [HACKERS] The real reason why TAP testing isn't ready for prime time

2015-06-19 Thread Michael Paquier
On Thu, Jun 18, 2015 at 3:52 PM, Michael Paquier wrote: > I think that it would be useful as well to improve the buildfarm > output. Thoughts? And after running the tests more or less 6~7 times in a row on a PI, I have been able to trigger the problem and I think that I have found its origin. Firs

[HACKERS] Missing tab-complete for PASSWORD word in CREATE ROLE syntax

2015-06-19 Thread Jeevan Chalke
Hi, I have observed that we are not tab-completing word PASSWORD in the following syntaxes: 1. CREATE|ALTER ROLE|USER rolname 2. CREATE|ALTER ROLE|USER rolname WITH PASSWORD is used many times and should be in the tab-complete list. Was there any reason we have deliberately kept this out? If y

Re: [HACKERS] [Proposal] Progress bar for pg_dump/pg_restore

2015-06-19 Thread Taiki Kondo
Hi, andres Thank you for your comment, and sorry for late response. > The question is how to actually get useful estimates. As there's no > progress report for indvidiual COPY and CREATE INDEX commands you'll, in > many cases, have very irregular progress updates. In many many cases > most of the

Re: [HACKERS] Tab completion for TABLESAMPLE

2015-06-19 Thread Brendan Jurd
On Sun, 14 Jun 2015 at 20:44 Petr Jelinek wrote: > looks like I omitted psql tab completion from the TABLESAMPLE patch. The > attached patch adds it. > Hi Petr, I'm doing an initial review of this patch. It applies and compiles cleanly. Code style is consistent with its surroundings. With th