Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

2017-08-29 Thread Fabien COELHO
Hello, Hmmm. The existing "is_no_vacuum" variable is typed *both* as int (in "main") and as bool (in "init"), called by main (yuk!). I see no reason to choose the bad one for the global:-) Yeah, I think this might be a good timing to re-consider int-for-bool habits in pgbench. If we decided t

Re: [HACKERS] segment size depending *_wal_size defaults (was increasing the default WAL segment size)

2017-08-29 Thread Magnus Hagander
On Wed, Aug 30, 2017 at 6:45 AM, Andres Freund wrote: > On 2017-08-30 12:52:26 +0900, Michael Paquier wrote: > > On Wed, Aug 30, 2017 at 11:20 AM, Peter Eisentraut > > wrote: > > > On 8/29/17 20:36, Andres Freund wrote: > > >> So the question is whether we want {max,min}_wal_size be sized in > >

Re: [HACKERS] segment size depending *_wal_size defaults (was increasing the default WAL segment size)

2017-08-29 Thread Andres Freund
On 2017-08-30 12:52:26 +0900, Michael Paquier wrote: > On Wed, Aug 30, 2017 at 11:20 AM, Peter Eisentraut > wrote: > > On 8/29/17 20:36, Andres Freund wrote: > >> So the question is whether we want {max,min}_wal_size be sized in > >> multiples of segment sizes or as a proper byte size. I'm leanin

Re: [HACKERS] A design for amcheck heapam verification

2017-08-29 Thread Peter Geoghegan
On Tue, Aug 29, 2017 at 7:22 PM, Thomas Munro wrote: > Indeed. Thank you for working on this! To summarise a couple of > ideas that Peter and I discussed off-list a while back: (1) While > building the hash table for a hash join we could build a Bloom filter > per future batch and keep it in me

Re: [HACKERS] segment size depending *_wal_size defaults (was increasing the default WAL segment size)

2017-08-29 Thread Michael Paquier
On Wed, Aug 30, 2017 at 11:20 AM, Peter Eisentraut wrote: > On 8/29/17 20:36, Andres Freund wrote: >> So the question is whether we want {max,min}_wal_size be sized in >> multiples of segment sizes or as a proper byte size. I'm leaning >> towards the latter. > > I'm not sure what the question is

Re: [HACKERS] expanding inheritance in partition bound order

2017-08-29 Thread Robert Haas
On Tue, Aug 29, 2017 at 10:36 PM, Amit Langote wrote: >> I keep having the feeling that this is a big patch with a small patch >> struggling to get out. Is it really necessary to change >> RelationGetPartitionDispatchInfo so much or could you just do a really >> minimal surgery to remove the code

Re: [HACKERS] expanding inheritance in partition bound order

2017-08-29 Thread Amit Langote
On 2017/08/29 4:26, Robert Haas wrote: > I think this patch could be further simplified by continuing to use > the existing function signature for RelationGetPartitionDispatchInfo > instead of changing it to return a List * rather than an array. I > don't see any benefit to such a change. The cur

Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

2017-08-29 Thread Masahiko Sawada
On Tue, Aug 29, 2017 at 4:47 PM, Fabien COELHO wrote: > > Hello, > > Patch applies and works. > >>> I would like to insist a little bit: the name was declared as an int but >>> passed to init and used as a bool there before the patch. Conceptually >>> what >>> is meant is really a bool, and I see

Re: [HACKERS] A design for amcheck heapam verification

2017-08-29 Thread Thomas Munro
On Wed, Aug 30, 2017 at 1:00 PM, Peter Geoghegan wrote: > On Tue, Aug 29, 2017 at 4:34 PM, Thomas Munro > wrote: >> Some drive-by comments on the lib patch: > > I was hoping that you'd look at this, since you'll probably want to > use a bloom filter for parallel hash join at some point. I've trie

Re: [HACKERS] segment size depending *_wal_size defaults (was increasing the default WAL segment size)

2017-08-29 Thread Peter Eisentraut
On 8/29/17 20:36, Andres Freund wrote: > So the question is whether we want {max,min}_wal_size be sized in > multiples of segment sizes or as a proper byte size. I'm leaning > towards the latter. I'm not sure what the question is or what its impact would be. -- Peter Eisentraut htt

Re: [HACKERS] Update low-level backup documentation to match actual behavior

2017-08-29 Thread Michael Paquier
On Tue, Aug 29, 2017 at 10:59 PM, David Steele wrote: > Hi Robert, > > On 8/25/17 4:03 PM, David Steele wrote: >> On 8/25/17 3:26 PM, Robert Haas wrote: >>> On Fri, Aug 25, 2017 at 3:21 PM, David Steele >>> wrote: No problem. I'll base it on your commit to capture any changes you made.

Re: [HACKERS] segment size depending *_wal_size defaults (was increasing the default WAL segment size)

2017-08-29 Thread Andres Freund
On 2017-08-30 10:14:22 +0900, Michael Paquier wrote: > On Wed, Aug 30, 2017 at 10:06 AM, Andres Freund wrote: > > On 2017-08-30 09:49:14 +0900, Michael Paquier wrote: > >> Do you think that we should worry about wal segment sizes higher than > >> 2GB? Support for int64 GUCs is not here yet. > > >

Re: [HACKERS] segment size depending *_wal_size defaults (was increasing the default WAL segment size)

2017-08-29 Thread Michael Paquier
On Wed, Aug 30, 2017 at 10:06 AM, Andres Freund wrote: > On 2017-08-30 09:49:14 +0900, Michael Paquier wrote: >> Do you think that we should worry about wal segment sizes higher than >> 2GB? Support for int64 GUCs is not here yet. > > 1GB will be the limit anyway. Yeah, but imagine that we'd want

Re: [HACKERS] segment size depending *_wal_size defaults (was increasing the default WAL segment size)

2017-08-29 Thread Andres Freund
On 2017-08-30 09:49:14 +0900, Michael Paquier wrote: > On Wed, Aug 30, 2017 at 9:36 AM, Andres Freund wrote: > > So the question is whether we want {max,min}_wal_size be sized in > > multiples of segment sizes or as a proper byte size. I'm leaning > > towards the latter. > > Logically in the cod

Re: [HACKERS] A design for amcheck heapam verification

2017-08-29 Thread Peter Geoghegan
On Tue, Aug 29, 2017 at 4:34 PM, Thomas Munro wrote: > Some drive-by comments on the lib patch: I was hoping that you'd look at this, since you'll probably want to use a bloom filter for parallel hash join at some point. I've tried to keep this one as simple as possible. I think that there is a g

Re: [HACKERS] segment size depending *_wal_size defaults (was increasing the default WAL segment size)

2017-08-29 Thread Michael Paquier
On Wed, Aug 30, 2017 at 9:36 AM, Andres Freund wrote: > So the question is whether we want {max,min}_wal_size be sized in > multiples of segment sizes or as a proper byte size. I'm leaning > towards the latter. Logically in the code it is just a matter of adjusting multipliers. Do you think that

Re: [HACKERS] [PATCH] Fix drop replication slot blocking instead of returning error

2017-08-29 Thread Craig Ringer
On 29 August 2017 at 22:02, Andres Freund wrote: > Hi, > > On 2017-08-29 13:42:05 +0200, Simone Gotti wrote: > > On Tue, Aug 29, 2017 at 12:13 PM, Alvaro Herrera > > wrote: > > > > > > > Hi Alvaro, > > > > > Simone Gotti wrote: > > > > Hi all, > > > > > > > > I noticed that in postgres 10beta3,

[HACKERS] segment size depending *_wal_size defaults (was increasing the default WAL segment size)

2017-08-29 Thread Andres Freund
Hi, intentionally breaking the thread here, I want this one point to get a bit wider audience. The excerpt of the relevant discussion is: On 2017-08-23 12:13:15 +0530, Beena Emerson wrote: > >> + /* set default max_wal_size and min_wal_size */ > >> + snprintf(repltok, sizeof(repltok), "m

Re: [HACKERS] A design for amcheck heapam verification

2017-08-29 Thread Michael Paquier
On Wed, Aug 30, 2017 at 8:34 AM, Thomas Munro wrote: > It'd be nice to replace both with fls() or flsl(), though it's > annoying to have to think about long vs int64 etc. We already use > fls() in two places and supply an implementation in src/port/fls.c for > platforms that lack it (Windows?), b

Re: [HACKERS] error-handling in ReorderBufferCommit() seems somewhat broken

2017-08-29 Thread Tomas Vondra
On 08/30/2017 02:19 AM, Andres Freund wrote: > On 2017-08-30 02:11:10 +0200, Tomas Vondra wrote: >> >> I'm not really following your reasoning. You may very well be right that >> the BeginInternalSubTransaction() example is not quite valid on postgres >> core, but I don't see how that implies the

Re: [HACKERS] error-handling in ReorderBufferCommit() seems somewhat broken

2017-08-29 Thread Andres Freund
On 2017-08-30 02:11:10 +0200, Tomas Vondra wrote: > > > On 08/30/2017 01:34 AM, Andres Freund wrote: > > Hi, > > > > On 2017-08-30 01:27:34 +0200, Tomas Vondra wrote: > >> I've been investigating some failures in test_decoding regression tests, > >> and it seems to me the error-handling in Reord

Re: [HACKERS] Tuple-routing for certain partitioned tables not working as expected

2017-08-29 Thread Amit Langote
On 2017/08/29 20:18, Etsuro Fujita wrote: > On 2017/08/25 22:26, Robert Haas wrote: >> On Wed, Aug 23, 2017 at 4:55 AM, Etsuro Fujita >> wrote: >>> Agreed, but I'd vote for fixing this in v10 as proposed; I agree that just >>> ripping the CheckValidResultRel checks out entirely is not a good idea,

Re: [HACKERS] error-handling in ReorderBufferCommit() seems somewhat broken

2017-08-29 Thread Tomas Vondra
On 08/30/2017 01:34 AM, Andres Freund wrote: > Hi, > > On 2017-08-30 01:27:34 +0200, Tomas Vondra wrote: >> I've been investigating some failures in test_decoding regression tests, >> and it seems to me the error-handling in ReorderBufferCommit() is >> somewhat broken, leading to segfault crashe

Re: [HACKERS] A design for amcheck heapam verification

2017-08-29 Thread Thomas Munro
On Wed, Aug 30, 2017 at 7:58 AM, Peter Geoghegan wrote: > On Thu, May 11, 2017 at 4:30 PM, Peter Geoghegan wrote: >> I spent only a few hours writing a rough prototype, and came up with >> something that does an IndexBuildHeapScan() scan following the >> existing index verification steps. Its amc

Re: [HACKERS] error-handling in ReorderBufferCommit() seems somewhat broken

2017-08-29 Thread Andres Freund
Hi, On 2017-08-30 01:27:34 +0200, Tomas Vondra wrote: > I've been investigating some failures in test_decoding regression tests, > and it seems to me the error-handling in ReorderBufferCommit() is > somewhat broken, leading to segfault crashes. > > The problematic part is this: > > PG_CATCH() >

[HACKERS] error-handling in ReorderBufferCommit() seems somewhat broken

2017-08-29 Thread Tomas Vondra
Hi, I've been investigating some failures in test_decoding regression tests, and it seems to me the error-handling in ReorderBufferCommit() is somewhat broken, leading to segfault crashes. The problematic part is this: PG_CATCH() { /* * Force cache invalidation to happen outside of a va

Re: [HACKERS] increasing the default WAL segment size

2017-08-29 Thread Andres Freund
Hi, On 2017-08-23 12:13:15 +0530, Beena Emerson wrote: > >> + /* > >> + * The calculation of XLOGbuffers requires the run-time > >> parameter > >> + * XLogSegSize which is set from the control file. This > >> value is > >> + * required to creat

[HACKERS] Improve history file reasons when doing promotion

2017-08-29 Thread Michael Paquier
Hi all, When triggering a promotion, the history file generated for the timeline bump provides a reason behind the timeline bump not really helpful: $ cat 0002.history 1 0/3000858 no recovery target specified I was wondering if we could improve that a bit for a promotion. One thing

Re: [HACKERS] Make pg_regress print a connstring with sockdir

2017-08-29 Thread Robert Haas
On Mon, Aug 28, 2017 at 7:57 AM, Craig Ringer wrote: > I'm frequently debugging postmasters that are around long enough. Deadlocks, > etc. > > It's also way easier to debug shmem related issues with a live postmaster vs > a core. Yeah. I don't *frequently* debug postmasters that hang during the

Re: [HACKERS] postgres_fdw bug in 9.6

2017-08-29 Thread Tom Lane
Etsuro Fujita writes: > [ epqpath-for-foreignjoin-11.patch ] I started looking at this. I've not yet wrapped my head around what CreateLocalJoinPath() is doing, but I noted that Robert's concerns about ABI breakage in the back branches would not be very difficult to satisfy. What we'd need to d

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2017-08-29 Thread Robert Haas
On Tue, Jul 4, 2017 at 12:33 AM, Amit Kapila wrote: > I have updated the patch to support wait events and moved it to upcoming CF. This patch doesn't apply any more, but I made it apply with a hammer and then did a little benchmarking (scylla, EDB server, Intel Xeon E5-2695 v3 @ 2.30GHz, 2 socket

Re: [HACKERS] Proposal: Improve bitmap costing for lossy pages

2017-08-29 Thread Robert Haas
On Tue, Aug 29, 2017 at 1:08 AM, Dilip Kumar wrote: > (Time in ms) > Queryhead patch > > 6 2381914571 > 14 1351411183 > 15 49980 32400 > 20204441 188978 These are cool results, but this patch is obviously not ready for p

Re: [HACKERS] Improving overflow checks when adding tuple to PGresult Re: [GENERAL] Retrieving query results

2017-08-29 Thread Michael Paquier
On Wed, Aug 30, 2017 at 4:24 AM, Tom Lane wrote: > Michael Paquier writes: >> On Mon, Aug 28, 2017 at 3:05 PM, Michael Paquier >> wrote: >>> Attached are two patches: >>> 1) 0001 refactors the code around pqAddTuple to be able to handle >>> error messages and assign them in PQsetvalue particular

Re: [HACKERS] A design for amcheck heapam verification

2017-08-29 Thread Peter Geoghegan
On Thu, May 11, 2017 at 4:30 PM, Peter Geoghegan wrote: > I spent only a few hours writing a rough prototype, and came up with > something that does an IndexBuildHeapScan() scan following the > existing index verification steps. Its amcheck callback does an > index_form_tuple() call, hashes the re

Re: [HACKERS] Improving overflow checks when adding tuple to PGresult Re: [GENERAL] Retrieving query results

2017-08-29 Thread Tom Lane
Michael Paquier writes: > On Mon, Aug 28, 2017 at 3:05 PM, Michael Paquier > wrote: >> Attached are two patches: >> 1) 0001 refactors the code around pqAddTuple to be able to handle >> error messages and assign them in PQsetvalue particularly. >> 2) 0002 adds sanity checks in pqAddTuple for overf

Re: [HACKERS] Hash Functions

2017-08-29 Thread Robert Haas
On Tue, Aug 22, 2017 at 8:14 AM, amul sul wrote: > Attaching patch 0002 for the reviewer's testing. I think that this 0002 is not something we can think of committing because there's no guarantee that hash functions will return the same results on all platforms. However, what we could and, I thi

Re: [HACKERS] [BUGS] [postgresql 10 beta3] unrecognized node type: 90

2017-08-29 Thread Tom Lane
Amit Kapila writes: > On Tue, Aug 29, 2017 at 8:32 AM, Robert Haas wrote: >> There's already ExecParallelReinitialize, which could be made to walk >> the nodes in addition to what it does already, but I don't understand >> exactly what else needs fixing. > Sure, but it is not advisable to reset

[HACKERS] OpenFile() Permissions Refactor

2017-08-29 Thread David Steele
Hackers, While working on the patch to allow group reads of $PGDATA I refactored the various OpenFile() functions to use default/global permissions rather than permissions defined in each call. I think the patch stands on its own regardless of whether we accept the patch to allow group permission

Re: [HACKERS] Explicit relation name in VACUUM VERBOSE log

2017-08-29 Thread Robert Haas
On Tue, Aug 22, 2017 at 2:23 AM, Simon Riggs wrote: > Yes, we can. I'm not sure why you would do this only for VACUUM > though? I see many messages in various places that need same treatment I'm skeptical about the idea of doing this too generally. rhaas=> select * from foo; ERROR: permission d

Re: [HACKERS] [PATCH] Fix drop replication slot blocking instead of returning error

2017-08-29 Thread Andres Freund
Hi, On 2017-08-29 13:42:05 +0200, Simone Gotti wrote: > On Tue, Aug 29, 2017 at 12:13 PM, Alvaro Herrera > wrote: > > > > Hi Alvaro, > > > Simone Gotti wrote: > > > Hi all, > > > > > > I noticed that in postgres 10beta3, calling pg_drop_replication_slot on an > > > active slot will block until

Re: [HACKERS] Update low-level backup documentation to match actual behavior

2017-08-29 Thread David Steele
Hi Robert, On 8/25/17 4:03 PM, David Steele wrote: > On 8/25/17 3:26 PM, Robert Haas wrote: >> On Fri, Aug 25, 2017 at 3:21 PM, David Steele >> wrote: >>> No problem.  I'll base it on your commit to capture any changes you >>> made. >> >> Thanks, but you incorporated everything I wanted in respon

Re: [HACKERS] More replication race conditions

2017-08-29 Thread Michael Paquier
On Mon, Aug 28, 2017 at 8:25 AM, Michael Paquier wrote: > Today's run has finished with the same failure: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dangomushi&dt=2017-08-27%2018%3A00%3A13 > Attached is a patch to make this code path wait that the transaction > has been replayed. We

Re: [HACKERS] [PATCH] Fix drop replication slot blocking instead of returning error

2017-08-29 Thread Simone Gotti
On Tue, Aug 29, 2017 at 12:13 PM, Alvaro Herrera wrote: > Hi Alvaro, > Simone Gotti wrote: > > Hi all, > > > > I noticed that in postgres 10beta3, calling pg_drop_replication_slot on an > > active slot will block until it's released instead of returning an error > > like > > done in pg 9.6. Sinc

Re: [HACKERS] Tuple-routing for certain partitioned tables not working as expected

2017-08-29 Thread Etsuro Fujita
On 2017/08/25 22:26, Robert Haas wrote: On Wed, Aug 23, 2017 at 4:55 AM, Etsuro Fujita wrote: Agreed, but I'd vote for fixing this in v10 as proposed; I agree that just ripping the CheckValidResultRel checks out entirely is not a good idea, but that seems OK to me at least as a fix just for v10

Re: [HACKERS] Adding support for Default partition in partitioning

2017-08-29 Thread Ashutosh Bapat
On Mon, Aug 21, 2017 at 4:47 PM, Jeevan Ladhe wrote: > > Hi, > > On Thu, Aug 17, 2017 at 3:29 PM, Jeevan Ladhe > wrote: >> >> Hi, >> >> On Tue, Aug 15, 2017 at 7:20 PM, Robert Haas >> wrote: >>> >>> On Wed, Jul 26, 2017 at 8:14 AM, Jeevan Ladhe >>> wrote: >>> > I have rebased the patches on the

Re: [HACKERS] hash partitioning based on v10Beta2

2017-08-29 Thread yangjie
Hi, When the number of partitions and the data are more, adding new partitions, there will be some efficiency problems. I don't know how the solution you're talking about is

Re: [HACKERS] [POC] hash partitioning

2017-08-29 Thread yangjie
Hi, This is my patch, before I forgot to add attachments, and the following address is also discussed. https://www.postgresql.org/message-id/2017082612390093777512%40highgo.com

Re: [HACKERS] [PATCH] Fix drop replication slot blocking instead of returning error

2017-08-29 Thread Alvaro Herrera
Simone Gotti wrote: > Hi all, > > I noticed that in postgres 10beta3, calling pg_drop_replication_slot on an > active slot will block until it's released instead of returning an error > like > done in pg 9.6. Since this is a change in the previous behavior and the docs > wasn't changed I made a pa

Re: [HACKERS] Re: [PATCH] Fix drop replication slot blocking instead of returning error

2017-08-29 Thread Alvaro Herrera
Noah Misch wrote: > On Thu, Aug 24, 2017 at 03:38:20PM +0200, Simone Gotti wrote: > > I noticed that in postgres 10beta3, calling pg_drop_replication_slot on an > > active slot will block until it's released instead of returning an error > > like > > done in pg 9.6. Since this is a change in the pr

Re: [HACKERS] WIP: Separate log file for extension

2017-08-29 Thread Masahiko Sawada
On Fri, Aug 25, 2017 at 4:12 PM, Antonin Houska wrote: > Attached is a draft patch to allow extension to write log messages to a > separate file. Does it allow postgres core code to write log messages to multiple log files as well? I can imagine a use case where we want to write log messages to s

Re: [HACKERS] POC: Cache data in GetSnapshotData()

2017-08-29 Thread Mithun Cy
On Thu, Aug 3, 2017 at 2:16 AM, Andres Freund wrote: >I think this patch should have a "cover letter" explaining what it's > trying to achieve, how it does so and why it's safe/correct. Cache the SnapshotData for reuse: === In one of our perf analysis using perf tool it sh

Re: [HACKERS] postgres_fdw bug in 9.6

2017-08-29 Thread Etsuro Fujita
On 2017/08/21 20:37, Etsuro Fujita wrote: Attached is an updated version of the patch. I corrected some comments. New version attached. Best regards, Etsuro Fujita *** a/contrib/postgres_fdw/expected/postgres_fdw.out --- b/contrib/postgres_fdw/expected/postgres_fdw.out *** *** 170

Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

2017-08-29 Thread Fabien COELHO
Hello, Patch applies and works. I would like to insist a little bit: the name was declared as an int but passed to init and used as a bool there before the patch. Conceptually what is meant is really a bool, and I see no added value bar saving a few strokes to have an int. ISTM that recent pgb

Re: [HACKERS] [BUGS] [postgresql 10 beta3] unrecognized node type: 90

2017-08-29 Thread Amit Kapila
On Tue, Aug 29, 2017 at 8:32 AM, Robert Haas wrote: > On Mon, Aug 28, 2017 at 10:17 PM, Tom Lane wrote: >> In the meantime, I think what we should do is commit the bug fix more or >> less as I have it, and then work on Amit's concern about losing parallel >> efficiency by separating the resetting

Re: [HACKERS] assorted code cleanup

2017-08-29 Thread Ryan Murphy
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed I've reviewed the code changes, and it's pretty clear to me t

Re: [HACKERS] Hash Functions

2017-08-29 Thread amul sul
On Tue, Aug 22, 2017 at 5:44 PM, amul sul wrote: > On Fri, Aug 18, 2017 at 11:01 PM, Robert Haas > wrote: > >> On Fri, Aug 18, 2017 at 1:12 PM, amul sul wrote: >> > I have a small query, what if I want a cache entry with extended hash >> > function instead standard one, I might require that wh

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-08-29 Thread Masahiko Sawada
On Thu, Aug 24, 2017 at 4:27 PM, Masahiko Sawada wrote: > On Thu, Aug 24, 2017 at 3:11 AM, Josh Berkus wrote: >> On 08/22/2017 11:04 PM, Masahiko Sawada wrote: >>> WARNING: what you did is ok, but you might have wanted to do something else >>> >>> First of all, whether or not that can properly b

Re: [HACKERS] POC: Cache data in GetSnapshotData()

2017-08-29 Thread Mithun Cy
Hi all please ignore this mail, this email is incomplete I have to add more information and Sorry accidentally I pressed the send button while replying. On Tue, Aug 29, 2017 at 11:27 AM, Mithun Cy wrote: > Cache the SnapshotData for reuse: > === > In one of our perf analy