Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 8:24 AM, Michael Paquier wrote: > You can use for example dd in non-truncate mode to corrupt on-disk > page data, say that for example: > dd if=/dev/random bs=8192 count=1 \ > seek=$BLOCK_ID of=base/$DBOID/$RELFILENODE \ > conv=notrunc Sure, but that would probably

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 1:09 PM, Peter Geoghegan wrote: > Or, you could add code like this to comparetup_index_btree(), to > simulate a broken opclass: > > diff --git a/src/backend/utils/sort/tuplesort.c > b/src/backend/utils/sort/tuplesort.c > index 67d86ed..23712ff 100644 > --- a/src/backend/uti

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Andres Freund
On 2016-03-11 23:53:15 +0300, Vladimir Borodin wrote: > It was many times stated in threads about waits monitoring [0, 1, 2] > and supported by different people, but ultimately waits information > was stored in PgBackendStatus. Only that it isn't. It's stored in PGPROC. This criticism is true of

Re: [HACKERS] Relation extension scalability

2016-03-11 Thread Robert Haas
On Thu, Mar 10, 2016 at 8:54 PM, Petr Jelinek wrote: > I am not talking about extension locks, the lock queue can be long because > there is concurrent DDL for example and then once DDL finishes suddenly 100 > connections that tried to insert into table will try to get extension lock > and this wi

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Thu, Mar 10, 2016 at 9:18 AM, Tomas Vondra wrote: > I've looked at this patch today, mostly to educate myself, so this > probably should not count as a full review. Anyway, the patch seems in > excellent shape - it'd be great if all patches (including those written > by me) had this level of co

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-11 Thread Tom Lane
So I started re-reading this thread in preparation for looking at the patch, and this bit in your initial message jumped out at me: > In all of our join algorithms in the executor, if the join type is SEMI, > we skip to the next outer row once we find a matching inner row. This is > because we don

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Joel Jacobson
On Sat, Mar 12, 2016 at 4:08 AM, Robert Haas wrote: > > I don't think my experience in this area is as deep as you seem to > think. I can tell you that most of the requests EnterpriseDB gets for > PL/pgsql enhancements involve wanting it to be more like Oracle's > PL/SQL, which of course has very

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Pavel Stehule
2016-03-11 22:08 GMT+01:00 Robert Haas : > On Fri, Mar 11, 2016 at 3:44 PM, Joel Jacobson wrote: > > On Fri, Mar 11, 2016 at 11:14 AM, Robert Haas > wrote: > >> I'm not direly opposed to most of what's on that page, > >> but I'm not excited about most of it, either. > > > > May I ask, what impro

Re: [HACKERS] Proposal: BSD Authentication support

2016-03-11 Thread Thomas Munro
On Sat, Mar 12, 2016 at 5:14 AM, David Steele wrote: > On 1/14/16 11:22 PM, Robert Haas wrote: >> On Tue, Jan 12, 2016 at 2:27 AM, Marisa Emerson wrote: >>> I've attached the latest version of this patch. I've fixed up an issue with >>> the configuration scripts that I missed. >> Looks reasonable

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Andres Freund
Hi, On 2016-03-11 11:16:32 -0800, Andres Freund wrote: > It seems rather worthwhile to think about how we can expand the coverage > of progress tracking to other types of background processes. WRT the progress reporting patch, I think we should split (as afaics was discussed in the thread for a

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Pavel Stehule
2016-03-11 22:32 GMT+01:00 Joel Jacobson : > On Sat, Mar 12, 2016 at 4:08 AM, Robert Haas > wrote: > > > > I don't think my experience in this area is as deep as you seem to > > think. I can tell you that most of the requests EnterpriseDB gets for > > PL/pgsql enhancements involve wanting it to

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 1:31 PM, Peter Geoghegan wrote: > You could have a race, where > there was a concurrent page deletion of the left sibling of the child > page, then a concurrent insertion into the newly expanded keyspace of > the parent. Therefore, the downlink in the parent (which is the >

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Joel Jacobson
On Sat, Mar 12, 2016 at 4:41 AM, Pavel Stehule wrote: > I afraid so you try to look on your use case as global/generic issue. The > PL/SQL, ADA. PL/pgSQL are verbose languages, and too shortcuts does the > languages dirty. In this point we have different opinion. > > I proposed some enhanced PLpgS

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Joel Jacobson
On Sat, Mar 12, 2016 at 4:48 AM, Joel Jacobson wrote: > neither you nor me have nothing to add. Correction: neither you nor me have anything to add. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pg

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Vladimir Borodin
> 12 марта 2016 г., в 0:22, Andres Freund написал(а): > > On 2016-03-11 23:53:15 +0300, Vladimir Borodin wrote: >> It was many times stated in threads about waits monitoring [0, 1, 2] >> and supported by different people, but ultimately waits information >> was stored in PgBackendStatus. > > On

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Pavel Stehule
2016-03-11 22:48 GMT+01:00 Joel Jacobson : > On Sat, Mar 12, 2016 at 4:41 AM, Pavel Stehule > wrote: > > I afraid so you try to look on your use case as global/generic issue. The > > PL/SQL, ADA. PL/pgSQL are verbose languages, and too shortcuts does the > > languages dirty. In this point we have

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Andres Freund
On 2016-03-12 01:05:43 +0300, Vladimir Borodin wrote: > > 12 марта 2016 г., в 0:22, Andres Freund написал(а): > > Only that it isn't. It's stored in PGPROC. > > Sorry, I missed that. So monitoring of wait events for auxiliary processes > still could be implemented? It's basically a question o

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Joel Jacobson
On Sat, Mar 12, 2016 at 5:09 AM, Pavel Stehule wrote: >> What we need is more input on proposed changes from other companies >> who are also heavy users of PL/pgSQL. >> >> Only then can we move forward. It's like Robert is saying, there is a >> risk for bikeshedding here, >> we must widen our pers

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Pavel Stehule
2016-03-11 23:22 GMT+01:00 Joel Jacobson : > On Sat, Mar 12, 2016 at 5:09 AM, Pavel Stehule > wrote: > >> What we need is more input on proposed changes from other companies > >> who are also heavy users of PL/pgSQL. > >> > >> Only then can we move forward. It's like Robert is saying, there is a

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-11 Thread Tom Lane
I wrote: > I wondered why, instead of inventing an extra semantics-modifying flag, > we couldn't just change the jointype to *be* JOIN_SEMI when we've > discovered that the inner side is unique. BTW, to clarify: I'm not imagining that we'd make this change in the query jointree, as for example pre

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 8:19 AM, Anastasia Lubennikova wrote: > I do hope that my patch will be accepted in 9.6, so this conflict looks > really bad. I hope so too. I'll have to look into this issue. > I think that error is caused by changes in pages layout. To save some space > nonkey attribute

Re: [HACKERS] Relation extension scalability

2016-03-11 Thread Petr Jelinek
On 11/03/16 22:29, Robert Haas wrote: On Thu, Mar 10, 2016 at 8:54 PM, Petr Jelinek wrote: I am not talking about extension locks, the lock queue can be long because there is concurrent DDL for example and then once DDL finishes suddenly 100 connections that tried to insert into table will try

Re: [HACKERS] pl/pgSQL, get diagnostics and big data

2016-03-11 Thread Tom Lane
"Andreas 'ads' Scherbaum" writes: > On 09.02.2016 20:32, Christian Ullrich wrote: >> To fix this, I think it will be enough to change the format strings to >> use "%zu" instead of "%lu". > Attached is a new version of the patch, with %lu replaced by %zu. Nonono ... that just moves the portabilit

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Alexander Korotkov
On Sat, Mar 12, 2016 at 12:22 AM, Andres Freund wrote: > On 2016-03-11 23:53:15 +0300, Vladimir Borodin wrote: > > It was many times stated in threads about waits monitoring [0, 1, 2] > > and supported by different people, but ultimately waits information > > was stored in PgBackendStatus. > > On

Re: [HACKERS] WIP: Upper planner pathification

2016-03-11 Thread Andres Freund
On 2016-03-10 23:38:14 -0500, Tom Lane wrote: > > Would you rather add back the exports or should I? > > I'll do it ... just send me the list. After exporting make_agg, make_limit, make_sort_from_sortclauses and making some trivial adjustments due to the pull_var_clause changes change, Citus' tes

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Jim Nasby
On 3/10/16 8:36 PM, Robert Haas wrote: 1. We make it true only for heavyweight lock waits, and false for other kinds of waits. That's pretty strange. 2. We make it true for all kinds of waits that we now know how to report. That still breaks compatibility. I would absolutely vote for 2 here.

[HACKERS] PREPARE dynamic SQL in plpgsql

2016-03-11 Thread Koichi Suzuki
Hi, Does someone know how to prepare a synamic SQL statement in plpgsql? All the examples, PG documents describe only about preparing static SQL statement. Thank you; -- Koichi Suzuki

Re: [HACKERS] Background Processes and reporting

2016-03-11 Thread Andres Freund
On 2016-03-12 02:24:33 +0300, Alexander Korotkov wrote: > Idea of individual time measurement of every wait event met criticism > because it might have high overhead [1]. Right. And that's actually one of the point which I meant with "didn't listen to criticism". There've been a lot of examples, o

[HACKERS] Perl's newSViv() versus 64-bit ints?

2016-03-11 Thread Tom Lane
Anybody know what will happen when passing a uint64 to newSViv()? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Jim Nasby
On 3/11/16 3:31 PM, Peter Geoghegan wrote: Can we come up with names that more clearly identify the difference >between those two functions? I mean,_parent_ does not make it >particularly obvious that the second function acquires exclusive lock >and performs more thorough checks. Dunno about th

Re: [HACKERS] PREPARE dynamic SQL in plpgsql

2016-03-11 Thread David G. Johnston
On Fri, Mar 11, 2016 at 4:45 PM, Koichi Suzuki wrote: > Hi, > > Does someone know how to prepare a synamic SQL statement in plpgsql? > > All the examples, PG documents describe only about preparing static SQL > statement. > > You might want to rephrase the question. From the pl/pgsql documentati

Re: [HACKERS] Perl's newSViv() versus 64-bit ints?

2016-03-11 Thread Tom Lane
I wrote: > Anybody know what will happen when passing a uint64 to newSViv()? Oh, and how about Python's PyInt_FromLong()? Or PyList_New()? Or the second argument of PyList_SetItem()? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] PREPARE dynamic SQL in plpgsql

2016-03-11 Thread David G. Johnston
On Fri, Mar 11, 2016 at 4:45 PM, Koichi Suzuki wrote: > Hi, > > Does someone know how to prepare a synamic SQL statement in plpgsql? > > All the examples, PG documents describe only about preparing static SQL > statement. > > This is not an appropriate question for the -hackers list. It is bette

Re: [HACKERS] Relation extension scalability

2016-03-11 Thread Jim Nasby
On 3/11/16 5:14 PM, Petr Jelinek wrote: I don't really understand this part about concurrent DDL. If there were concurrent DDL going on, presumably other backends would be blocked on the relation lock, not the relation extension lock - and it doesn't seem likely that you'd often have a huge pile

Re: [HACKERS] 2016-03 Commitfest

2016-03-11 Thread David Steele
We're are now one third of the way through the 2016-03 Commitfest. There are still some patches left that need review but have no reviewer (https://commitfest.postgresql.org/9/?reviewer=-2) though a lot have been picked up in the last week. Needs review: 56 Needs *reviewer*: 15 (was 58 last week)

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 3:50 PM, Jim Nasby wrote: > I also agree that the nmodule name isn't very clear. If this is meant to be > the start of a generic consistency checker, lets call it that. Otherwise, it > should be marked as being specific to btrees, because presumably we might > eventually wa

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Jim Nasby
On 3/11/16 6:17 PM, Peter Geoghegan wrote: Not sure about the cost delay thing. Delays are disabled by default for manually issued VACUUM, so have doubts that that's useful. Right, but you still have the option to enable them if you don't want to swamp your IO system. That's why CIC obeys it t

Re: [HACKERS] Is there a way around function search_path killing SQL function inlining? - and backup / restore issue

2016-03-11 Thread Jim Nasby
On 3/10/16 3:29 PM, Regina Obe wrote: Take for example, I have tiger geocoder which relies on fuzzystrmatch. I have no idea where someone installs fuzzystrmatch so I can't schema qualify those calls. I use that dependent function to use to build an index on tables. This is something I've th

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 4:17 PM, Peter Geoghegan wrote: > If you want the tool to limp on when it finds an error, that can be > done by changing the constant for the CORRUPTION macro in amcheck.c. > But having that be dynamically configurable is not really compatible > with the goal of having amch

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Peter Geoghegan
On Fri, Mar 11, 2016 at 4:30 PM, Jim Nasby wrote: > Right, but you still have the option to enable them if you don't want to > swamp your IO system. That's why CIC obeys it too. If I was running a > consistency check on a production system I'd certainly want the option to > throttle it. Without th

Re: [HACKERS] Perl's newSViv() versus 64-bit ints?

2016-03-11 Thread Andrew Dunstan
On 03/11/2016 06:49 PM, Tom Lane wrote: Anybody know what will happen when passing a uint64 to newSViv()? A perl IV is guaranteed large enough to hold a pointer, if that's any help. But for an unsigned value you might be better off calling newSVuv() cheers andre

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Tomas Vondra
On Fri, 2016-03-11 at 16:40 -0800, Peter Geoghegan wrote: > On Fri, Mar 11, 2016 at 4:17 PM, Peter Geoghegan > wrote: > > > > If you want the tool to limp on when it finds an error, that can be > > done by changing the constant for the CORRUPTION macro in > > amcheck.c. > > But having that be dy

Re: [HACKERS] Relation extension scalability

2016-03-11 Thread Dilip Kumar
On Sat, Mar 12, 2016 at 5:31 AM, Jim Nasby wrote: > FWIW, this is definitely a real possibility in any shop that has very high > downtime costs and high transaction rates. > > I also think some kind of clamp is a good idea. It's not that uncommon to > run max_connections significantly higher than

Re: [HACKERS] Proposal: BSD Authentication support

2016-03-11 Thread Peter Eisentraut
On 1/7/16 9:40 PM, Marisa Emerson wrote: > There's a port for PAM, but we would prefer to use BSD Auth as its quite > a lot cleaner and is standard on OpenBSD. > > I've attached an updated patch that includes documentation. It has been > tested against OpenBSD 5.8. I'll add this thread to the comm

Re: [HACKERS] Proposal: BSD Authentication support

2016-03-11 Thread Peter Eisentraut
On 3/11/16 4:38 PM, Thomas Munro wrote: > It looks like this needs review from an OpenBSD user specifically. > FreeBSD and NetBSD use PAM instead of BSD auth. FreeBSD has man pages for this stuff, so maybe they also have it now. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.

Re: [HACKERS] Relation extension scalability

2016-03-11 Thread Amit Kapila
On Sat, Mar 12, 2016 at 8:16 AM, Dilip Kumar wrote: > > > On Sat, Mar 12, 2016 at 5:31 AM, Jim Nasby wrote: >> >> FWIW, this is definitely a real possibility in any shop that has very high downtime costs and high transaction rates. >> >> I also think some kind of clamp is a good idea. It's not th

[HACKERS] Re: [COMMITTERS] pgsql: Only try to push down foreign joins if the user mapping OIDs mat

2016-03-11 Thread Andres Freund
Hi, On 2016-01-28 19:09:01 +, Robert Haas wrote: > Only try to push down foreign joins if the user mapping OIDs match. > > Previously, the foreign join pushdown infrastructure left the question > of security entirely up to individual FDWs, but it would be easy for > a foreign data wrapper to

Re: [HACKERS] pl/pgSQL, get diagnostics and big data

2016-03-11 Thread Tom Lane
I wrote: > I'll take it from here, unless I find bigger issues. Hmm ... so the more I pulled on this string, the more stuff I found. The attached updated patch fixes several additional significant areas: * call_cntr and max_calls in FuncCallContext are now uint64 * Result widths for PortalRunFet

Re: [HACKERS] Parallel Aggregate

2016-03-11 Thread David Rowley
On 11 March 2016 at 03:39, David Rowley wrote: > A couple of things which I'm not 100% happy with. > > 1. make_partialgroup_input_target() doing lookups to the syscache. > Perhaps this job can be offloaded to a new function in a more suitable > location. Ideally the Aggref would already store the

Re: [HACKERS] Relation extension scalability

2016-03-11 Thread Petr Jelinek
On 12/03/16 01:01, Jim Nasby wrote: On 3/11/16 5:14 PM, Petr Jelinek wrote: I don't really understand this part about concurrent DDL. If there were concurrent DDL going on, presumably other backends would be blocked on the relation lock, not the relation extension lock - and it doesn't seem lik

Re: [HACKERS] Relation extension scalability

2016-03-11 Thread Petr Jelinek
On 12/03/16 03:46, Dilip Kumar wrote: On Sat, Mar 12, 2016 at 5:31 AM, Jim Nasby mailto:jim.na...@bluetreble.com>> wrote: FWIW, this is definitely a real possibility in any shop that has very high downtime costs and high transaction rates. I also think some kind of clamp is a good

Re: [HACKERS] pl/pgSQL, get diagnostics and big data

2016-03-11 Thread Petr Jelinek
On 12/03/16 04:30, Tom Lane wrote: 1. I found two places (marked XXX in this patch) that are using strtoul() to parse a tuple count back out of a command tag. That won't do anymore. pg_stat_statements has a messy hack for the same problem (look for HAVE_STRTOULL), which is probably what we want

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Joel Jacobson
On Sat, Mar 12, 2016 at 6:31 AM, Jim Nasby wrote: > On 3/10/16 8:36 PM, Robert Haas wrote: >> >> 1. We make it true only for heavyweight lock waits, and false for >> other kinds of waits. That's pretty strange. >> 2. We make it true for all kinds of waits that we now know how to >> report. That

Re: [HACKERS] Relation extension scalability

2016-03-11 Thread Dilip Kumar
On Sat, Mar 12, 2016 at 8:37 AM, Amit Kapila wrote: > Can you post the numbers for 1, 5, 10, 15, 25 or whatever other multiplier > you have tried, so that it is clear that 20 is best? I had Tried with 1, 10, 20 and 50. 1. With base code it was almost the same as base code. 2. With 10 thread d

Re: [HACKERS] raw output from copy

2016-03-11 Thread Pavel Stehule
Hi 2016-03-09 18:41 GMT+01:00 Corey Huinker : > >>> The regression tests seem to adequately cover all new functionality, >>> though I wonder if we should add some cases that highlight situations where >>> BINARY mode is insufficient. >>> >>> > One thing I tried to test RAW was to load an existing

Re: [HACKERS] Explain [Analyze] produces parallel scan for select Into table statements.

2016-03-11 Thread Amit Kapila
On Fri, Mar 11, 2016 at 3:34 PM, Mithun Cy wrote: > > On Thu, Mar 10, 2016 at 9:39 PM, Robert Haas wrote: > >I guess there must not be an occurrence of this pattern in the > >regression tests, or previous force_parallel_mode testing would have > >found this problem. Perhaps this patch should add

[HACKERS] [PATCH] Use MemoryContextAlloc() in the MemoryContextAllocZero() and MemoryContextAllocZeroAligned()

2016-03-11 Thread Alexander Kuleshov
Hello all, Attached patch simplifies the MemoryContextAllocZero() and MemoryContextAllocZeroAligned(). The MemoryContextAllocZero() and MemoryContextAllocZeroAligned() functions does almost the same that MemoryContextAlloc() does. Additionally these functions fills allocated memory context with ze

Re: [HACKERS] Perl's newSViv() versus 64-bit ints?

2016-03-11 Thread Salvador Fandiño
On 03/12/2016 12:49 AM, Tom Lane wrote: Anybody know what will happen when passing a uint64 to newSViv()? On 64 bit platforms, it is just interpreted as a signed integer, any number with the upper bit set will become negative. Perl provides newSVuv for unsigned numbers. On 32bit platforms a

Re: [HACKERS] [COMMITTERS] pgsql: Provide much better wait information in pg_stat_activity.

2016-03-11 Thread Amit Kapila
On Sat, Mar 12, 2016 at 5:01 AM, Jim Nasby wrote: > > On 3/10/16 8:36 PM, Robert Haas wrote: >> >> 1. We make it true only for heavyweight lock waits, and false for >> other kinds of waits. That's pretty strange. >> 2. We make it true for all kinds of waits that we now know how to >> report. Tha

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread Konstantin Knizhnik
On 03/11/2016 11:35 PM, Tom Lane wrote: Robert Haas writes: On Fri, Mar 11, 2016 at 1:11 PM, David Steele wrote: Is anyone willing to volunteer a review or make an argument for the importance of this patch? There's been a lot of discussion on another thread about this patch. The subject is "

<    1   2