[COMMITTERS] pgsql: PL/Python: Clean up extended error reporting docs and tests

2016-06-15 Thread Peter Eisentraut
PL/Python: Clean up extended error reporting docs and tests Format the example and test code more to Python style standards. Improve whitespace. Improve documentation formatting. Branch -- master Details --- http://git.postgresql.org/pg/commitdiff/f0688d6e6c595cdceef3ad218b86f064f4909b4

[COMMITTERS] pgsql: Force idle_in_transaction_session_timeout off in pg_dump and aut

2016-06-15 Thread Tom Lane
Force idle_in_transaction_session_timeout off in pg_dump and autovacuum. We disable statement_timeout and lock_timeout during dump and restore, to prevent any global settings that might exist from breaking routine backups. Commit c6dda1f48 should have added idle_in_transaction_session_timeout to t

[COMMITTERS] pgsql: Mark some functions parallel-unsafe.

2016-06-15 Thread Robert Haas
Mark some functions parallel-unsafe. currtid() and currtid2() call GetLatestSnapshot(), which fails in parallel mode. pg_export_snapshot() calls ExportSnapshot() which attempts to assign an XID for the current transaction if it does not already have one; that, too, will fail in parallel mode. An

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Robert Haas
On Sat, Jun 11, 2016 at 11:29 AM, Kevin Grittner wrote: > I have reviewed the code and run tests to try to find something > here which could be considered a bug, without finding any problem. > When reading pages for the random sample for ANALYZE (or > auto-analyze) there is not an age check; so AN

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 10:46 AM, Robert Haas wrote: > On Sat, Jun 11, 2016 at 11:29 AM, Kevin Grittner wrote: >> I have reviewed the code and run tests to try to find something >> here which could be considered a bug, without finding any problem. >> When reading pages for the random sample for A

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Robert Haas
On Wed, Jun 15, 2016 at 1:44 PM, Kevin Grittner wrote:>> So what happens in this scenario: >> 1. ANALYZE runs really slowly - maybe the user-defined function it's >> running for the expression index is extremely long-running. >> 2. Eventually, the snapshot for ANALYZE is older than the configured

[COMMITTERS] pgsql: Fix lazy_scan_heap so that it won't mark pages all-frozen too so

2016-06-15 Thread Robert Haas
Fix lazy_scan_heap so that it won't mark pages all-frozen too soon. Commit a892234f830e832110f63fc0a2afce2fb21d1584 added a new bit per page to the visibility map fork indicating whether the page is all-frozen, but incorrectly assumed that if lazy_scan_heap chose to freeze a tuple then that tuple

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 1:29 PM, Robert Haas wrote: > On Wed, Jun 15, 2016 at 1:44 PM, Kevin Grittner > wrote:>> So what happens in this scenario: >>> 1. ANALYZE runs really slowly - maybe the user-defined function it's >>> running for the expression index is extremely long-running. >>> 2. Eventu

[COMMITTERS] pgsql: Add integrity-checking functions to pg_visibility.

2016-06-15 Thread Robert Haas
Add integrity-checking functions to pg_visibility. The new pg_check_visible() and pg_check_frozen() functions can be used to verify that the visibility map bits for a relation's data pages match the actual state of the tuples on those pages. Amit Kapila and Robert Haas, reviewed (in earlier versi

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Alvaro Herrera
Robert Haas wrote: > On Wed, Jun 15, 2016 at 1:44 PM, Kevin Grittner > wrote: > > The test I showed creates a situation which (to ANALYZE) is > > identical to what you describe -- ANALYZE sees a page with an LSN > > recent enough that it could have been (and actually has been) > > pruned. Why wo

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 1:45 PM, Alvaro Herrera wrote: > Maybe it is possible to get into trouble if you're taking a sample for > an expression index. Maybe -- if you are using a function for an index expression which does an explicit SELECT against some database table rather than only using val

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Robert Haas
On Wed, Jun 15, 2016 at 2:45 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> On Wed, Jun 15, 2016 at 1:44 PM, Kevin Grittner >> wrote: >> > The test I showed creates a situation which (to ANALYZE) is >> > identical to what you describe -- ANALYZE sees a page with an LSN >> > recent enough that

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Alvaro Herrera
Robert Haas wrote: > On Wed, Jun 15, 2016 at 2:45 PM, Alvaro Herrera > wrote: > > Maybe it is possible to get into trouble if you're taking a sample for > > an expression index. > > The expression index case is the one to worry about; if there is a > problem, that's where it is. What bothers me

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 1:50 PM, Robert Haas wrote: > The expression index case is the one to worry about; if there is a > problem, that's where it is. What bothers me is that a function used > in an expression index could do anything at all - it can read any > table in the database. It *can*,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Alvaro Herrera
Kevin Grittner wrote: > On Wed, Jun 15, 2016 at 1:50 PM, Robert Haas wrote: > > > The expression index case is the one to worry about; if there is a > > problem, that's where it is. What bothers me is that a function used > > in an expression index could do anything at all - it can read any > >

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 1:54 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> The expression index case is the one to worry about; if there is a >> problem, that's where it is. What bothers me is that a function used >> in an expression index could do anything at all - it can read any >> table

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 1:59 PM, Alvaro Herrera wrote: > Kevin Grittner wrote: >> On Wed, Jun 15, 2016 at 1:50 PM, Robert Haas wrote: >> >>> The expression index case is the one to worry about; if there is a >>> problem, that's where it is. What bothers me is that a function used >>> in an expre

[COMMITTERS] pgsql: Fix regression test for force_parallel_mode=on.

2016-06-15 Thread Robert Haas
Fix regression test for force_parallel_mode=on. Commit 14a254fb52423c57059851abafbd1247261f7f03 managed not to exercise the code it was intended to test, and the comment explaining why no "parallel worker" line showed up in the context wasn't right. Amit Kapila, tweaked by me per Amit's analysis.

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Andres Freund
On 2016-06-15 14:50:46 -0400, Robert Haas wrote: > On Wed, Jun 15, 2016 at 2:45 PM, Alvaro Herrera > wrote: > > Robert Haas wrote: > >> On Wed, Jun 15, 2016 at 1:44 PM, Kevin Grittner > >> wrote: > >> > The test I showed creates a situation which (to ANALYZE) is > >> > identical to what you descr

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 2:20 PM, Andres Freund wrote: > We might fetch a toast tuple which > since have been re-purposed for a datum of a different type. How would that happen? -- Kevin Grittner EDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-committers m

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Alvaro Herrera
Kevin Grittner wrote: > On Wed, Jun 15, 2016 at 1:59 PM, Alvaro Herrera > wrote: > > We actually go quite some lengths to support this case, even when it's > > the opinion of many that we shouldn't. For example VACUUM doesn't try > > to find index entries using the values in each deleted tuple;

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Robert Haas
On Wed, Jun 15, 2016 at 2:59 PM, Alvaro Herrera wrote: > Kevin Grittner wrote: >> On Wed, Jun 15, 2016 at 1:50 PM, Robert Haas wrote: >> >> > The expression index case is the one to worry about; if there is a >> > problem, that's where it is. What bothers me is that a function used >> > in an ex

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread David G. Johnston
On Wed, Jun 15, 2016 at 3:40 PM, Alvaro Herrera wrote: > Kevin Grittner wrote: > > On Wed, Jun 15, 2016 at 1:59 PM, Alvaro Herrera > > wrote: > > > > We actually go quite some lengths to support this case, even when it's > > > the opinion of many that we shouldn't. For example VACUUM doesn't tr

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Andres Freund
On 2016-06-15 14:24:58 -0500, Kevin Grittner wrote: > On Wed, Jun 15, 2016 at 2:20 PM, Andres Freund wrote: > > > We might fetch a toast tuple which > > since have been re-purposed for a datum of a different type. > > How would that happen? Autovac vacuums toast and heap tables independently. O

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 3:25 PM, Andres Freund wrote: > On 2016-06-15 14:24:58 -0500, Kevin Grittner wrote: >> On Wed, Jun 15, 2016 at 2:20 PM, Andres Freund wrote: >> >> > We might fetch a toast tuple which >> > since have been re-purposed for a datum of a different type. >> >> How would that ha

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 2:40 PM, Alvaro Herrera wrote: > Kevin Grittner wrote: >> On Wed, Jun 15, 2016 at 1:59 PM, Alvaro Herrera >> wrote: > >> > We actually go quite some lengths to support this case, even when it's >> > the opinion of many that we shouldn't. For example VACUUM doesn't try >>

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Andres Freund
On 2016-06-15 16:58:25 -0500, Kevin Grittner wrote: > On Wed, Jun 15, 2016 at 3:25 PM, Andres Freund wrote: > > On 2016-06-15 14:24:58 -0500, Kevin Grittner wrote: > >> On Wed, Jun 15, 2016 at 2:20 PM, Andres Freund wrote: > >> > >> > We might fetch a toast tuple which > >> > since have been re-p

[COMMITTERS] pgsql: Use strftime("%c") to format timestamps in psql's \watch command

2016-06-15 Thread Tom Lane
Use strftime("%c") to format timestamps in psql's \watch command. This allows the timestamps to follow local conventions (in particular, they respond to the LC_TIME environment setting). In C locale you get the same results as before. It seems like a good idea to do this now not later because we

[COMMITTERS] pgsql: Widen buffer for headers in psql's \watch command.

2016-06-15 Thread Tom Lane
Widen buffer for headers in psql's \watch command. This is to make sure there's enough room for translated versions of the message. HEAD's already addressed this issue, but back-patch a simple increase in the array size. Discussion: <[email protected]> Branch -- REL9_5_ST

[COMMITTERS] pgsql: Widen buffer for headers in psql's \watch command.

2016-06-15 Thread Tom Lane
Widen buffer for headers in psql's \watch command. This is to make sure there's enough room for translated versions of the message. HEAD's already addressed this issue, but back-patch a simple increase in the array size. Discussion: <[email protected]> Branch -- REL9_4_ST

[COMMITTERS] pgsql: Widen buffer for headers in psql's \watch command.

2016-06-15 Thread Tom Lane
Widen buffer for headers in psql's \watch command. This is to make sure there's enough room for translated versions of the message. HEAD's already addressed this issue, but back-patch a simple increase in the array size. Discussion: <[email protected]> Branch -- REL9_3_ST

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Kevin Grittner
On Wed, Jun 15, 2016 at 5:34 PM, Andres Freund wrote: > On 2016-06-15 16:58:25 -0500, Kevin Grittner wrote: >> On Wed, Jun 15, 2016 at 3:25 PM, Andres Freund wrote: >>> On 2016-06-15 14:24:58 -0500, Kevin Grittner wrote: On Wed, Jun 15, 2016 at 2:20 PM, Andres Freund wrote: > We mi

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Andres Freund
On 2016-06-15 20:22:24 -0500, Kevin Grittner wrote: > I'm not clear where you see this as being in any way different with > STO. Above it seemed that you saw this as an issue related to > ANALYZE. If there is not early pruning for the table being > analyzed, nothing is at all different. If there

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

2016-06-15 Thread Alvaro Herrera
Kevin Grittner wrote: > test=# -- connection 1 > analyze verbose t1; -- when run after threshold, STO error occurs > INFO: analyzing "public.t1" > INFO: "t1": scanned 45 of 45 pages, containing 8999 live rows and > 1001 dead rows; 8999 rows in sample, 8999 estimated total rows > ERROR: snapsho