[COMMITTERS] pgsql: Update back-branch release notes for the last few commits.

[Tom Lane]

Update back-branch release notes for the last few commits.

OpenSSL error queue fix no longer needs to be documented under 9.6.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/7dc1d359699345a2b2af6af6d21bd6f7bd6cfb27

Modified Files
--
doc/src/sgml/release-9.5.sgml | 48 +++
doc/src/sgml/release-9.6.sgml | 20 --
2 files changed, 48 insertions(+), 20 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Tom Lane]

Peter Eisentraut  writes:
> Again, I applaud this, but this is clearly a new major chunk of mostly 
> not-peer-reviewed code with potential for portability problems and the 
> possibility to break downstream packaging routines.  There is no harm in 
> leaving this for 9.7.

FWIW, I was and remain perfectly on board with putting this into 9.6;
adding new tests seems to me to be very much within the charter of beta
test period.  My only gripe was that shoving it in so soon before beta1
wrap is risky.  But it looks from the current buildfarm results that
Stephen has gotten away with that.  If we see any more buildfarm failures
over the weekend, I'd counsel a temporary revert, but if we don't I think
it's OK to ship beta1 with these tests.  In either case I'd aim to have
them in place in beta2.

regards, tom lane


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Distrust external OpenSSL clients; clear err queue

[Peter Eisentraut]

Distrust external OpenSSL clients; clear err queue

OpenSSL has an unfortunate tendency to mix per-session state error
handling with per-thread error handling.  This can cause problems when
programs that link to libpq with OpenSSL enabled have some other use of
OpenSSL; without care, one caller of OpenSSL may cause problems for the
other caller.  Backend code might similarly be affected, for example
when a third party extension independently uses OpenSSL without taking
the appropriate precautions.

To fix, don't trust other users of OpenSSL to clear the per-thread error
queue.  Instead, clear the entire per-thread queue ahead of certain I/O
operations when it appears that there might be trouble (these I/O
operations mostly need to call SSL_get_error() to check for success,
which relies on the queue being empty).  This is slightly aggressive,
but it's pretty clear that the other callers have a very dubious claim
to ownership of the per-thread queue.  Do this is both frontend and
backend code.

Finally, be more careful about clearing our own error queue, so as to
not cause these problems ourself.  It's possibly that control previously
did not always reach SSLerrmessage(), where ERR_get_error() was supposed
to be called to clear the queue's earliest code.  Make sure
ERR_get_error() is always called, so as to spare other users of OpenSSL
the possibility of similar problems caused by libpq (as opposed to
problems caused by a third party OpenSSL library like PHP's OpenSSL
extension).  Again, do this is both frontend and backend code.

See bug #12799 and https://bugs.php.net/bug.php?id=68276

Based on patches by Dave Vitek and Peter Eisentraut.

From: Peter Geoghegan 

Branch
--
REL9_4_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/e3b14de9f0453d29c1d77b081d312b2eac6b192a

Modified Files
--
src/backend/libpq/be-secure.c| 68 ---
src/interfaces/libpq/fe-secure.c | 78 +++-
2 files changed, 102 insertions(+), 44 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Distrust external OpenSSL clients; clear err queue

[Peter Eisentraut]

Distrust external OpenSSL clients; clear err queue

OpenSSL has an unfortunate tendency to mix per-session state error
handling with per-thread error handling.  This can cause problems when
programs that link to libpq with OpenSSL enabled have some other use of
OpenSSL; without care, one caller of OpenSSL may cause problems for the
other caller.  Backend code might similarly be affected, for example
when a third party extension independently uses OpenSSL without taking
the appropriate precautions.

To fix, don't trust other users of OpenSSL to clear the per-thread error
queue.  Instead, clear the entire per-thread queue ahead of certain I/O
operations when it appears that there might be trouble (these I/O
operations mostly need to call SSL_get_error() to check for success,
which relies on the queue being empty).  This is slightly aggressive,
but it's pretty clear that the other callers have a very dubious claim
to ownership of the per-thread queue.  Do this is both frontend and
backend code.

Finally, be more careful about clearing our own error queue, so as to
not cause these problems ourself.  It's possibly that control previously
did not always reach SSLerrmessage(), where ERR_get_error() was supposed
to be called to clear the queue's earliest code.  Make sure
ERR_get_error() is always called, so as to spare other users of OpenSSL
the possibility of similar problems caused by libpq (as opposed to
problems caused by a third party OpenSSL library like PHP's OpenSSL
extension).  Again, do this is both frontend and backend code.

See bug #12799 and https://bugs.php.net/bug.php?id=68276

Based on patches by Dave Vitek and Peter Eisentraut.

From: Peter Geoghegan 

Branch
--
REL9_2_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/e3a493ac08b7b0fb9de85a002aeff69e4f8fba43

Modified Files
--
src/backend/libpq/be-secure.c| 68 ---
src/interfaces/libpq/fe-secure.c | 78 +++-
2 files changed, 102 insertions(+), 44 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Distrust external OpenSSL clients; clear err queue

[Peter Eisentraut]

Distrust external OpenSSL clients; clear err queue

OpenSSL has an unfortunate tendency to mix per-session state error
handling with per-thread error handling.  This can cause problems when
programs that link to libpq with OpenSSL enabled have some other use of
OpenSSL; without care, one caller of OpenSSL may cause problems for the
other caller.  Backend code might similarly be affected, for example
when a third party extension independently uses OpenSSL without taking
the appropriate precautions.

To fix, don't trust other users of OpenSSL to clear the per-thread error
queue.  Instead, clear the entire per-thread queue ahead of certain I/O
operations when it appears that there might be trouble (these I/O
operations mostly need to call SSL_get_error() to check for success,
which relies on the queue being empty).  This is slightly aggressive,
but it's pretty clear that the other callers have a very dubious claim
to ownership of the per-thread queue.  Do this is both frontend and
backend code.

Finally, be more careful about clearing our own error queue, so as to
not cause these problems ourself.  It's possibly that control previously
did not always reach SSLerrmessage(), where ERR_get_error() was supposed
to be called to clear the queue's earliest code.  Make sure
ERR_get_error() is always called, so as to spare other users of OpenSSL
the possibility of similar problems caused by libpq (as opposed to
problems caused by a third party OpenSSL library like PHP's OpenSSL
extension).  Again, do this is both frontend and backend code.

See bug #12799 and https://bugs.php.net/bug.php?id=68276

Based on patches by Dave Vitek and Peter Eisentraut.

From: Peter Geoghegan 

Branch
--
REL9_1_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/9b676fd49a34e1e4001f19331e9cb4a8d688bb06

Modified Files
--
src/backend/libpq/be-secure.c| 69 ---
src/interfaces/libpq/fe-secure.c | 78 +++-
2 files changed, 103 insertions(+), 44 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Distrust external OpenSSL clients; clear err queue

[Peter Eisentraut]

Distrust external OpenSSL clients; clear err queue

OpenSSL has an unfortunate tendency to mix per-session state error
handling with per-thread error handling.  This can cause problems when
programs that link to libpq with OpenSSL enabled have some other use of
OpenSSL; without care, one caller of OpenSSL may cause problems for the
other caller.  Backend code might similarly be affected, for example
when a third party extension independently uses OpenSSL without taking
the appropriate precautions.

To fix, don't trust other users of OpenSSL to clear the per-thread error
queue.  Instead, clear the entire per-thread queue ahead of certain I/O
operations when it appears that there might be trouble (these I/O
operations mostly need to call SSL_get_error() to check for success,
which relies on the queue being empty).  This is slightly aggressive,
but it's pretty clear that the other callers have a very dubious claim
to ownership of the per-thread queue.  Do this is both frontend and
backend code.

Finally, be more careful about clearing our own error queue, so as to
not cause these problems ourself.  It's possibly that control previously
did not always reach SSLerrmessage(), where ERR_get_error() was supposed
to be called to clear the queue's earliest code.  Make sure
ERR_get_error() is always called, so as to spare other users of OpenSSL
the possibility of similar problems caused by libpq (as opposed to
problems caused by a third party OpenSSL library like PHP's OpenSSL
extension).  Again, do this is both frontend and backend code.

See bug #12799 and https://bugs.php.net/bug.php?id=68276

Based on patches by Dave Vitek and Peter Eisentraut.

From: Peter Geoghegan 

Branch
--
REL9_3_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/a9d8644de09e9b27ea9eb9148d2476b0d6c9eb90

Modified Files
--
src/backend/libpq/be-secure.c| 68 ---
src/interfaces/libpq/fe-secure.c | 78 +++-
2 files changed, 102 insertions(+), 44 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Distrust external OpenSSL clients; clear err queue

[Peter Eisentraut]

Distrust external OpenSSL clients; clear err queue

OpenSSL has an unfortunate tendency to mix per-session state error
handling with per-thread error handling.  This can cause problems when
programs that link to libpq with OpenSSL enabled have some other use of
OpenSSL; without care, one caller of OpenSSL may cause problems for the
other caller.  Backend code might similarly be affected, for example
when a third party extension independently uses OpenSSL without taking
the appropriate precautions.

To fix, don't trust other users of OpenSSL to clear the per-thread error
queue.  Instead, clear the entire per-thread queue ahead of certain I/O
operations when it appears that there might be trouble (these I/O
operations mostly need to call SSL_get_error() to check for success,
which relies on the queue being empty).  This is slightly aggressive,
but it's pretty clear that the other callers have a very dubious claim
to ownership of the per-thread queue.  Do this is both frontend and
backend code.

Finally, be more careful about clearing our own error queue, so as to
not cause these problems ourself.  It's possibly that control previously
did not always reach SSLerrmessage(), where ERR_get_error() was supposed
to be called to clear the queue's earliest code.  Make sure
ERR_get_error() is always called, so as to spare other users of OpenSSL
the possibility of similar problems caused by libpq (as opposed to
problems caused by a third party OpenSSL library like PHP's OpenSSL
extension).  Again, do this is both frontend and backend code.

See bug #12799 and https://bugs.php.net/bug.php?id=68276

Based on patches by Dave Vitek and Peter Eisentraut.

From: Peter Geoghegan 

Branch
--
REL9_5_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/a3c17b2af89cd46b47df3483bb693312d7521795

Modified Files
--
src/backend/libpq/be-secure-openssl.c| 70 -
src/interfaces/libpq/fe-secure-openssl.c | 77 ++--
2 files changed, 102 insertions(+), 45 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix SSL tests

[Peter Eisentraut]

Fix SSL tests

These were accidentally broken by the great backpatching of
331828b754378733cb5c2e49227603e7354e4e39.

Branch
--
REL9_5_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/ab32a4057229dc59ee83614b1fcc7fe242213a78

Modified Files
--
src/test/ssl/ServerSetup.pm| 2 ++
src/test/ssl/t/001_ssltests.pl | 2 +-
2 files changed, 3 insertions(+), 1 deletion(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Peter Eisentraut]

On 5/6/16 3:11 PM, Stephen Frost wrote:

These are just new tests..?


This is a matter of degree, but I think there is a difference between 
new test cases and a whole new test suite.



I assumed that would be welcome during post
feature-freeze, and certainly no one raised any concerns about adding
these tests during the discussion prior to my commiting them.


I didn't see that discussion and I still can't find it.

--
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Peter Eisentraut]

On 5/6/16 4:07 PM, Stephen Frost wrote:

Are you suggesting commiting to still-9.6-HEAD post-beta1?  I took
Peter's comment as suggesting that adding the tests would have to wait
til after we branched 9.6, as we do for features.

I'd really like to have these tests included as that will make them
available to others more easily to add on to, and I'm certainly planning
to continue adding tests until I get pg_dump.c's coverage a lot better.
That seems like the perfect kind of effort that should be happening
right now- adding more tests and working to make sure that what's been
committed is correct (and fixing it when it isn't, as discovered by the
test suite with transforms and casts...).


Well, we all want to get our code in and let others make it better.  But 
here we are trying to get a release out in a few days with only a 
weekend in between, and we are still trying to figure out basic 
portability issues using build-farm-whacking technology.  This is not 
the time.


Again, I applaud this, but this is clearly a new major chunk of mostly 
not-peer-reviewed code with potential for portability problems and the 
possibility to break downstream packaging routines.  There is no harm in 
leaving this for 9.7.


--
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Clean up after pg_dump test runs.

[Tom Lane]

Clean up after pg_dump test runs.

The tmp_check directory needs to be removed by "make clean",
and also ignored by .gitignore.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/74a73b17225385e54dbf9fc2f77aaa59191ac04b

Modified Files
--
src/bin/pg_dump/.gitignore | 2 ++
src/bin/pg_dump/Makefile   | 1 +
2 files changed, 3 insertions(+)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix pg_upgrade to not fail when new-cluster TOAST rules differ f

[Tom Lane]

Fix pg_upgrade to not fail when new-cluster TOAST rules differ from old.

This patch essentially reverts commit 4c6780fd17aa43ed, in favor of a much
simpler solution for the case where the new cluster would choose to create
a TOAST table but the old cluster doesn't have one: just don't create a
TOAST table.

The existing code failed in at least two different ways if the situation
arose: (1) ALTER TABLE RESET didn't grab an exclusive lock, so that the
lock sanity check in create_toast_table failed; (2) pg_upgrade did not
provide a pg_type OID for the new toast table, so that the crosscheck in
TypeCreate failed.  While both these problems were introduced by later
patches, they show that the hack being used to cause TOAST table creation
is overwhelmingly fragile (and untested).  I also note that before the
TypeCreate crosscheck was added, the code would have resulted in assigning
an indeterminate pg_type OID to the toast table, possibly causing a later
OID conflict in that catalog; so that it didn't really work even when
committed.

If we simply don't create a TOAST table, there will only be a problem if
the code tries to store a tuple that's wider than a page, and field
compression isn't sufficient to get it under a page.  Given that the TOAST
creation threshold is intended to be about a quarter of a page, it's very
hard to believe that cross-version differences in the do-we-need-a-toast-
table heuristic could result in an observable problem.  So let's just
follow the old version's conclusion about whether a TOAST table is needed.

(If we ever do change needs_toast_table() so much that this conclusion
doesn't apply, we can devise a solution at that time, and hopefully do
it in a less klugy way than 4c6780fd17aa43ed did.)

Back-patch to 9.3, like the previous patch.

Discussion: <8110.1462291...@sss.pgh.pa.us>

Branch
--
REL9_3_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/e1d88f983e4037f209d6274acfe0c20e6cb42472

Modified Files
--
contrib/pg_upgrade/dump.c   | 72 -
contrib/pg_upgrade/pg_upgrade.c |  4 +--
contrib/pg_upgrade/pg_upgrade.h |  1 -
src/backend/catalog/toasting.c  | 58 -
4 files changed, 22 insertions(+), 113 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix pg_upgrade to not fail when new-cluster TOAST rules differ f

[Tom Lane]

Fix pg_upgrade to not fail when new-cluster TOAST rules differ from old.

This patch essentially reverts commit 4c6780fd17aa43ed, in favor of a much
simpler solution for the case where the new cluster would choose to create
a TOAST table but the old cluster doesn't have one: just don't create a
TOAST table.

The existing code failed in at least two different ways if the situation
arose: (1) ALTER TABLE RESET didn't grab an exclusive lock, so that the
lock sanity check in create_toast_table failed; (2) pg_upgrade did not
provide a pg_type OID for the new toast table, so that the crosscheck in
TypeCreate failed.  While both these problems were introduced by later
patches, they show that the hack being used to cause TOAST table creation
is overwhelmingly fragile (and untested).  I also note that before the
TypeCreate crosscheck was added, the code would have resulted in assigning
an indeterminate pg_type OID to the toast table, possibly causing a later
OID conflict in that catalog; so that it didn't really work even when
committed.

If we simply don't create a TOAST table, there will only be a problem if
the code tries to store a tuple that's wider than a page, and field
compression isn't sufficient to get it under a page.  Given that the TOAST
creation threshold is intended to be about a quarter of a page, it's very
hard to believe that cross-version differences in the do-we-need-a-toast-
table heuristic could result in an observable problem.  So let's just
follow the old version's conclusion about whether a TOAST table is needed.

(If we ever do change needs_toast_table() so much that this conclusion
doesn't apply, we can devise a solution at that time, and hopefully do
it in a less klugy way than 4c6780fd17aa43ed did.)

Back-patch to 9.3, like the previous patch.

Discussion: <8110.1462291...@sss.pgh.pa.us>

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/1a2c17f8e221b0c3c63cb2b1be1f862444f62516

Modified Files
--
src/backend/catalog/toasting.c   | 54 +++-
src/bin/pg_upgrade/dump.c| 69 
src/bin/pg_upgrade/pg_upgrade.c  |  4 +--
src/bin/pg_upgrade/pg_upgrade.h  |  1 -
src/include/catalog/binary_upgrade.h |  5 ---
5 files changed, 22 insertions(+), 111 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix pg_upgrade to not fail when new-cluster TOAST rules differ f

[Tom Lane]

Fix pg_upgrade to not fail when new-cluster TOAST rules differ from old.

This patch essentially reverts commit 4c6780fd17aa43ed, in favor of a much
simpler solution for the case where the new cluster would choose to create
a TOAST table but the old cluster doesn't have one: just don't create a
TOAST table.

The existing code failed in at least two different ways if the situation
arose: (1) ALTER TABLE RESET didn't grab an exclusive lock, so that the
lock sanity check in create_toast_table failed; (2) pg_upgrade did not
provide a pg_type OID for the new toast table, so that the crosscheck in
TypeCreate failed.  While both these problems were introduced by later
patches, they show that the hack being used to cause TOAST table creation
is overwhelmingly fragile (and untested).  I also note that before the
TypeCreate crosscheck was added, the code would have resulted in assigning
an indeterminate pg_type OID to the toast table, possibly causing a later
OID conflict in that catalog; so that it didn't really work even when
committed.

If we simply don't create a TOAST table, there will only be a problem if
the code tries to store a tuple that's wider than a page, and field
compression isn't sufficient to get it under a page.  Given that the TOAST
creation threshold is intended to be about a quarter of a page, it's very
hard to believe that cross-version differences in the do-we-need-a-toast-
table heuristic could result in an observable problem.  So let's just
follow the old version's conclusion about whether a TOAST table is needed.

(If we ever do change needs_toast_table() so much that this conclusion
doesn't apply, we can devise a solution at that time, and hopefully do
it in a less klugy way than 4c6780fd17aa43ed did.)

Back-patch to 9.3, like the previous patch.

Discussion: <8110.1462291...@sss.pgh.pa.us>

Branch
--
REL9_5_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/196870f2fca7a4eec6f5b795dc75799f41e3ec7b

Modified Files
--
src/backend/catalog/toasting.c   | 54 +++-
src/bin/pg_upgrade/dump.c| 69 
src/bin/pg_upgrade/pg_upgrade.c  |  4 +--
src/bin/pg_upgrade/pg_upgrade.h  |  1 -
src/include/catalog/binary_upgrade.h |  5 ---
5 files changed, 22 insertions(+), 111 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix pg_upgrade to not fail when new-cluster TOAST rules differ f

[Tom Lane]

Fix pg_upgrade to not fail when new-cluster TOAST rules differ from old.

This patch essentially reverts commit 4c6780fd17aa43ed, in favor of a much
simpler solution for the case where the new cluster would choose to create
a TOAST table but the old cluster doesn't have one: just don't create a
TOAST table.

The existing code failed in at least two different ways if the situation
arose: (1) ALTER TABLE RESET didn't grab an exclusive lock, so that the
lock sanity check in create_toast_table failed; (2) pg_upgrade did not
provide a pg_type OID for the new toast table, so that the crosscheck in
TypeCreate failed.  While both these problems were introduced by later
patches, they show that the hack being used to cause TOAST table creation
is overwhelmingly fragile (and untested).  I also note that before the
TypeCreate crosscheck was added, the code would have resulted in assigning
an indeterminate pg_type OID to the toast table, possibly causing a later
OID conflict in that catalog; so that it didn't really work even when
committed.

If we simply don't create a TOAST table, there will only be a problem if
the code tries to store a tuple that's wider than a page, and field
compression isn't sufficient to get it under a page.  Given that the TOAST
creation threshold is intended to be about a quarter of a page, it's very
hard to believe that cross-version differences in the do-we-need-a-toast-
table heuristic could result in an observable problem.  So let's just
follow the old version's conclusion about whether a TOAST table is needed.

(If we ever do change needs_toast_table() so much that this conclusion
doesn't apply, we can devise a solution at that time, and hopefully do
it in a less klugy way than 4c6780fd17aa43ed did.)

Back-patch to 9.3, like the previous patch.

Discussion: <8110.1462291...@sss.pgh.pa.us>

Branch
--
REL9_4_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/e1aecebc041db969abe7cacb351577938f11a2af

Modified Files
--
contrib/pg_upgrade/dump.c| 69 
contrib/pg_upgrade/pg_upgrade.c  |  4 +--
contrib/pg_upgrade/pg_upgrade.h  |  1 -
src/backend/catalog/toasting.c   | 54 +++-
src/include/catalog/binary_upgrade.h |  5 ---
5 files changed, 22 insertions(+), 111 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

[Kevin Grittner]

On Fri, May 6, 2016 at 7:48 PM, Andres Freund  wrote:
> On 2016-05-06 19:43:24 -0500, Kevin Grittner wrote:
>> It's disappointing that I am not getting more consistent numbers,
>> but NUMA can be hard to manage that way.
>
> FWIW, in my experience, unless you disable autovacuum (or rather
> auto-analyze), the effects from non-predicable analyze runs with
> long-running snapshots are worse.  I mean the numa effects suck, but in
> r/w workload effects of analyze are often much worse.

Hm.  But the benefits of the patch are not there if autovacuum is
off.  I'm gonna need to ponder the best way to test given all that.

> That comment reminds me of a question I had: Did you consider the effect
> of this patch on analyze? It uses a snapshot, and by memory you've not
> built in a defense against analyze being cancelled.

Will need to check on that.

>> Will push shortly with the nit-pick fixes you requested.
>
> Cool.

Done.

I will be checking in on the buildfarm, but if it starts causing
problems while I'm, say, sleeping -- I won't be offended if someone
else reverts 7e3da1c4737fd6582e12c80983987e4d2cbc1d17.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

[COMMITTERS] pgsql: Disable BLOB test in pg_dump TAP tests

[Stephen Frost]

Disable BLOB test in pg_dump TAP tests

Buildfarm member jacana appears to have an issue with running this
test.  It's not entirely clear to me why, but rather than try to
fight with it, just disable it for now.

None of the other tests try to write out from psql directly as
this test does, so it seems likely that the rest of the tests will
be fine (as they have been on numerous other systems).

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/0f97c722bb201c67312eec0a9ff3691cda37878f

Modified Files
--
src/bin/pg_dump/t/002_pg_dump.pl | 68 
1 file changed, 34 insertions(+), 34 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Mitigate "snapshot too old" performance regression on NUMA

[Kevin Grittner]

Mitigate "snapshot too old" performance regression on NUMA

Limit maintenance of time to xid mapping to once per minute.  At
least in the tested case this brings performance within 5% of when
the feature is off, compared to several times slower without this
patch.

While there, fix comments and whitespace.

Ants Aasma, with cosmetic adjustments suggested by Andres Freund
Reviewed by Kevin Grittner and Andres Freund

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/7e3da1c4737fd6582e12c80983987e4d2cbc1d17

Modified Files
--
src/backend/utils/time/snapmgr.c | 85 +---
1 file changed, 62 insertions(+), 23 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

[Andres Freund]

On 2016-05-06 19:43:24 -0500, Kevin Grittner wrote:
> It's disappointing that I am not getting more consistent numbers,
> but NUMA can be hard to manage that way.

FWIW, in my experience, unless you disable autovacuum (or rather
auto-analyze), the effects from non-predicable analyze runs with
long-running snapshots are worse.  I mean the numa effects suck, but in
r/w workload effects of analyze are often much worse.


That comment reminds me of a question I had: Did you consider the effect
of this patch on analyze? It uses a snapshot, and by memory you've not
built in a defense against analyze being cancelled.


> Will push shortly with the nit-pick fixes you requested.

Cool.

Greetings,

Andres Freund


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

[Kevin Grittner]

On Fri, May 6, 2016 at 5:07 PM, Andres Freund  wrote:

> On 2016-05-06 14:18:22 -0500, Kevin Grittner wrote:
>> I rebased the patch Ants posted (attached), and am running
>> benchmarks on a cthulhu (a big NUMA machine with 8 memory nodes).
>> Normally I wouldn't post results without a lot more data points
>> with multiple samples at each, but the initial results have me
>> wondering whether people would like to see this pushed later today
>> so that it has some time in the buildfarm and then into beta1.
>
> I think that generally would make sense. We quite possibly need some
> further changes, but it seems more likely that we can find them if the
> patch runs close to the disabled performance.

ok

>> Running the r/w TPC-B (sort of) load with scale, jobs, and threads
>> at 1000, and the database configured as I would for a production
>> server of that size, preliminary TPS results are:
>>
>> master, -1:  8158
>> master, 10min: 2019
>> Ants' patch, 10min: 7804
>
> That's rather nice.  Did you test read-only as well?

Not yet.  I don't trust short runs, so I've been going with -T2400;
with setup times and so on, that limits me to one run per hour of
time I book the machine, and I'm competing with others for that.  I
do plan to run read-only, too.

>From the 40 minute tests so far with Ants' patch (alternating settings):

old_snapshot_threshold = 10
7804
9524
9512

old_snapshot_threshold = -1
10421
8691
8977

It's disappointing that I am not getting more consistent numbers,
but NUMA can be hard to manage that way.

> If you'd feel more comfortable committing after I've run some
> performance tests, I could kick off some soon.

I think I should get it onto the buildfarm if we're going for
beta2, so there's time to recognize any problem (unlikely as that
*seems*) and back this out before beta if needed.  That said, all
additional data points welcome!

>> I can see arguments for tuning this far in time for the beta, as
>> well as the argument to wait until after the beta, so I'm just
>> throwing it out there to see what other people think.  I wouldn't
>> do it unless I have three runs at -1 and 10min with the patch, all
>> showing similar numbers.  If the BF chokes on it I would revert
>> this optimization attempt.
>
> +1 for going forward.  I'm still doubtful that it's a good idea to the
> map maintenance from GetSnapshotData(), but the issue becomes much less
> severe when addressed like this.
>
> The primary reasons why I'd like to move it is because of the
> significant amount of added gettimeofday() calls which are a real hog in
> some virtualized environments, and because I'm doubtful of tying the
> current time to the xmin horizon.

When I initially presented the proof of concept patch during the
9.5 development cycle it was based on transaction counts, and that
was the biggest criticism, and it came from many quarters.  Using
time was the big demand from just about everyone, and I'm not sure
how you do that without a mapping of time to xmin horizon.  If you
have some other idea, I'm all ears.

> Looks roughly sensible.

Will push shortly with the nit-pick fixes you requested.

Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

[COMMITTERS] pgsql: First-draft release notes for 9.5.3.

[Tom Lane]

First-draft release notes for 9.5.3.

As usual, the release notes for other branches will be made by cutting
these down, but put them up for community review first.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/eb7de00ac2d282263541ece849ec71e2809e9467

Modified Files
--
doc/src/sgml/release-9.5.sgml | 474 ++
1 file changed, 474 insertions(+)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

[Andres Freund]

Hi,

On 2016-05-06 14:18:22 -0500, Kevin Grittner wrote:
> I rebased the patch Ants posted (attached), and am running
> benchmarks on a cthulhu (a big NUMA machine with 8 memory nodes).
> Normally I wouldn't post results without a lot more data points
> with multiple samples at each, but the initial results have me
> wondering whether people would like to see this pushed later today
> so that it has some time in the buildfarm and then into beta1.

I think that generally would make sense. We quite possibly need some
further changes, but it seems more likely that we can find them if the
patch runs close to the disabled performance.


> Running the r/w TPC-B (sort of) load with scale, jobs, and threads
> at 1000, and the database configured as I would for a production
> server of that size, preliminary TPS results are:
> 
> master, -1:  8158
> master, 10min: 2019
> Ants' patch, 10min: 7804

That's rather nice.  Did you test read-only as well?


If you'd feel more comfortable committing after I've run some
performance tests, I could kick off some soon.


> I can see arguments for tuning this far in time for the beta, as
> well as the argument to wait until after the beta, so I'm just
> throwing it out there to see what other people think.  I wouldn't
> do it unless I have three runs at -1 and 10min with the patch, all
> showing similar numbers.  If the BF chokes on it I would revert
> this optimization attempt.

+1 for going forward.  I'm still doubtful that it's a good idea to the
map maintenance from GetSnapshotData(), but the issue becomes much less
severe when addressed like this.

The primary reasons why I'd like to move it is because of the
significant amount of added gettimeofday() calls which are a real hog in
some virtualized environments, and because I'm doubtful of tying the
current time to the xmin horizon.


> diff --git a/src/backend/utils/time/snapmgr.c 
> b/src/backend/utils/time/snapmgr.c
> index e1551a3..b7d965a 100644
> --- a/src/backend/utils/time/snapmgr.c
> +++ b/src/backend/utils/time/snapmgr.c
> @@ -80,8 +80,11 @@ typedef struct OldSnapshotControlData
>*/
>   slock_t mutex_current;  /* protect current 
> timestamp */
>   int64   current_timestamp;  /* latest snapshot 
> timestamp */
> - slock_t mutex_latest_xmin;  /* protect latest 
> snapshot xmin */
> + slock_t mutex_latest_xmin;  /* protect latest 
> snapshot xmin
> + 
>  * and next_map_update
> + 
>  */
>   TransactionId latest_xmin;  /* latest snapshot xmin 
> */
> + int64   next_map_update;/* latest snapshot 
> valid up to */
>   slock_t mutex_threshold;/* protect threshold 
> fields */
>   int64   threshold_timestamp;/* earlier snapshot is old */
>   TransactionId threshold_xid;/* earlier xid may be gone */

Overly nitpickily I'd refer to the actual variable name (instead of
"latest snapshot xmin") in the mutex_latest_xmin comment.


>   if (!same_ts_as_threshold)
>   {
> + if (ts == update_ts)
> + {
> + xlimit = latest_xmin;
> + if (NormalTransactionIdFollows(xlimit, 
> recentXmin))
> + SetOldSnapshotThresholdTimestamp(ts, 
> xlimit);
> + }
> + else
> + {
>   LWLockAcquire(OldSnapshotTimeMapLock, LW_SHARED);
>  
>   if (oldSnapshotControl->count_used > 0

I guess it's just an issue in my mail-reader, but the indentation looks
funky here.


Looks roughly sensible.


Greetings,

Andres Freund


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Rename pgbench min/max to least/greatest, and fix handling of do

[Tom Lane]

> The very minor patch attached re-establishes the alphabetical order when 
> listing functions names in pgbench documentation.

Pushed, thanks.

regards, tom lane


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Docs: fix alphabetization of table entries.

[Tom Lane]

Docs: fix alphabetization of table entries.

Fabien Coelho

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/bbbae5ead3952bee9184da4864e2f4078940cac7

Modified Files
--
doc/src/sgml/ref/pgbench.sgml | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Docs: minor copy-editing for GSSAPI/SSPI authentication docs.

[Tom Lane]

Docs: minor copy-editing for GSSAPI/SSPI authentication docs.

Describe compat_realm = 0 as "disabled" not "enabled", per discussion
with Christian Ullrich.  I failed to resist the temptation to do some
other minor copy-editing in the same area.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/36db18eaa0def33b3f7ea5e3980c43431ca9c923

Modified Files
--
doc/src/sgml/client-auth.sgml | 35 ---
1 file changed, 20 insertions(+), 15 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Rename pgbench min/max to least/greatest, and fix handling of do

[Fabien COELHO]

Rename pgbench min/max to least/greatest, and fix handling of double 
args.


Ok.

The very minor patch attached re-establishes the alphabetical order when 
listing functions names in pgbench documentation.


--
Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index e6c7c94..ee363ef 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -955,13 +955,6 @@ pgbench  options  dbname
5432.0
   
   
-   int(x)
-   integer
-   cast to int
-   int(5.4 + 3.8)
-   9
-  
-  
greatest(a [, ... ] )
double if any a is double, else integer
largest value among arguments
@@ -969,6 +962,13 @@ pgbench  options  dbname
5
   
   
+   int(x)
+   integer
+   cast to int
+   int(5.4 + 3.8)
+   9
+  
+  
least(a [, ... ] )
double if any a is double, else integer
smallest value among arguments

-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Add test_pg_dump to @contrib_excludes

[Stephen Frost]

Add test_pg_dump to @contrib_excludes

The test_pg_dump extension doesn't have a C component, so we need
to exclude it from the MSVC build system trying to figure out how
to build it.

Also add a "MODULES" line to the Makefile, as test_extensions has.
Might not be necessary, but seems good to keep things consistent.

Lastly, remove the 'installcheck' line from test_pg_dump, as that
was causing redefinition errors, at least on my box.  This also
makes test_pg_dump consistent with how commit_ts is set up.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/6e243c43c9b2d35f91a73d3982db6a0cad0eb64b

Modified Files
--
src/test/modules/test_pg_dump/Makefile | 4 +---
src/tools/msvc/Mkvcbuild.pm| 2 +-
2 files changed, 2 insertions(+), 4 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Alvaro Herrera]

Tom Lane wrote:
> Alvaro Herrera  writes:
> > Stephen Frost wrote:
> >> I'm guessing the right answer here is to just add test_pg_dump to that
> >> list.
> 
> > I don't like this solution, because it means pg_dump will get little
> > testing on Windows.
> 
> No, that's incorrect: @contrib_excludes stops the MSVC stuff from trying
> to *build* in that directory, which is fine because there's nothing to
> build.  It doesn't prevent running the test case.  See eg.
> http://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=thrips=2016-05-06%2015%3A37%3A27=testmodules-install-check-C
> where the test_extensions test does get run even though it's in
> @contrib_excludes.

Ah, that's surprising, but much better of course.  Works for me.

> In any case, reverting the patch would mean no Windows testing, so
> your argument doesn't seem to have much force even if it were true?

I was suggesting to revert it for the beta and put it back with a
working Windows implementation after the beta, before GA.  I was afraid
that a hack to hide the test from Windows would become permanent.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Tom Lane]

Alvaro Herrera  writes:
> Stephen Frost wrote:
>> I'm guessing the right answer here is to just add test_pg_dump to that
>> list.

> I don't like this solution, because it means pg_dump will get little
> testing on Windows.

No, that's incorrect: @contrib_excludes stops the MSVC stuff from trying
to *build* in that directory, which is fine because there's nothing to
build.  It doesn't prevent running the test case.  See eg.
http://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=thrips=2016-05-06%2015%3A37%3A27=testmodules-install-check-C
where the test_extensions test does get run even though it's in
@contrib_excludes.

In any case, reverting the patch would mean no Windows testing, so
your argument doesn't seem to have much force even if it were true?

regards, tom lane


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: More small 9.6 release note improvements.

[Tom Lane]

More small 9.6 release note improvements.

Corrections per Jeff Janes, Christian Ullrich, and Daniel Vérité.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/76ef266a86fdc3c4bdbd12b9ea9cacd523e00779

Modified Files
--
doc/src/sgml/release-9.5.sgml |  2 +-
doc/src/sgml/release-9.6.sgml | 11 ++-
2 files changed, 7 insertions(+), 6 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Stephen Frost]

* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote:
> Stephen Frost wrote:
> > * Stephen Frost (sfr...@snowman.net) wrote:
> 
> > > The intagg contrib modules doesn't have a .c file either, nor a
> > > MODULES_big line, and I don't see any errors with it.  I'm not terribly
> > > familiar with how the Windows builds are run though.
> > 
> > It looks like there's an exclude list that intagg is on for the MSVC
> > build animals, presumably because it doesn't have a .c file to be
> > compiled.
> > 
> > I'm guessing the right answer here is to just add test_pg_dump to that
> > list.
> 
> I don't like this solution, because it means pg_dump will get little
> testing on Windows.  I agree with reverting now and figuring out a way
> to enable this test on Windows after the beta.  I suggest you grab hold
> of your own Windows installation so that you can whack it locally until
> it works, rather than waiting for buildfarm cycles.

This is only for the test_pg_dump extension, which is specifically just
testing pg_dump with extensions.  The vast majority of the pg_dump tests
are in src/bin/pg_dump and should be getting run on the Windows systems
(once this issue is addressed).

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Alvaro Herrera]

Stephen Frost wrote:
> * Stephen Frost (sfr...@snowman.net) wrote:

> > The intagg contrib modules doesn't have a .c file either, nor a
> > MODULES_big line, and I don't see any errors with it.  I'm not terribly
> > familiar with how the Windows builds are run though.
> 
> It looks like there's an exclude list that intagg is on for the MSVC
> build animals, presumably because it doesn't have a .c file to be
> compiled.
> 
> I'm guessing the right answer here is to just add test_pg_dump to that
> list.

I don't like this solution, because it means pg_dump will get little
testing on Windows.  I agree with reverting now and figuring out a way
to enable this test on Windows after the beta.  I suggest you grab hold
of your own Windows installation so that you can whack it locally until
it works, rather than waiting for buildfarm cycles.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Correct query in pg_dumpall:dumpRoles

[Stephen Frost]

Correct query in pg_dumpall:dumpRoles

We need to use a new branch due to the 9.5 addition of bypassrls
when adding in the clause to exclude pg_* roles from being dumped
by pg_dumpall.

Pointed out by Noah, patch by me.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/c778e27e13e883fb759f6100727aba80102933bd

Modified Files
--
src/bin/pg_dump/pg_dumpall.c | 10 ++
1 file changed, 10 insertions(+)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Stephen Frost]

* Tom Lane (t...@sss.pgh.pa.us) wrote:
> Stephen Frost  writes:
> > * Stephen Frost (sfr...@snowman.net) wrote:
> >> Looks like the test_pg_dump extension made the Windows builds upset.
> >> I'm guessing that's because I set 'MODULES_big' even though there isn't
> >> a .c component.
> >> 
> >> Doing a local build with that commented out, assuming that works and
> >> doesn't generate the .so any more on my Linux box, I'll push the change
> >> up to hopefully fix those buildfarm members.
> 
> > Alright, apparently that made other Windows buildfarm members unhappy...
> 
> > I guess the next approach will be to add back MODULES_big and add in a
> > .c file for the Windows systems to be happy about.  I'm certainly open
> > to other suggestions.
> 
> You should not need to do that; cf src/test/modules/test_extensions,
> which has got SQL-only extensions.

test_extensions is also included in the "contrib_excludes" list in
src/tools/msvc/Mkvcbuild.pm that I mentioned in my last email, so I'm
thinking that's what is needed.

> But at this point I think Peter's complaint has some force to it, and that
> what you ought to do is revert the testing patch.  You can have another go
> after beta1.

Are you suggesting commiting to still-9.6-HEAD post-beta1?  I took
Peter's comment as suggesting that adding the tests would have to wait
til after we branched 9.6, as we do for features.

I'd really like to have these tests included as that will make them
available to others more easily to add on to, and I'm certainly planning
to continue adding tests until I get pg_dump.c's coverage a lot better.
That seems like the perfect kind of effort that should be happening
right now- adding more tests and working to make sure that what's been
committed is correct (and fixing it when it isn't, as discovered by the
test suite with transforms and casts...).

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Tom Lane]

Stephen Frost  writes:
> * Stephen Frost (sfr...@snowman.net) wrote:
>> Looks like the test_pg_dump extension made the Windows builds upset.
>> I'm guessing that's because I set 'MODULES_big' even though there isn't
>> a .c component.
>> 
>> Doing a local build with that commented out, assuming that works and
>> doesn't generate the .so any more on my Linux box, I'll push the change
>> up to hopefully fix those buildfarm members.

> Alright, apparently that made other Windows buildfarm members unhappy...

> I guess the next approach will be to add back MODULES_big and add in a
> .c file for the Windows systems to be happy about.  I'm certainly open
> to other suggestions.

You should not need to do that; cf src/test/modules/test_extensions,
which has got SQL-only extensions.

But at this point I think Peter's complaint has some force to it, and that
what you ought to do is revert the testing patch.  You can have another go
after beta1.

regards, tom lane


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Stephen Frost]

* Stephen Frost (sfr...@snowman.net) wrote:
> Alright, apparently that made other Windows buildfarm members unhappy...
> 
> I guess the next approach will be to add back MODULES_big and add in a
> .c file for the Windows systems to be happy about.  I'm certainly open
> to other suggestions.
> 
> The intagg contrib modules doesn't have a .c file either, nor a
> MODULES_big line, and I don't see any errors with it.  I'm not terribly
> familiar with how the Windows builds are run though.

It looks like there's an exclude list that intagg is on for the MSVC
build animals, presumably because it doesn't have a .c file to be
compiled.

I'm guessing the right answer here is to just add test_pg_dump to that
list.

If there aren't any other suggestions, I'll go ahead and do that.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Stephen Frost]

* Stephen Frost (sfr...@snowman.net) wrote:
> * Stephen Frost (sfr...@snowman.net) wrote:
> > src/test/modules/test_pg_dump/Makefile |   25 +
> > src/test/modules/test_pg_dump/README   |2 +
> > .../modules/test_pg_dump/expected/test_pg_dump.out |6 +
> > src/test/modules/test_pg_dump/sql/test_pg_dump.sql |1 +
> > src/test/modules/test_pg_dump/t/001_base.pl|  535 
> > .../modules/test_pg_dump/test_pg_dump--1.0.sql |   15 +
> > src/test/modules/test_pg_dump/test_pg_dump.control |3 +
> 
> Looks like the test_pg_dump extension made the Windows builds upset.
> I'm guessing that's because I set 'MODULES_big' even though there isn't
> a .c component.
> 
> Doing a local build with that commented out, assuming that works and
> doesn't generate the .so any more on my Linux box, I'll push the change
> up to hopefully fix those buildfarm members.

Alright, apparently that made other Windows buildfarm members unhappy...

I guess the next approach will be to add back MODULES_big and add in a
.c file for the Windows systems to be happy about.  I'm certainly open
to other suggestions.

The intagg contrib modules doesn't have a .c file either, nor a
MODULES_big line, and I don't see any errors with it.  I'm not terribly
familiar with how the Windows builds are run though.

Thanks!

Stephen


signature.asc
Description: Digital signature

[COMMITTERS] pgsql: Remove MODULES_big from test_pg_dump

[Stephen Frost]

Remove MODULES_big from test_pg_dump

The Makefile for test_pg_dump shouldn't have a MODULES_big line
because there's no actual compiled bit for that extension.  Hopefully
this will fix the Windows buildfarm members which were complaining.

In passing, also add the 'prove_installcheck' bit to the pg_dump and
test_pg_dump Makefiles, to get the buildfarm members to actually run
those tests.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/eccfeeb631fa44850b644661d8b2ce94d9ef4fc1

Modified Files
--
src/bin/pg_dump/Makefile   | 3 +++
src/test/modules/test_pg_dump/Makefile | 4 +++-
2 files changed, 6 insertions(+), 1 deletion(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Avoid extra locks in GetSnapshotData if old_snapshot_threshold

[Kevin Grittner]

On Wed, Apr 20, 2016 at 8:08 PM, Ants Aasma  wrote:

> I had an idea I wanted to test out. The gist of it is to effectively
> have the last slot of timestamp to xid map stored in the latest_xmin
> field and only update the mapping when slot boundaries are crossed.
> See attached WIP patch for details. This way the exclusive lock only
> needs to be acquired once per minute. The common case is a spinlock
> that could be replaced with atomics later.

I rebased the patch Ants posted (attached), and am running
benchmarks on a cthulhu (a big NUMA machine with 8 memory nodes).
Normally I wouldn't post results without a lot more data points
with multiple samples at each, but the initial results have me
wondering whether people would like to see this pushed later today
so that it has some time in the buildfarm and then into beta1.

Running the r/w TPC-B (sort of) load with scale, jobs, and threads
at 1000, and the database configured as I would for a production
server of that size, preliminary TPS results are:

master, -1:  8158
master, 10min: 2019
Ants' patch, 10min: 7804

Basically it just skips the maintenance of the time/xid mapping
unless current time has advanced to a new minute.

I can see arguments for tuning this far in time for the beta, as
well as the argument to wait until after the beta, so I'm just
throwing it out there to see what other people think.  I wouldn't
do it unless I have three runs at -1 and 10min with the patch, all
showing similar numbers.  If the BF chokes on it I would revert
this optimization attempt.

Thoughts?

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


update-old-snapshot-map-once-per-tick-v2.patch
Description: binary/octet-stream

-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Andres Freund]

On 2016-05-06 15:11:53 -0400, Stephen Frost wrote:
> * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote:
> > On 5/6/16 2:06 PM, Stephen Frost wrote:
> > >Add TAP tests for pg_dump
> > 
> > I'd be the first to welcome this, but what happened to feature freeze?
> 
> These are just new tests..?  I assumed that would be welcome during post
> feature-freeze, and certainly no one raised any concerns about adding
> these tests during the discussion prior to my commiting them.
> 
> We back-patch new tests from time to time too, when they're associated
> with bug fixes, so I'm pretty confused why TAP tests would be an issue
> to add on HEAD post feature-freeze.
> 
> If the consensus is that we shouldn't add new tests during feature
> freeze, I'll revert the patch that added them and add them later, but,
> for my 2c at least, I think we should be happy to add these even after
> feature freeze.

+1


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Stephen Frost]

* Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote:
> On 5/6/16 2:06 PM, Stephen Frost wrote:
> >Add TAP tests for pg_dump
> 
> I'd be the first to welcome this, but what happened to feature freeze?

These are just new tests..?  I assumed that would be welcome during post
feature-freeze, and certainly no one raised any concerns about adding
these tests during the discussion prior to my commiting them.

We back-patch new tests from time to time too, when they're associated
with bug fixes, so I'm pretty confused why TAP tests would be an issue
to add on HEAD post feature-freeze.

If the consensus is that we shouldn't add new tests during feature
freeze, I'll revert the patch that added them and add them later, but,
for my 2c at least, I think we should be happy to add these even after
feature freeze.

Thanks!

Stephen


signature.asc
Description: Digital signature

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Peter Eisentraut]

On 5/6/16 2:06 PM, Stephen Frost wrote:

Add TAP tests for pg_dump


I'd be the first to welcome this, but what happened to feature freeze?

--
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Minimal fix for crash bug in quals_match_foreign_key.

[Robert Haas]

Minimal fix for crash bug in quals_match_foreign_key.

Discussion is still underway as to whether to revert the entire patch
that added this function, but that discussion may not conclude before
beta1.  So, in the meantime, let's do at least this much.

David Rowley

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/68d704edbfe14fd8d21545b244c3f6824e0fbb8a

Modified Files
--
src/backend/optimizer/path/costsize.c | 8 
1 file changed, 4 insertions(+), 4 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Limit maximum parallel degree to 1024.

[Robert Haas]

Limit maximum parallel degree to 1024.

This new limit affects both the max_parallel_degree GUC and the
parallel_degree reloption.  There may some day be a use case for using
more than 1024 CPUs for a single query, but that's surely not the case
right now.  Not only do not very many people have that many CPUs, but
the code hasn't been tested at that kind of scale and is very unlikely
to perform well, or even work at all, without a lot more work.  The
issue addressed by commit 06bd458cb812623c3f1fdd55216c4c08b06a8447 is
probably just one problem of many.

The idea of a more reasonable limit here was suggested by Tom Lane;
the value of 1024 was suggested by Amit Kapila.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/c7ea68ff8dfafc22c6cfefdb5929e12ec5d1e02a

Modified Files
--
src/backend/access/common/reloptions.c | 2 +-
src/backend/utils/misc/guc.c   | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

[Stephen Frost]

* Stephen Frost (sfr...@snowman.net) wrote:
> src/test/modules/test_pg_dump/Makefile |   25 +
> src/test/modules/test_pg_dump/README   |2 +
> .../modules/test_pg_dump/expected/test_pg_dump.out |6 +
> src/test/modules/test_pg_dump/sql/test_pg_dump.sql |1 +
> src/test/modules/test_pg_dump/t/001_base.pl|  535 
> .../modules/test_pg_dump/test_pg_dump--1.0.sql |   15 +
> src/test/modules/test_pg_dump/test_pg_dump.control |3 +

Looks like the test_pg_dump extension made the Windows builds upset.
I'm guessing that's because I set 'MODULES_big' even though there isn't
a .c component.

Doing a local build with that commented out, assuming that works and
doesn't generate the .so any more on my Linux box, I'll push the change
up to hopefully fix those buildfarm members.

Thanks!

Stephen


signature.asc
Description: Digital signature

[COMMITTERS] pgsql: Improve pg_upgrade's report about failure to match up old and ne

[Tom Lane]

Improve pg_upgrade's report about failure to match up old and new tables.

Ordinarily, pg_upgrade shouldn't have any difficulty in matching up all
the relations it sees in the old and new databases.  If it does, however,
it just goes belly-up with a pretty unhelpful error message.  That seemed
fine as long as we expected the case never to occur in the wild, but
Alvaro reported that it had been seen in a database whose pg_largeobject
table had somehow acquired a TOAST table.  That doesn't quite seem like
a case that pg_upgrade actually needs to handle, but it would be good if
the report were more diagnosable.  Hence, extend the logic to print out
as much information as we can about the mismatch(es) before we quit.

In passing, improve the readability of get_rel_infos()'s data collection
query, which had suffered seriously from lets-not-bother-to-update-comments
syndrome, and generally was unnecessarily disrespectful to readers.

It could be argued that this is a bug fix, but given that we have so few
reports, I don't feel a need to back-patch; at least not before this has
baked awhile in HEAD.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/73b9952e8278b9c9a6f5f8e2df196fea5abb61f0

Modified Files
--
src/bin/pg_upgrade/info.c   | 372 +---
src/bin/pg_upgrade/pg_upgrade.h |  13 +-
2 files changed, 244 insertions(+), 141 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Use mul_size when multiplying by the number of parallel workers.

[Robert Haas]

Use mul_size when multiplying by the number of parallel workers.

That way, if the result overflows size_t, you'll get an error instead
of undefined behavior, which seems like a plus.  This also has the
effect of casting the number of workers from int to Size, which is
better because it's harder to overflow int than size_t.

Dilip Kumar reported this issue and provided a patch upon which this
patch is based, but his version did use mul_size.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/06bd458cb812623c3f1fdd55216c4c08b06a8447

Modified Files
--
src/backend/access/transam/parallel.c |  6 --
src/backend/executor/execParallel.c   | 18 +++---
2 files changed, 15 insertions(+), 9 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Correct pg_dump WHERE clause for functions/aggregates

[Stephen Frost]

Correct pg_dump WHERE clause for functions/aggregates

The query to grab the function/aggregate information is now joining
to pg_init_privs, so we can simplify (and correct) the WHERE clause
used to determine if a given function's ACL has changed from the
initial ACL on the function.

Bug found by Noah, patch by me.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/64d60c8bf0703011d79dcb8a55dc42dcedc1e10f

Modified Files
--
src/bin/pg_dump/pg_dump.c | 12 ++--
1 file changed, 2 insertions(+), 10 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Add TAP tests for pg_dump

[Stephen Frost]

Add TAP tests for pg_dump

This TAP test suite will create a new cluster, populate it based on
the 'create_sql' values in the '%tests' hash, run all of the runs
defined in the '%pgdump_runs' hash, and then for each test in the
'%tests' hash, compare each run's output the the regular expression
defined for the test under the 'like' and 'unlike' functions, as
appropriate.

While this test suite covers a fair bit of ground (67% of pg_dump.c
and quite a bit of the other files in src/bin/pg_dump), there is
still quite a bit which remains to be added to provide better code
coverage.  Still, this is quite a bit better than we had, and has
found a few bugs already (note that the CREATE TRANSFORM test is
commented out, as it is currently failing).

Idea for using the TAP system from Tom, though all of the code is mine.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/6bd356c33a3cf3a49313dc8638ea4bb066c4cf37

Modified Files
--
src/bin/pg_dump/Makefile   |3 +
src/bin/pg_dump/pg_dump.c  |2 +-
src/bin/pg_dump/t/001_basic.pl |   42 +
src/bin/pg_dump/t/002_pg_dump.pl   | 2859 
src/test/modules/Makefile  |1 +
src/test/modules/test_pg_dump/.gitignore   |4 +
src/test/modules/test_pg_dump/Makefile |   25 +
src/test/modules/test_pg_dump/README   |2 +
.../modules/test_pg_dump/expected/test_pg_dump.out |6 +
src/test/modules/test_pg_dump/sql/test_pg_dump.sql |1 +
src/test/modules/test_pg_dump/t/001_base.pl|  535 
.../modules/test_pg_dump/test_pg_dump--1.0.sql |   15 +
src/test/modules/test_pg_dump/test_pg_dump.control |3 +
13 files changed, 3497 insertions(+), 1 deletion(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: pg_dump performance and other fixes

[Stephen Frost]

pg_dump performance and other fixes

Do not try to dump objects which do not have ACLs when only ACLs are
being requested.  This results in a significant performance improvement
as we can avoid querying for further information on these objects when
we don't need to.

When limiting the components to dump for an extension, consider what
components have been requested.  Initially, we incorrectly hard-coded
the components of the extension objects to dump, which would mean that
we wouldn't dump some components even with they were asked for and in
other cases we would dump components which weren't requested.

Correct defaultACLs to use 'dump_contains' instead of 'dump'.  The
defaultACL is considered a member of the namespace and should be
dumped based on the same set of components that the other objects in
the schema are, not based on what we're dumping for the namespace
itself (which might not include ACLs, if the namespace has just the
default or initial ACL).

Use DUMP_COMPONENT_ACL for from-initdb objects, to allow users to
change their ACLs, should they wish to.  This just extends what we
are doing for the pg_catalog namespace to objects which are not
members of namespaces.

Due to column ACLs being treated a bit differently from other ACLs
(they are actually reset to NULL when all privileges are revoked),
adjust the query which gathers column-level ACLs to consider all of
the ACL-relevant columns.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/5d589993cad212f7d556d52cc1e42fe18f65b057

Modified Files
--
src/bin/pg_dump/pg_dump.c | 188 +++---
1 file changed, 159 insertions(+), 29 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Only issue LOCK TABLE commands when necessary

[Stephen Frost]

Only issue LOCK TABLE commands when necessary

Reviewing the cases where we need to LOCK a given table during a dump,
it was pointed out by Tom that we really don't need to LOCK a table if
we are only looking to dump the ACL for it, or certain other
components.  After reviewing the queries run for all of the component
pieces, a list of components were determined to not require LOCK'ing
of the table.

This implements a check to avoid LOCK'ing those tables.

Initial complaint from Rushabh Lathia, discussed with Robert and Tom,
the patch is mine.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/e1b120a8cbb0f39a79926fd173f33e57202cdfa0

Modified Files
--
src/bin/pg_dump/pg_dump.c |  5 -
src/bin/pg_dump/pg_dump.h | 27 +++
2 files changed, 31 insertions(+), 1 deletion(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Remove various special checks around default roles

[Stephen Frost]

Remove various special checks around default roles

Default roles really should be like regular roles, for the most part.
This removes a number of checks that were trying to make default roles
extra special by not allowing them to be used as regular roles.

We still prevent users from creating roles in the "pg_" namespace or
from altering roles which exist in that namespace via ALTER ROLE, as
we can't preserve such changes, but otherwise the roles are very much
like regular roles.

Based on discussion with Robert and Tom.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/a89505fd21da337b81172871d8f65d9a4fa22a8b

Modified Files
--
src/backend/catalog/aclchk.c|  7 ---
src/backend/commands/alter.c|  3 ---
src/backend/commands/foreigncmds.c  | 13 -
src/backend/commands/policy.c   |  5 -
src/backend/commands/schemacmds.c   |  4 
src/backend/commands/tablecmds.c|  2 --
src/backend/commands/tablespace.c   |  4 
src/backend/commands/user.c | 11 ---
src/backend/commands/variable.c |  7 ---
src/test/regress/expected/rolenames.out | 18 +-
src/test/regress/sql/rolenames.sql  | 10 +-
11 files changed, 10 insertions(+), 74 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Update config.guess and config.sub

[Peter Eisentraut]

Update config.guess and config.sub

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/e324f8ad610bdc83a1392e54da5d9613e710b02f

Modified Files
--
config/config.guess | 114 ++--
config/config.sub   |  22 +-
2 files changed, 77 insertions(+), 59 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix possible read past end of string in to_timestamp().

[Tom Lane]

Fix possible read past end of string in to_timestamp().

to_timestamp() handles the TH/th format codes by advancing over two input
characters, whatever those are.  It failed to notice whether there were
two characters available to be skipped, making it possible to advance
the pointer past the end of the input string and keep on parsing.
A similar risk existed in the handling of "Y,YYY" format: it would advance
over three characters after the "," whether or not three characters were
available.

In principle this might be exploitable to disclose contents of server
memory.  But the security team concluded that it would be very hard to use
that way, because the parsing loop would stop upon hitting any zero byte,
and TH/th format codes can't be consecutive --- they have to follow some
other format code, which would have to match whatever data is there.
So it seems impractical to examine memory very much beyond the end of the
input string via this bug; and the input string will always be in local
memory not in disk buffers, making it unlikely that anything very
interesting is close to it in a predictable way.  So this doesn't quite
rise to the level of needing a CVE.

Thanks to Wolf Roediger for reporting this bug.

Branch
--
REL9_5_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/504af1fe0497bceb0421a63fd8ef18e069ae057a

Modified Files
--
src/backend/utils/adt/formatting.c | 84 ++
1 file changed, 39 insertions(+), 45 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix possible read past end of string in to_timestamp().

[Tom Lane]

Fix possible read past end of string in to_timestamp().

to_timestamp() handles the TH/th format codes by advancing over two input
characters, whatever those are.  It failed to notice whether there were
two characters available to be skipped, making it possible to advance
the pointer past the end of the input string and keep on parsing.
A similar risk existed in the handling of "Y,YYY" format: it would advance
over three characters after the "," whether or not three characters were
available.

In principle this might be exploitable to disclose contents of server
memory.  But the security team concluded that it would be very hard to use
that way, because the parsing loop would stop upon hitting any zero byte,
and TH/th format codes can't be consecutive --- they have to follow some
other format code, which would have to match whatever data is there.
So it seems impractical to examine memory very much beyond the end of the
input string via this bug; and the input string will always be in local
memory not in disk buffers, making it unlikely that anything very
interesting is close to it in a predictable way.  So this doesn't quite
rise to the level of needing a CVE.

Thanks to Wolf Roediger for reporting this bug.

Branch
--
REL9_4_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/1180868d11343f4ec01bf5d4964e360fceaab5ff

Modified Files
--
src/backend/utils/adt/formatting.c | 84 ++
1 file changed, 39 insertions(+), 45 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix possible read past end of string in to_timestamp().

[Tom Lane]

Fix possible read past end of string in to_timestamp().

to_timestamp() handles the TH/th format codes by advancing over two input
characters, whatever those are.  It failed to notice whether there were
two characters available to be skipped, making it possible to advance
the pointer past the end of the input string and keep on parsing.
A similar risk existed in the handling of "Y,YYY" format: it would advance
over three characters after the "," whether or not three characters were
available.

In principle this might be exploitable to disclose contents of server
memory.  But the security team concluded that it would be very hard to use
that way, because the parsing loop would stop upon hitting any zero byte,
and TH/th format codes can't be consecutive --- they have to follow some
other format code, which would have to match whatever data is there.
So it seems impractical to examine memory very much beyond the end of the
input string via this bug; and the input string will always be in local
memory not in disk buffers, making it unlikely that anything very
interesting is close to it in a predictable way.  So this doesn't quite
rise to the level of needing a CVE.

Thanks to Wolf Roediger for reporting this bug.

Branch
--
REL9_2_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/11247dd99a6502799a3cb5fdbe1f5c2354862b1b

Modified Files
--
src/backend/utils/adt/formatting.c | 84 ++
1 file changed, 39 insertions(+), 45 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix possible read past end of string in to_timestamp().

[Tom Lane]

Fix possible read past end of string in to_timestamp().

to_timestamp() handles the TH/th format codes by advancing over two input
characters, whatever those are.  It failed to notice whether there were
two characters available to be skipped, making it possible to advance
the pointer past the end of the input string and keep on parsing.
A similar risk existed in the handling of "Y,YYY" format: it would advance
over three characters after the "," whether or not three characters were
available.

In principle this might be exploitable to disclose contents of server
memory.  But the security team concluded that it would be very hard to use
that way, because the parsing loop would stop upon hitting any zero byte,
and TH/th format codes can't be consecutive --- they have to follow some
other format code, which would have to match whatever data is there.
So it seems impractical to examine memory very much beyond the end of the
input string via this bug; and the input string will always be in local
memory not in disk buffers, making it unlikely that anything very
interesting is close to it in a predictable way.  So this doesn't quite
rise to the level of needing a CVE.

Thanks to Wolf Roediger for reporting this bug.

Branch
--
REL9_3_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/462456d8d08339cbceb0cd3dad32bdddb0531ba5

Modified Files
--
src/backend/utils/adt/formatting.c | 84 ++
1 file changed, 39 insertions(+), 45 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix possible read past end of string in to_timestamp().

[Tom Lane]

Fix possible read past end of string in to_timestamp().

to_timestamp() handles the TH/th format codes by advancing over two input
characters, whatever those are.  It failed to notice whether there were
two characters available to be skipped, making it possible to advance
the pointer past the end of the input string and keep on parsing.
A similar risk existed in the handling of "Y,YYY" format: it would advance
over three characters after the "," whether or not three characters were
available.

In principle this might be exploitable to disclose contents of server
memory.  But the security team concluded that it would be very hard to use
that way, because the parsing loop would stop upon hitting any zero byte,
and TH/th format codes can't be consecutive --- they have to follow some
other format code, which would have to match whatever data is there.
So it seems impractical to examine memory very much beyond the end of the
input string via this bug; and the input string will always be in local
memory not in disk buffers, making it unlikely that anything very
interesting is close to it in a predictable way.  So this doesn't quite
rise to the level of needing a CVE.

Thanks to Wolf Roediger for reporting this bug.

Branch
--
REL9_1_STABLE

Details
---
http://git.postgresql.org/pg/commitdiff/7bad282c34d967be71f52b8812834beb060a68da

Modified Files
--
src/backend/utils/adt/formatting.c | 84 ++
1 file changed, 39 insertions(+), 45 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix possible read past end of string in to_timestamp().

[Tom Lane]

Fix possible read past end of string in to_timestamp().

to_timestamp() handles the TH/th format codes by advancing over two input
characters, whatever those are.  It failed to notice whether there were
two characters available to be skipped, making it possible to advance
the pointer past the end of the input string and keep on parsing.
A similar risk existed in the handling of "Y,YYY" format: it would advance
over three characters after the "," whether or not three characters were
available.

In principle this might be exploitable to disclose contents of server
memory.  But the security team concluded that it would be very hard to use
that way, because the parsing loop would stop upon hitting any zero byte,
and TH/th format codes can't be consecutive --- they have to follow some
other format code, which would have to match whatever data is there.
So it seems impractical to examine memory very much beyond the end of the
input string via this bug; and the input string will always be in local
memory not in disk buffers, making it unlikely that anything very
interesting is close to it in a predictable way.  So this doesn't quite
rise to the level of needing a CVE.

Thanks to Wolf Roediger for reporting this bug.

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/d136d600f9756232b36681fdfada0e40e1563a47

Modified Files
--
src/backend/utils/adt/formatting.c | 84 ++
1 file changed, 39 insertions(+), 45 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix pgbench's parsing of double values to notice trailing garbag

[Tom Lane]

Fix pgbench's parsing of double values to notice trailing garbage.

Noted by Fabien Coelho, though this isn't exactly his proposed patch.
(The technique used here is borrowed from the zic sources.)

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/6b8b4e4d8320d8c9daf9410175c40a09e58c4868

Modified Files
--
src/bin/pgbench/pgbench.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Improve handling of numeric-valued variables in pgbench.

[Tom Lane]

Improve handling of numeric-valued variables in pgbench.

The previous coding always stored variable values as strings, doing
conversion on-the-fly when a numeric value was needed or a number was to be
assigned.  This was a bit inefficient and risked loss of precision for
floating-point values.  The precision aspect had been hacked around by
printing doubles in "%.18e" format, which is ugly and has machine-dependent
results.  Instead, arrange to preserve an assigned numeric value in the
original binary numeric format, converting to string only when and if
needed.  When we do need to convert a double to string, convert in "%g"
format with DBL_DIG precision, which is the standard way to do it and
produces the least surprising results in most cases.

The implementation supports storing both a string value and a numeric
value for any one variable, with lazy conversion between them.  I also
arranged for lazy re-sorting of the variable array when new variables are
added.  That was mainly to allow a clean refactoring of putVariable()
into two levels of subroutine, but it may allow us to save a few sorts.

Discussion: <9188.1462475...@sss.pgh.pa.us>

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/9515299485a591b3a8f03c118d11809d01663665

Modified Files
--
src/bin/pgbench/pgbench.c | 274 +++---
1 file changed, 188 insertions(+), 86 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Docs: fix \crosstabview example.

[Tom Lane]

Docs: fix \crosstabview example.

This example missed being updated when we redefined \crosstabview's
argument processing.

Daniel Vérité

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/daa9856fcea775caeb4c92580b9693858509b43b

Modified Files
--
doc/src/sgml/ref/psql-ref.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix hash index vs "snapshot too old" problemms

[Kevin Grittner]

Fix hash index vs "snapshot too old" problemms

Hash indexes are not WAL-logged, and so do not maintain the LSN of
index pages.  Since the "snapshot too old" feature counts on
detecting error conditions using the LSN of a table and all indexes
on it, this makes it impossible to safely do early vacuuming on any
table with a hash index, so add this to the tests for whether the
xid used to vacuum a table can be adjusted based on
old_snapshot_threshold.

While at it, add a paragraph to the docs for old_snapshot_threshold
which specifically mentions this and other aspects of the feature
which may otherwise surprise users.

Problem reported and patch reviewed by Amit Kapila

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/2cc41acd8fa3ebb8f0501c6102a253fb7053cf46

Modified Files
--
doc/src/sgml/config.sgml | 13 ++
src/backend/access/hash/hash.c   |  1 -
src/backend/access/hash/hashsearch.c |  4 
src/backend/utils/cache/relcache.c   | 46 
src/backend/utils/time/snapmgr.c |  3 ++-
src/include/utils/rel.h  |  1 +
6 files changed, 62 insertions(+), 6 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Move and rename fmtReloptionsArray().

[Dean Rasheed]

Move and rename fmtReloptionsArray().

Move fmtReloptionsArray() from pg_dump.c to string_utils.c so that it
is available to other frontend code. In particular psql's \ev and \sv
commands need it to handle view reloptions. Also rename the function
to appendReloptionsArray(), which is a more accurate description of
what it does.

Author: Dean Rasheed
Reviewed-by: Peter Eisentraut
Discussion: 
http://www.postgresql.org/message-id/CAEZATCWZjCgKRyM-agE0p8ax15j9uyQoF=qew7d2xb6cf76...@mail.gmail.com

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/93a8c6fd6c5eeb61c12402f616a327d998a731c4

Modified Files
--
src/bin/pg_dump/pg_dump.c   | 77 +++--
src/fe_utils/string_utils.c | 72 ++
src/include/fe_utils/string_utils.h |  3 ++
3 files changed, 89 insertions(+), 63 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers

[COMMITTERS] pgsql: Fix psql's \ev and \sv commands so that they handle view relopti

[Dean Rasheed]

Fix psql's \ev and \sv commands so that they handle view reloptions.

Commit 8eb6407aaeb6cbd972839e356b436bb698f51cff added support for
editing and showing view definitions, but neglected to account for
view options such as security_barrier and WITH CHECK OPTION which are
not returned by pg_get_viewdef() and so need special handling.

Author: Dean Rasheed
Reviewed-by: Peter Eisentraut
Discussion: 
http://www.postgresql.org/message-id/CAEZATCWZjCgKRyM-agE0p8ax15j9uyQoF=qew7d2xb6cf76...@mail.gmail.com

Branch
--
master

Details
---
http://git.postgresql.org/pg/commitdiff/9b66aa006f81b2705337ca223daeeabf4db6453a

Modified Files
--
src/bin/psql/command.c | 77 +-
1 file changed, 70 insertions(+), 7 deletions(-)


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers