Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-15 Thread Bruce Momjian
On Fri, Jun 14, 2019 at 09:37:37PM -0400, Joe Conway wrote: > On 6/14/19 6:09 PM, Bruce Momjian wrote: > > On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote: > >> On 6/13/19 11:07 AM, Bruce Momjian wrote: > >> > In addition, while the 8k blocks would use a block cipher, the WAL would > >>

Fix typos and inconsistencies for v11+

2019-06-15 Thread Alexander Lakhin
Hello hackers, Please consider fixing the following typos and inconsistencies living in the source code starting from v11: 3.1 add_placeholders_to_child_joinrel -> remove (orphaned after 7cfdc770) 3.2 AttachIndexInfo -> IndexAttachInfo (an internal inconsistency) 3.3 BlockRecordInfo -> BlockInfoRe

assertion at postmaster start

2019-06-15 Thread Alvaro Herrera
Once in a blue moon I get this assertion failure on server start: 2019-06-15 12:00:29.650 -04 [30080] LOG: iniciando PostgreSQL 12beta1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0, 64-bit 2019-06-15 12:00:29.650 -04 [30080] LOG: escuchando en la dirección IPv4 «

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-15 Thread Tom Lane
Alvaro Herrera writes: > On 2019-Jun-14, Tom Lane wrote: >> BTW, after looking around a bit I wonder if this complaint isn't >> exposing an actual logic bug. Shouldn't skip_tuple_lock have >> a lifetime similar to first_time? > I think there are worse problems here. I tried the attached isolati

Re: Improve handling of pg_stat_statements handling of bind "IN" variables

2019-06-15 Thread Pavel Trukhanov
Thanks for the feedback. I thought once again about separating IN from ARRAY, with refactoring etc as Tom suggested, and now I don't think it's worth it to do so. I've tried to implement that, and besides that it will require to change things in every part of query processing pipeline, it seems th

Re: Extracting only the columns needed for a query

2019-06-15 Thread Tom Lane
Melanie Plageman writes: > While hacking on zedstore, we needed to get a list of the columns to > be projected--basically all of the columns needed to satisfy the > query. The two use cases we have for this is > 1) to pass this column list down to the AM layer for the AM to leverage it > 2) for us

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-15 Thread Alvaro Herrera
On 2019-Jun-14, Alvaro Herrera wrote: > I think there are worse problems here. I tried the attached isolation > spec. Note that the only difference in the two permutations is that s0 > finishes earlier in one than the other; yet the first one works fine and > the second one hangs until killed by

Re: Dead encoding conversion functions

2019-06-15 Thread Noah Misch
On Wed, May 29, 2019 at 03:03:13PM -0400, Tom Lane wrote: > Pursuant to today's discussion at PGCon about code coverage, I went > nosing into some of the particularly under-covered subdirectories > in our tree, and immediately tripped over an interesting factoid: > the ASCII<->MIC and ASCII<->UTF8

Re: pg_upgrade can result in early wraparound on databases with high transaction load

2019-06-15 Thread Noah Misch
On Tue, May 21, 2019 at 03:23:00PM -0700, Peter Geoghegan wrote: > On Mon, May 20, 2019 at 3:10 AM Jason Harvey wrote: > > This week I upgraded one of my large(2.8TB), high-volume databases from 9 > > to 11. The upgrade itself went fine. About two days later, we unexpectedly > > hit transaction

Re: Do we expect tests to work with default_transaction_isolation=serializable

2019-06-15 Thread Noah Misch
On Sun, May 19, 2019 at 03:55:06PM -0700, Andres Freund wrote: > I seem to recall that we expect tests to either work with > default_transaction_isolation=serializable, or to set it to a different > level where needed. > > Currently that's not the case. When running check-world with PGOPTIONS > se

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Noah Misch
On Fri, Jun 14, 2019 at 04:58:47PM -0400, Tom Lane wrote: > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=0995cefa74510ee0e38d1bf095b2eef2c1ea37c4 > + > + > + Avoid corruption of a btree index in the unlikely case that a failure > + occurs during key truncation

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Peter Geoghegan
On Sat, Jun 15, 2019 at 1:39 PM Noah Misch wrote: > To me, this text implies a cautious DBA should amcheck every index. Reading > the thread[1], I no longer think that. It's enough to monitor that VACUUM > doesn't start failing persistently on any index. I suggest replacing this > release note

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Peter Geoghegan
On Sat, Jun 15, 2019 at 2:11 PM Peter Geoghegan wrote: > On Sat, Jun 15, 2019 at 1:39 PM Noah Misch wrote: > > To me, this text implies a cautious DBA should amcheck every index. Reading > > the thread[1], I no longer think that. It's enough to monitor that VACUUM > > doesn't start failing pers

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Noah Misch
On Sat, Jun 15, 2019 at 02:11:41PM -0700, Peter Geoghegan wrote: > On Sat, Jun 15, 2019 at 1:39 PM Noah Misch wrote: > > To me, this text implies a cautious DBA should amcheck every index. Reading > > the thread[1], I no longer think that. It's enough to monitor that VACUUM > > doesn't start fai

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Tom Lane
Noah Misch writes: > On Sat, Jun 15, 2019 at 02:11:41PM -0700, Peter Geoghegan wrote: >> I agree that this isn't terribly significant in general. Your proposed >> wording seems better than what we have now, but a reference to INCLUDE >> indexes also seems like a good idea. They are the only type o

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-15 Thread Oleksii Kliukin
Alvaro Herrera wrote: > On 2019-Jun-14, Alvaro Herrera wrote: > >> I think there are worse problems here. I tried the attached isolation >> spec. Note that the only difference in the two permutations is that s0 >> finishes earlier in one than the other; yet the first one works fine and >> the

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Peter Geoghegan
On Sat, Jun 15, 2019 at 3:05 PM Tom Lane wrote: > Thanks for the input, guys. What do you think of > > Avoid writing an invalid empty btree index page in the unlikely case > that a failure occurs while processing INCLUDEd columns during a page > split (Peter Geoghegan) > > The

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Noah Misch
On Sat, Jun 15, 2019 at 06:05:00PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Sat, Jun 15, 2019 at 02:11:41PM -0700, Peter Geoghegan wrote: > >> I agree that this isn't terribly significant in general. Your proposed > >> wording seems better than what we have now, but a reference to INCLUD

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-15 Thread Alvaro Herrera
On 2019-Jun-16, Oleksii Kliukin wrote: > Alvaro Herrera wrote: > > > On 2019-Jun-14, Alvaro Herrera wrote: > > > >> I think there are worse problems here. I tried the attached isolation > >> spec. Note that the only difference in the two permutations is that s0 > >> finishes earlier in one th

Re: CREATE STATISTICS documentation bug

2019-06-15 Thread Tomas Vondra
On Fri, Jun 14, 2019 at 03:23:29PM -0400, Robert Haas wrote: https://www.postgresql.org/docs/12/sql-createstatistics.html contains this example command: CREATE STATISTICS s2 (mcv) ON (a, b) FROM t2; But that produces: psql: ERROR: only simple column references are allowed in CREATE STATISTICS

Re: Multivariate MCV stats can leak data to unprivileged users

2019-06-15 Thread Tomas Vondra
On Thu, Jun 13, 2019 at 07:37:45PM +0200, Tomas Vondra wrote: ... OK, attached are patches fixing the issues reported by you and John Naylor, and squashing the parts into just two patches (catalog split and pg_stats_ext). Barring objections, I'll push those tomorrow. I've renamed columns in the

Re: Improve handling of pg_stat_statements handling of bind "IN" variables

2019-06-15 Thread Greg Stark
On Sat., Jun. 15, 2019, 12:29 p.m. Pavel Trukhanov, < pavel.trukha...@gmail.com> wrote: > > So I don't think there's actually enough benefit to split those two apart. > > Now I want to do this: just add a meta info (basically a bool field) > to the ArrayExpr struct, so on later stages we could tel

Re: Improve handling of pg_stat_statements handling of bind "IN" variables

2019-06-15 Thread Greg Stark
On Sat., Jun. 15, 2019, 8:30 p.m. Greg Stark, wrote: > > > So what would this do for someone who explicitly writes: > > WHERE col = ANY ? > > and pass an array? > Actually thinking about this for two more seconds the question is what it would do with a query like WHERE col = ANY '1,2,3'::intege

The flinfo->fn_extra question, from me this time.

2019-06-15 Thread Chapman Flack
Hi hackers, I see evidence on this list that it's sort of a rite of passage to ask the flinfo->fn_extra question, and my time has come. So please let me know if I seem to correctly understand the limits on its use. I gather that various extensions use it to stash various things. But (I assume) .

Re: The flinfo->fn_extra question, from me this time.

2019-06-15 Thread Tom Lane
Chapman Flack writes: > So please let me know if I seem to correctly understand the limits > on its use. > I gather that various extensions use it to stash various things. But > (I assume) ... they will only touch fn_extra in FmgrInfo structs that > pertain to *their own functions*. (Please say t

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-15 Thread Bruce Momjian
On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote: > On 6/13/19 11:07 AM, Bruce Momjian wrote: > > On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote: > >> Yeah, in principle since data key of 2 tier key architecture should > >> not go outside database I think we should not te

Re: The flinfo->fn_extra question, from me this time.

2019-06-15 Thread Chapman Flack
On 06/15/19 21:21, Tom Lane wrote: > Yup. (Of course, you don't have to use the SRF_FIRSTCALL_INIT > infrastructure.) That had crossed my mind ... but it seems there's around 80 or 100 lines of good stuff there that'd be a shame to duplicate. If only init_MultiFuncCall() took an extra void ** arg

PG 12 beta 1 segfault during analyze

2019-06-15 Thread Steve Singer
I encountered the following segfault when running against a PG 12 beta1 during a analyze against a table. #0 0x56008ad0c826 in update_attstats (vacattrstats=0x0, natts=2139062143, inh=false, relid=0x40>) at analyze.c:572 #1 do_analyze_rel (onerel=onerel@entry=0x7f0bc59a7a38, params

Re: PG 12 beta 1 segfault during analyze

2019-06-15 Thread Tom Lane
Steve Singer writes: > I encountered the following segfault when running against a PG 12 beta1 > during a analyze against a table. Nobody else has reported this, so you're going to have to work on producing a self-contained test case, or else debugging it yourself. regar

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-15 Thread Bruce Momjian
On Fri, Jun 14, 2019 at 09:37:37PM -0400, Joe Conway wrote: > On 6/14/19 6:09 PM, Bruce Momjian wrote: > > On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote: > >> On 6/13/19 11:07 AM, Bruce Momjian wrote: > >> > In addition, while the 8k blocks would use a block cipher, the WAL would > >>

Fix typos and inconsistencies for v11+

2019-06-15 Thread Alexander Lakhin
Hello hackers, Please consider fixing the following typos and inconsistencies living in the source code starting from v11: 3.1 add_placeholders_to_child_joinrel -> remove (orphaned after 7cfdc770) 3.2 AttachIndexInfo -> IndexAttachInfo (an internal inconsistency) 3.3 BlockRecordInfo -> BlockInfoRe

assertion at postmaster start

2019-06-15 Thread Alvaro Herrera
Once in a blue moon I get this assertion failure on server start: 2019-06-15 12:00:29.650 -04 [30080] LOG: iniciando PostgreSQL 12beta1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0, 64-bit 2019-06-15 12:00:29.650 -04 [30080] LOG: escuchando en la dirección IPv4 «

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-15 Thread Tom Lane
Alvaro Herrera writes: > On 2019-Jun-14, Tom Lane wrote: >> BTW, after looking around a bit I wonder if this complaint isn't >> exposing an actual logic bug. Shouldn't skip_tuple_lock have >> a lifetime similar to first_time? > I think there are worse problems here. I tried the attached isolati

Re: Improve handling of pg_stat_statements handling of bind "IN" variables

2019-06-15 Thread Pavel Trukhanov
Thanks for the feedback. I thought once again about separating IN from ARRAY, with refactoring etc as Tom suggested, and now I don't think it's worth it to do so. I've tried to implement that, and besides that it will require to change things in every part of query processing pipeline, it seems th

Re: Extracting only the columns needed for a query

2019-06-15 Thread Tom Lane
Melanie Plageman writes: > While hacking on zedstore, we needed to get a list of the columns to > be projected--basically all of the columns needed to satisfy the > query. The two use cases we have for this is > 1) to pass this column list down to the AM layer for the AM to leverage it > 2) for us

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-15 Thread Alvaro Herrera
On 2019-Jun-14, Alvaro Herrera wrote: > I think there are worse problems here. I tried the attached isolation > spec. Note that the only difference in the two permutations is that s0 > finishes earlier in one than the other; yet the first one works fine and > the second one hangs until killed by

Re: Dead encoding conversion functions

2019-06-15 Thread Noah Misch
On Wed, May 29, 2019 at 03:03:13PM -0400, Tom Lane wrote: > Pursuant to today's discussion at PGCon about code coverage, I went > nosing into some of the particularly under-covered subdirectories > in our tree, and immediately tripped over an interesting factoid: > the ASCII<->MIC and ASCII<->UTF8

Re: pg_upgrade can result in early wraparound on databases with high transaction load

2019-06-15 Thread Noah Misch
On Tue, May 21, 2019 at 03:23:00PM -0700, Peter Geoghegan wrote: > On Mon, May 20, 2019 at 3:10 AM Jason Harvey wrote: > > This week I upgraded one of my large(2.8TB), high-volume databases from 9 > > to 11. The upgrade itself went fine. About two days later, we unexpectedly > > hit transaction

Re: Do we expect tests to work with default_transaction_isolation=serializable

2019-06-15 Thread Noah Misch
On Sun, May 19, 2019 at 03:55:06PM -0700, Andres Freund wrote: > I seem to recall that we expect tests to either work with > default_transaction_isolation=serializable, or to set it to a different > level where needed. > > Currently that's not the case. When running check-world with PGOPTIONS > se

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Noah Misch
On Fri, Jun 14, 2019 at 04:58:47PM -0400, Tom Lane wrote: > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=0995cefa74510ee0e38d1bf095b2eef2c1ea37c4 > + > + > + Avoid corruption of a btree index in the unlikely case that a failure > + occurs during key truncation

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Peter Geoghegan
On Sat, Jun 15, 2019 at 1:39 PM Noah Misch wrote: > To me, this text implies a cautious DBA should amcheck every index. Reading > the thread[1], I no longer think that. It's enough to monitor that VACUUM > doesn't start failing persistently on any index. I suggest replacing this > release note

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Peter Geoghegan
On Sat, Jun 15, 2019 at 2:11 PM Peter Geoghegan wrote: > On Sat, Jun 15, 2019 at 1:39 PM Noah Misch wrote: > > To me, this text implies a cautious DBA should amcheck every index. Reading > > the thread[1], I no longer think that. It's enough to monitor that VACUUM > > doesn't start failing pers

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Noah Misch
On Sat, Jun 15, 2019 at 02:11:41PM -0700, Peter Geoghegan wrote: > On Sat, Jun 15, 2019 at 1:39 PM Noah Misch wrote: > > To me, this text implies a cautious DBA should amcheck every index. Reading > > the thread[1], I no longer think that. It's enough to monitor that VACUUM > > doesn't start fai

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Tom Lane
Noah Misch writes: > On Sat, Jun 15, 2019 at 02:11:41PM -0700, Peter Geoghegan wrote: >> I agree that this isn't terribly significant in general. Your proposed >> wording seems better than what we have now, but a reference to INCLUDE >> indexes also seems like a good idea. They are the only type o

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-15 Thread Oleksii Kliukin
Alvaro Herrera wrote: > On 2019-Jun-14, Alvaro Herrera wrote: > >> I think there are worse problems here. I tried the attached isolation >> spec. Note that the only difference in the two permutations is that s0 >> finishes earlier in one than the other; yet the first one works fine and >> the

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Peter Geoghegan
On Sat, Jun 15, 2019 at 3:05 PM Tom Lane wrote: > Thanks for the input, guys. What do you think of > > Avoid writing an invalid empty btree index page in the unlikely case > that a failure occurs while processing INCLUDEd columns during a page > split (Peter Geoghegan) > > The

Re: Draft back-branch release notes are up for review

2019-06-15 Thread Noah Misch
On Sat, Jun 15, 2019 at 06:05:00PM -0400, Tom Lane wrote: > Noah Misch writes: > > On Sat, Jun 15, 2019 at 02:11:41PM -0700, Peter Geoghegan wrote: > >> I agree that this isn't terribly significant in general. Your proposed > >> wording seems better than what we have now, but a reference to INCLUD

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-15 Thread Alvaro Herrera
On 2019-Jun-16, Oleksii Kliukin wrote: > Alvaro Herrera wrote: > > > On 2019-Jun-14, Alvaro Herrera wrote: > > > >> I think there are worse problems here. I tried the attached isolation > >> spec. Note that the only difference in the two permutations is that s0 > >> finishes earlier in one th

Re: CREATE STATISTICS documentation bug

2019-06-15 Thread Tomas Vondra
On Fri, Jun 14, 2019 at 03:23:29PM -0400, Robert Haas wrote: https://www.postgresql.org/docs/12/sql-createstatistics.html contains this example command: CREATE STATISTICS s2 (mcv) ON (a, b) FROM t2; But that produces: psql: ERROR: only simple column references are allowed in CREATE STATISTICS

Re: Multivariate MCV stats can leak data to unprivileged users

2019-06-15 Thread Tomas Vondra
On Thu, Jun 13, 2019 at 07:37:45PM +0200, Tomas Vondra wrote: ... OK, attached are patches fixing the issues reported by you and John Naylor, and squashing the parts into just two patches (catalog split and pg_stats_ext). Barring objections, I'll push those tomorrow. I've renamed columns in the

Re: Improve handling of pg_stat_statements handling of bind "IN" variables

2019-06-15 Thread Greg Stark
On Sat., Jun. 15, 2019, 12:29 p.m. Pavel Trukhanov, < pavel.trukha...@gmail.com> wrote: > > So I don't think there's actually enough benefit to split those two apart. > > Now I want to do this: just add a meta info (basically a bool field) > to the ArrayExpr struct, so on later stages we could tel

Re: Improve handling of pg_stat_statements handling of bind "IN" variables

2019-06-15 Thread Greg Stark
On Sat., Jun. 15, 2019, 8:30 p.m. Greg Stark, wrote: > > > So what would this do for someone who explicitly writes: > > WHERE col = ANY ? > > and pass an array? > Actually thinking about this for two more seconds the question is what it would do with a query like WHERE col = ANY '1,2,3'::intege

The flinfo->fn_extra question, from me this time.

2019-06-15 Thread Chapman Flack
Hi hackers, I see evidence on this list that it's sort of a rite of passage to ask the flinfo->fn_extra question, and my time has come. So please let me know if I seem to correctly understand the limits on its use. I gather that various extensions use it to stash various things. But (I assume) .

Re: The flinfo->fn_extra question, from me this time.

2019-06-15 Thread Tom Lane
Chapman Flack writes: > So please let me know if I seem to correctly understand the limits > on its use. > I gather that various extensions use it to stash various things. But > (I assume) ... they will only touch fn_extra in FmgrInfo structs that > pertain to *their own functions*. (Please say t

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-15 Thread Bruce Momjian
On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote: > On 6/13/19 11:07 AM, Bruce Momjian wrote: > > On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote: > >> Yeah, in principle since data key of 2 tier key architecture should > >> not go outside database I think we should not te

Re: The flinfo->fn_extra question, from me this time.

2019-06-15 Thread Chapman Flack
On 06/15/19 21:21, Tom Lane wrote: > Yup. (Of course, you don't have to use the SRF_FIRSTCALL_INIT > infrastructure.) That had crossed my mind ... but it seems there's around 80 or 100 lines of good stuff there that'd be a shame to duplicate. If only init_MultiFuncCall() took an extra void ** arg

PG 12 beta 1 segfault during analyze

2019-06-15 Thread Steve Singer
I encountered the following segfault when running against a PG 12 beta1 during a analyze against a table. #0 0x56008ad0c826 in update_attstats (vacattrstats=0x0, natts=2139062143, inh=false, relid=0x40>) at analyze.c:572 #1 do_analyze_rel (onerel=onerel@entry=0x7f0bc59a7a38, params

Re: PG 12 beta 1 segfault during analyze

2019-06-15 Thread Tom Lane
Steve Singer writes: > I encountered the following segfault when running against a PG 12 beta1 > during a analyze against a table. Nobody else has reported this, so you're going to have to work on producing a self-contained test case, or else debugging it yourself. regar