Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-13 Thread Bruce Momjian
On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote: > On Thu, Jun 13, 2019 at 3:48 AM Bruce Momjian wrote: > > The big question is how many people will be mixing encrypted and > > unencrypted data in the same cluster, and care about performance? Just > > becau

Re: PG 12 draft release notes

2019-06-13 Thread Bruce Momjian
On Thu, Jun 13, 2019 at 03:33:48PM +0900, Michael Paquier wrote: > On Wed, Jun 12, 2019 at 05:25:37PM -0400, Bruce Momjian wrote: > > Since we did not backpatch this fix, I am hesitant to spell out exactly > > how to exploit this DOS attack. Yes, people can read it in the ema

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-12 Thread Bruce Momjian
On Wed, Jun 5, 2019 at 11:54:04AM +0900, Masahiko Sawada wrote: > On Fri, May 10, 2019 at 2:42 AM Bruce Momjian wrote: > > I think we need to step back and see what we want to do. There are six > > levels of possible encryption: > > > > 1. client-side column enc

Re: PG 12 draft release notes

2019-06-12 Thread Bruce Momjian
On Tue, May 21, 2019 at 02:22:53PM -0700, Peter Geoghegan wrote: > On Tue, May 21, 2019 at 1:51 PM Bruce Momjian wrote: > > > My concern here (which I believe Alexander shares) is that it doesn't > > > make sense to group these two items together. They're two totally &

Re: PG 12 draft release notes

2019-06-12 Thread Bruce Momjian
On Tue, May 21, 2019 at 12:57:56PM -0700, Andres Freund wrote: > Hi, > > On 2019-05-21 15:47:34 -0400, Bruce Momjian wrote: > > On Mon, May 20, 2019 at 03:17:19PM -0700, Andres Freund wrote: > > > Hi, > > > > > > Note that I've added a few questions t

Re: PG 12 draft release notes

2019-06-12 Thread Bruce Momjian
> Add function > linkend="functions-info-partition">pg_partition_tree() > to display information about partitions (Amit Langote) > > > > Can we combine these three? Good idea, done. -- Bruce Momjian ht

Re: PG 12 draft release notes

2019-06-12 Thread Bruce Momjian
On Wed, Jun 12, 2019 at 03:06:34PM -0700, Peter Geoghegan wrote: > On Wed, Jun 12, 2019 at 1:16 PM Bruce Momjian wrote: > > First, my apologies in getting to this so late. Peter Geoghegan > > supplied me with slides and a video to study, and I now understand how > &

Re: PG 12 draft release notes

2019-06-12 Thread Bruce Momjian
On Fri, Jun 7, 2019 at 12:04:33PM +0200, Adrien Nayrat wrote: > On 5/11/19 10:33 PM, Bruce Momjian wrote: > > I have posted a draft copy of the PG 12 release notes here: > > > > http://momjian.us/pgsql_docs/release-12.html > > > > They are committed to gi

Re: PG 12 draft release notes

2019-06-12 Thread Bruce Momjian
> explanation along those lines. Since we did not backpatch this fix, I am hesitant to spell out exactly how to exploit this DOS attack. Yes, people can read it in the email archives, and commit messages, but I don't see the value in spelling it out the release notes too. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: POC: converting Lists into arrays

2019-06-13 Thread Bruce Momjian
ed the performance impact? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: release notes: tids & self-joins

2019-06-13 Thread Bruce Momjian
ented that > joining on TID is only likely to be useful in a self-join. > > > So I think that this should probably be changed to say something like > > "Improve optimization of self-joins on ctid columns" or "Improve > > optimization of joins involvin

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-13 Thread Bruce Momjian
On Fri, Jun 14, 2019 at 12:41:17AM +0200, Tomas Vondra wrote: > On Thu, Jun 13, 2019 at 11:07:25AM -0400, Bruce Momjian wrote: > IMHO we should implement the simplest system possible, and optimize the > hell out of it without sacrificing any safety/security aspects. No smart > tunabl

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-13 Thread Bruce Momjian
t; external tools (e.g. relfilenode/block, needed by pg_waldump). > > Then we wouldn't need to reshuffle the WAL, I think. I was thinking we would just encrypt the entire WAL file, and use the WAL file name as the IV. -- Bruce Momjian http://momjian.us EnterpriseDB

Re: vacuumdb as server application in v12 release note

2019-06-13 Thread Bruce Momjian
reSQL > Client Applications" section in our manual. Sorry, fixed, thanks. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: PG 12 draft release notes

2019-06-12 Thread Bruce Momjian
> work. The term "suffix truncation" isn't something users have heard > of, and we shouldn't use it here, but the *idea* of suffix truncation > is very well established. As I mentioned, it matters for things like > covering indexes (indexes that are designed to be used by index

Re: PG 12 draft release notes

2019-06-12 Thread Bruce Momjian
On Wed, Jun 12, 2019 at 07:42:31PM -0700, Peter Geoghegan wrote: > On Wed, Jun 12, 2019 at 7:29 PM Bruce Momjian wrote: > > OK, I mentioned something about increased locality now. Patch attached. > > Looks good -- locality is a good catch-all term. Great, patch applied. --

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-15 Thread Bruce Momjian
On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote: > On 6/13/19 11:07 AM, Bruce Momjian wrote: > > On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote: > >> Yeah, in principle since data key of 2 tier key architecture should > >> not go outside

Re: pg12 release notes

2019-05-09 Thread Bruce Momjian
On Wed, May 8, 2019 at 03:32:04PM -0500, Justin Pryzby wrote: > I noticed you added release notes at bdf595adbca195fa54a909c74a5233ebc30641a1, > thanks for writing them. > -This improve optimization for columns with non-uniform distributions that > often appear in WHERE clauses. > +This

Re: pg12 release notes

2019-05-09 Thread Bruce Momjian
cause > > conflicts (Surafel Temesgen) > > + > > restore *using* INSERT statements ? > cause "unique index" conflicts ? I am not sure "unique index" helps since most people think of ON CONFLICT and not its implementation. Applied patch attached, URL updated:

Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2019-05-09 Thread Bruce Momjian
e release notes that it was odd we could control this via a CREATE TABLE option but not VACUUM. I have updated the release notes. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I.

Re: pg12 release notes

2019-05-09 Thread Bruce Momjian
that are implementation-behavior in the release notes? Usually we don't. Applied patch attached, docs updated: http://momjian.us/pgsql_docs/release-12.html -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are

Re: Problems with pg_upgrade and extensions referencing catalog tables/views

2019-05-09 Thread Bruce Momjian
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/bin/pg_upgrade/pg_upgrade.c;h=0b304bbd56ab0204396838618e86dfad757c2812;hb=HEAD It doesn't mention extensions. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-05-09 Thread Bruce Momjian
oes this by allowing specification of a encrypted data key by name or key_id. It might be necessary to allow decryption to try several versions of a key to see which one decrypts the data. While this is possible with PGP because there is a checksum payload, it isn't possible with AES256 because the inpu

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-05-09 Thread Bruce Momjian
don't think it is very valuable to use these options so reencryption will be easier. In many cases, taking any object offline might cause the application to fail, and having multiple encrypted data keys active will allow key replacement to be done on an as-needed basis. -- Bruce Momjian

Re: pg12 release notes

2019-05-10 Thread Bruce Momjian
On Thu, May 9, 2019 at 08:34:49PM -0500, Justin Pryzby wrote: > On Thu, May 09, 2019 at 09:02:51PM -0400, Bruce Momjian wrote: > > These were all very helpful. I adjusted your changes to create the > > attached applied patch. URL updated: > > Thanks. > >

Re: pg12 release notes

2019-05-11 Thread Bruce Momjian
On Sat, May 11, 2019 at 03:06:40AM +0100, Andrew Gierth wrote: > >>>>> "Bruce" == Bruce Momjian writes: > Bruce> Do I need more? > > That isn't quite how I'd have worded it, but I'm not sure what the best > wording is. Something like: > > * Ou

Re: pg12 release notes

2019-05-11 Thread Bruce Momjian
On Fri, May 10, 2019 at 07:31:21PM -0700, Peter Geoghegan wrote: > On Fri, May 10, 2019 at 7:11 PM Bruce Momjian wrote: > > > Whether or not you include more details is not what I care about here > > > -- I *agree* that this is insignificant. > > > Well,

Re: pg12 release notes

2019-05-11 Thread Bruce Momjian
On Sat, May 11, 2019 at 10:36:13AM -0400, Bruce Momjian wrote: > On Fri, May 10, 2019 at 07:31:21PM -0700, Peter Geoghegan wrote: > > > I think I have understood the nuances, as listed above --- I just don't > > > agree with the solution. > > > > To be c

Re: pg12 release notes

2019-05-10 Thread Bruce Momjian
On Thu, May 9, 2019 at 07:10:43PM -0700, Peter Geoghegan wrote: > On Thu, May 9, 2019 at 6:03 PM Bruce Momjian wrote: > > These were all very helpful. I adjusted your changes to create the > > attached applied patch. URL updated: > > I noticed that the compatibility not

Re: pg12 release notes

2019-05-10 Thread Bruce Momjian
On Fri, May 10, 2019 at 06:53:55PM -0700, Peter Geoghegan wrote: > On Fri, May 10, 2019 at 6:02 PM Bruce Momjian wrote: > > Have new btree indexes sort duplicate index entries in heap-storage > > order (Peter Geoghegan) > > > > This slightly

Re: pg12 release notes

2019-05-11 Thread Bruce Momjian
On Sat, May 11, 2019 at 11:47:42AM -0700, Peter Geoghegan wrote: > On Sat, May 11, 2019 at 11:02 AM Bruce Momjian wrote: > > OK, commit removed. > > You're mistaken -- nothing has been pushed to master in the last 3 hours. I am not mistaken. I have removed it from my local

Re: pg12 release notes

2019-05-11 Thread Bruce Momjian
On Sat, May 11, 2019 at 10:28:08AM -0700, Peter Geoghegan wrote: > On Sat, May 11, 2019 at 7:40 AM Bruce Momjian wrote: > > > > > I think I have understood the nuances, as listed above --- I just > > > > > don't > > > > > agree with the solution.

PG 12 draft release notes

2019-05-11 Thread Bruce Momjian
I have posted a draft copy of the PG 12 release notes here: http://momjian.us/pgsql_docs/release-12.html They are committed to git. It includes links to the main docs, where appropriate. Our official developer docs will rebuild in a few hours. -- Bruce Momjian http

Re: PG 12 draft release notes

2019-05-13 Thread Bruce Momjian
On Mon, May 13, 2019 at 10:08:57AM +0300, Oleg Bartunov wrote: > On Mon, May 13, 2019 at 6:52 AM Bruce Momjian wrote: > > > > On Sun, May 12, 2019 at 09:49:40AM -0400, Bruce Momjian wrote: > > > On Sun, May 12, 2019 at 10:00:38AM +0300, Oleg Bartunov wrote: > >

Re: remove doc/bug.template?

2019-05-13 Thread Bruce Momjian
src/tools/version_stamp.pl. > (Looks like there's a reference in .gitattributes, too) Yes, please remove. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: pg12 release notes

2019-05-13 Thread Bruce Momjian
t is: SELECT * will now output those columns for the many system tables which have them. Previously, the columns had to be selected explicitly. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: PG 12 draft release notes

2019-05-13 Thread Bruce Momjian
includes links to the main docs, where > > appropriate. Our official developer docs will rebuild in a few hours. > > Pgbench entry "Improve pgbench error reporting with clearer messages and > return codes" is by Peter Eisentraut, not me. I just reviewed it. Thanks, fixed

Re: PG 12 draft release notes

2019-05-22 Thread Bruce Momjian
On Tue, May 21, 2019 at 05:00:31PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Mon, May 20, 2019 at 08:48:15PM -0400, Tom Lane wrote: > >> Yes, this should be in "source code". I think it should be merged > >> with a391ff3c and 74dfe58a into somet

Re: pg12 release notes

2019-05-09 Thread Bruce Momjian
On Thu, May 9, 2019 at 07:13:35PM -0500, Justin Pryzby wrote: > On Thu, May 09, 2019 at 07:35:18PM -0400, Bruce Momjian wrote: > > I have made your other changes, with adjustment, patch attached. > > > > The results are here: > > > > http://momjian.us/pgsql

Re: PG 12 draft release notes

2019-05-21 Thread Bruce Momjian
On Tue, May 21, 2019 at 09:09:10AM -0700, Shawn Debnath wrote: > On Sat, May 11, 2019 at 04:33:24PM -0400, Bruce Momjian wrote: > > I have posted a draft copy of the PG 12 release notes here: > > > > http://momjian.us/pgsql_docs/release-12.html > > > > They a

Re: PG 12 draft release notes

2019-05-21 Thread Bruce Momjian
id Rowley, Dimitri Golgov > >otherwise? > > > >I think it might actually make sense to take David off this list, > >because his tableam work is essentially part of it's own entry, as > > Yeah, please take me off that one. My focus there was mostly on >

Re: PG 12 draft release notes

2019-05-21 Thread Bruce Momjian
turn RECORD >(Elvis Pranskevichus) > > You could link to "queries-tablefunctions" which describes the column > definition business; it's much more specific than "sql-createfunction". Done. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: PG 12 draft release notes

2019-05-21 Thread Bruce Momjian
n rare cases. > >> > >> > > It's probably worth mentioning, but I'd say something like > > Fix bugs that could cause ALTER TABLE DETACH PARTITION > to not drop objects that should be dropped, such as > automatically-created child indexes. > > The rest of it is not terribly interesting from a user's standpoint, > I think. Done. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: PG 12 draft release notes

2019-05-21 Thread Bruce Momjian
qOn Mon, May 20, 2019 at 03:17:19PM -0700, Andres Freund wrote: > Hi, > > Note that I've added a few questions to individuals involved with > specific points. If you're in the To: list, please search for your name. > > > On 2019-05-11 16:33:24 -0400, Bruce Momjian wro

Re: PG 12 draft release notes

2019-05-21 Thread Bruce Momjian
efits? > > It could stand to say something about the benefits. As I said, there > is already a little bit about the benefits, but that ended up being > tied to the "Improve speed of btree index insertions" item. Moving > that snippet to the correct item would be a good s

Re: PG 12 draft release notes

2019-05-14 Thread Bruce Momjian
On Tue, May 14, 2019 at 11:53:23AM +0200, nickb wrote: > On Sat, May 11, 2019, at 22:33, Bruce Momjian wrote: > > http://momjian.us/pgsql_docs/release-12.html > > There is a typo in E.1.3.1.1.: > > Expressions are evaluated at table partitioned table creation time.

Re: pg12 release notes

2019-05-10 Thread Bruce Momjian
On Fri, May 10, 2019 at 03:34:15PM +1200, David Rowley wrote: > On Fri, 10 May 2019 at 13:03, Bruce Momjian wrote: > > > > On Thu, May 9, 2019 at 07:13:35PM -0500, Justin Pryzby wrote: > > > On Thu, May 09, 2019 at 07:35:18PM -0400, Bruce Momjian wrote: > > >

Re: PG 12 draft release notes

2019-05-12 Thread Bruce Momjian
w path syntax. Is it a new storage format for JSON, or something else? I need help on this. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: Question about new pg12 feature no-rewrite timestamp to timestamptz conversion

2019-05-20 Thread Bruce Momjian
ime zone columns, the new data would take your local time and assume it was stored in UTC, which I don't think you want. I don't know of a way to make the adjustment you want without a table rewrite. It is unfortunate that the SQL standard requires timestamp _without_ time zone to be the default for 'ti

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread Bruce Momjian
d by the churn caused by this change, and have therefore questioned the value of it. Frankly, there has been so much churn I am unclear if it can be easily reverted. -- Bruce Momjian http://momjian.us EnterpriseDB htt

to_timestamp docs

2019-05-01 Thread Bruce Momjian
1976-01-01 00:00:00-05 Proposed patch attached. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription + diff --git

Re: to_timestamp docs

2019-05-01 Thread Bruce Momjian
On Wed, May 1, 2019 at 10:01:50AM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I don't think the changes made in PG 12 are documented accurately. > > That code is swapped out of my head at the moment, but it looks > to me like the para before the one you changed i

Re: Unhappy about API changes in the no-fsm-for-small-rels patch

2019-05-01 Thread Bruce Momjian
On Wed, May 1, 2019 at 09:08:54AM -0700, Andres Freund wrote: > So sure, there's a few typo fixes, one bugfix, and one buildfarm test > stability issue. Doesn't seem crazy for a nontrivial improvement. OK, my ignorant opinion was just based on the length of discussion threads. --

Re: to_timestamp docs

2019-05-01 Thread Bruce Momjian
On Wed, May 1, 2019 at 11:20:05PM +0300, Arthur Zakirov wrote: > Hello, > > On Wed, May 1, 2019 at 6:05 PM Bruce Momjian wrote: > > Thanks. I think I see the sentence you are thinking of: > > > >to_timestamp and to_date > >skip multi

Re: to_timestamp docs

2019-05-01 Thread Bruce Momjian
ch any of the allowed values for this > > field. > > Actually, FX takes effect on subsequent format patterns. This is not > documented, but it copycats Oracle behavior. Sure, normally FX should > be specified as the first item. We could document current behavior or > r

Re: Optimizer items in the release notes

2019-04-26 Thread Bruce Momjian
They are hard to explain I see the argument as wanting vague warm and fuzzy feelings that we are improving the optimizer, which we are. I will see what I can do to get those ideas into the PG 12 release notes in as concrete a way as possible. -- Bruce Momjian http://momjian.us

Re: Optimizer items in the release notes

2019-04-26 Thread Bruce Momjian
On Sat, Apr 27, 2019 at 02:04:33PM +1200, David Rowley wrote: > On Sat, 27 Apr 2019 at 11:49, Bruce Momjian wrote: > > > > On Wed, Apr 24, 2019 at 02:46:15PM -0400, Adam Brusselback wrote: > > > As a user, I am interested in the optimizer changes for sure, and I &

Naming of pg_checksums

2019-05-06 Thread Bruce Momjian
Is there a reason pg_checksums is plural and not singular, i.e., pg_checksum? I know it is being renamed for PG 12. It might have needed to be plural when it was pg_verify_checksums. -- Bruce Momjian http://momjian.us EnterpriseDB http

Re: finding changed blocks using WAL scanning

2019-04-18 Thread Bruce Momjian
On Thu, Apr 18, 2019 at 04:25:24PM -0400, Robert Haas wrote: > On Thu, Apr 18, 2019 at 3:51 PM Bruce Momjian wrote: > > How would you choose the STARTLSN/ENDLSN? If you could do it per > > checkpoint, rather than per-WAL, I think that would be great. > > I thought o

Re: finding changed blocks using WAL scanning

2019-04-18 Thread Bruce Momjian
y would probably still be happy to have a new > modified block file only, say, every 10 seconds. Agreed. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: proposal: psql PSQL_TABULAR_PAGER variable

2019-04-18 Thread Bruce Momjian
On Thu, Apr 18, 2019 at 06:06:40PM +0200, Pavel Stehule wrote: > > > čt 18. 4. 2019 v 17:58 odesílatel Bruce Momjian napsal: > > On Thu, Apr 18, 2019 at 05:45:24PM +0200, Pavel Stehule wrote: > > čt 18. 4. 2019 v 15:51 odesílatel Bruce Momjian > napsal: >

Re: block-level incremental backup

2019-04-18 Thread Bruce Momjian
On Thu, Apr 18, 2019 at 05:32:57PM +0200, David Fetter wrote: > On Wed, Apr 17, 2019 at 11:57:35AM -0400, Bruce Momjian wrote: > > Also, instead of storing the file name and block number in the modblock > > file, using the database oid, relfilenode, and block number (3 int32 >

Re: proposal: psql PSQL_TABULAR_PAGER variable

2019-04-18 Thread Bruce Momjian
On Thu, Apr 18, 2019 at 05:45:24PM +0200, Pavel Stehule wrote: > čt 18. 4. 2019 v 15:51 odesílatel Bruce Momjian napsal: > In testing pspg, it seems to work fine with tabular and \x-non-tabular > data.  Are you asking for a pager option that is only used for non-\x > displ

Re: finding changed blocks using WAL scanning

2019-04-20 Thread Bruce Momjian
be about the same as having one additional > standby, give or take. Good point. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: finding changed blocks using WAL scanning

2019-04-20 Thread Bruce Momjian
On Sat, Apr 20, 2019 at 12:09:42AM -0400, Robert Haas wrote: > On Thu, Apr 18, 2019 at 5:47 PM Bruce Momjian wrote: > > How would the modblock file record all the modified blocks across > > restarts and crashes? I assume that 1G of WAL would not be available > > for

Re: finding changed blocks using WAL scanning

2019-04-20 Thread Bruce Momjian
On Sat, Apr 20, 2019 at 04:17:08PM -0400, Robert Haas wrote: > On Sat, Apr 20, 2019 at 9:18 AM Bruce Momjian wrote: > > > I think you've got to prevent the WAL from being removed until a > > > .modblock file has been written. In more detail, you should (a) scan >

Re: Optimizer items in the release notes

2019-04-27 Thread Bruce Momjian
On Sat, Apr 27, 2019 at 02:47:44PM +1200, David Rowley wrote: > On Sat, 27 Apr 2019 at 14:22, Bruce Momjian wrote: > > > > * They are hard to explain > > > > > > That can be true, but we generally get there if not the first time > > > then afte

Re: Logging the feature of SQL-level read/write commits

2019-05-05 Thread Bruce Momjian
only physical block-level redo records. My blog entry covers some of this: https://momjian.us/main/blogs/pgblog/2019.html#March_6_2019 -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, s

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-05 Thread Bruce Momjian
On Fri, Jul 5, 2019 at 03:46:28PM -0400, Alvaro Herrera wrote: > On 2019-Jul-05, Bruce Momjian wrote: > > > What people really want with more-granular-than-cluster encryption is > > the ability to supply their passphrase key _when_ they want to access > > their data, an

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-07-05 Thread Bruce Momjian
, just like GIN, which is not really appropriate > for nbtree. I am also encouraged and am happy we can finally move this duplicate optimization forward. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As y

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-05 Thread Bruce Momjian
On Sun, Jun 16, 2019 at 03:57:46PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Sun, Jun 16, 2019 at 12:42:55PM -0400, Joe Conway wrote: > > > On 6/16/19 9:45 AM, Bruce Momjian wrote: > > > > On Sun, Jun 16,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-05 Thread Bruce Momjian
On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote: > On 6/15/19 9:28 PM, Bruce Momjian wrote: > > There are no known non-exhaustive plaintext attacks on AES: > > > > > > https://crypto.stackexchange.com/questions/1512/why-is-aes-resistant-to-known-

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-05 Thread Bruce Momjian
On Fri, Jul 5, 2019 at 04:10:04PM -0400, Bruce Momjian wrote: > On Fri, Jul 5, 2019 at 03:46:28PM -0400, Alvaro Herrera wrote: > > On 2019-Jul-05, Bruce Momjian wrote: > > > > > What people really want with more-granular-than-cluster encryption is > > > the a

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-05 Thread Bruce Momjian
On Fri, Jul 5, 2019 at 05:00:42PM -0400, Bruce Momjian wrote: > On Fri, Jul 5, 2019 at 04:24:54PM -0400, Alvaro Herrera wrote: > > On 2019-Jul-05, Bruce Momjian wrote: > > > > > Uh, well, you have the WAL record, and you want to write it to an 8k > > > page.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-05 Thread Bruce Momjian
On Fri, Jul 5, 2019 at 05:00:42PM -0400, Bruce Momjian wrote: > On Fri, Jul 5, 2019 at 04:24:54PM -0400, Alvaro Herrera wrote: > > On 2019-Jul-05, Bruce Momjian wrote: > > > > > Uh, well, you have the WAL record, and you want to write it to an 8k > > > page.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-05 Thread Bruce Momjian
On Fri, Jul 5, 2019 at 04:24:54PM -0400, Alvaro Herrera wrote: > On 2019-Jul-05, Bruce Momjian wrote: > > > Uh, well, you have the WAL record, and you want to write it to an 8k > > page. You have to read the 8k page from disk into shared buffers, and > > you have to d

Re: Missing PG 12 stable branch

2019-06-27 Thread Bruce Momjian
On Thu, Jun 27, 2019 at 10:28:40AM -0400, Bruce Momjian wrote: > Does anyone know why there is no PG 12 stable branch in our git tree? > > $ git branch -l > REL9_4_STABLE > REL9_5_STABLE > REL9_6_STABLE > REL_10_STABLE > REL_11

Missing PG 12 stable branch

2019-06-27 Thread Bruce Momjian
Does anyone know why there is no PG 12 stable branch in our git tree? $ git branch -l REL9_4_STABLE REL9_5_STABLE REL9_6_STABLE REL_10_STABLE REL_11_STABLE master They exist for earlier releases. Is this a problem? -- Bruce

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-11 Thread Bruce Momjian
On Thu, Jul 11, 2019 at 03:47:50PM -0400, Robert Haas wrote: > On Tue, Jul 9, 2019 at 10:43 AM Bruce Momjian wrote: > > FYI, pg_upgrade already preserves the pg_class.oid, which is why I > > recommended it over pg_class.relfilenode: > > I think it's strange that pg_upgr

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 02:44:30PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote: > > > > > > what is it that gets stored in the page for > > &g

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
that the overhead of this will be minimal for 8k page writes, and for WAL, we only need to generate the IV when we start a new 16MB segment, so again, minimal overhead. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: doc: minor update for description of "pg_roles" view

2019-07-10 Thread Bruce Momjian
do joins to other catalogs." > > I think it's superfluous to mention this now OIDs are exposed by default; > attached patch (for REL_12_STABLE and HEAD) simply removes this sentence. Patch applied though PG 12. Thanks. -- Bruce Momjian http://momjian.us EnterpriseDB

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 01:04:47PM -0400, Alvaro Herrera wrote: > On 2019-Jul-10, Bruce Momjian wrote: > > > * Using the LSN as part of the nonce fixes both problems, and has a > >third benefit: > > > > * We don't need to decrypt/re-encryp

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
f the nonce to create an IV. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
rs (pg_class.oid, LSN, page-number) that could be concatenated to be the IV, we could run those through a hash, or we could run them through the encryption function with the secret. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
case being discussed above is where we modify a page with a new row, write the page with an LSN, then change a hint bit on the page, and, if log_hint_bits is false, write the page with the hint bit change without changing the LSN since we didn't log hint bits. -- Bruce Momjian http://mom

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 03:53:55PM -0400, Alvaro Herrera wrote: > On 2019-Jul-10, Bruce Momjian wrote: > > > Good, so I think we all now agree we have to put the nonce > > (pg_class.oid, LSN, page-number) though the cipher using the secret. > > Actually, why do

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-10 Thread Bruce Momjian
a wrote: > > > >On 2019-Jul-10, Bruce Momjian wrote: > > > > > > > >>Uh, what if a transaction modifies page 0 and page 1 of the same table > > > >>--- don't those pages have the same LSN. > > > > > > > >No, because WAL being

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-07-11 Thread Bruce Momjian
Wow, I never thought of that. The only things I know we lock until transaction end are rows we update (against concurrent updates), and additions to unique indexes. By definition, indexes with many duplicates are not unique, so that doesn't apply. -- Bruce Momjian http://momjian.us Ent

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-11 Thread Bruce Momjian
rs (for WAL) do not overlap in numbering, for our nonce. I think we will eventually have to have this setup verified by security experts but I think we need to work through what is possible using existing Postgres facilities before we present a possible solution. -- Bruce Momjian http://momjian

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-11 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 12:26:24PM -0400, Bruce Momjian wrote: > On Wed, Jul 10, 2019 at 08:31:17AM -0400, Joe Conway wrote: > > Please see my other reply (and > > https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf > > appendix C as pointed o

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-11 Thread Bruce Momjian
On Thu, Jul 11, 2019 at 08:41:52PM -0400, Joe Conway wrote: > On 7/11/19 6:37 PM, Bruce Momjian wrote: > > Our first implementation will encrypt the entire cluster. We can later > > consider encryption per table or tablespace. It is unclear if > > encrypting differen

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-11 Thread Bruce Momjian
On Thu, Jul 11, 2019 at 06:37:41PM -0400, Bruce Momjian wrote: > wal_log_hints will be enabled automatically in encryption mode, like we > do for checksum mode, so we never encrypt different 8k pages with the > same IV. Checksum mode can be enabled in encrypted clusters to detect modi

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-12 Thread Bruce Momjian
LSN. You are asking if an update that just expires one row and adds it to a new page gets the same LSN. I don't know. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-12 Thread Bruce Momjian
On Fri, Jul 12, 2019 at 03:20:37PM +0900, Masahiko Sawada wrote: > Thank you for summarizing the discussion, it's really helpful. I'll > update the wiki page based on the summary. > > On Fri, Jul 12, 2019 at 10:05 AM Bruce Momjian wrote: > > > The keys themselves shoul

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-12 Thread Bruce Momjian
On Fri, Jul 12, 2019 at 07:26:21AM -0400, Sehrope Sarkuni wrote: > On Thu, Jul 11, 2019 at 9:05 PM Bruce Momjian wrote: > > > > On Thu, Jul 11, 2019 at 08:41:52PM -0400, Joe Conway wrote: > > > I vote for AES 256 rather than 128. > > > > Why? This p

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-12 Thread Bruce Momjian
ut security, I did not consider performance > impacts in my decision. Thank you for this exhaustive research. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 06:43:31PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jul 8, 2019 at 06:04:46PM -0400, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > On Mon, Jul

Re: Increasing default value for effective_io_concurrency?

2019-07-08 Thread Bruce Momjian
- presumably unloaded - would be a good guide to what > to use in a production situation. Maybe it would; I'm just not sure. I think it would be better than what we have now. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + As y

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-07-08 Thread Bruce Momjian
On Mon, Jul 8, 2019 at 06:04:46PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jul 8, 2019 at 05:41:51PM -0400, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > Well, if it wa

<    1   2   3   4   5   6   7   8   9   10   >