Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2020-02-01 Thread Sehrope Sarkuni
On Fri, Jan 31, 2020 at 1:21 AM Masahiko Sawada wrote: > On Thu, 30 Jan 2020 at 20:36, Sehrope Sarkuni wrote: > > That > > would allow the internal usage to have a fixed output length of > > LEN(IV) + LEN(HMAC) + LEN(DATA) = 16 + 32 + 64 = 112 bytes. > > Probably you meant LEN(DATA) is 32? DATA

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2020-01-30 Thread Masahiko Sawada
On Thu, 30 Jan 2020 at 20:36, Sehrope Sarkuni wrote: > > On Wed, Jan 29, 2020 at 3:43 AM Masahiko Sawada > wrote: > > Thank you for the suggestion. I like your suggestion. We can do an > > integrity check of the user input wrapped key by using HMAC when > > unwrapping. Regarding the output

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2020-01-30 Thread Sehrope Sarkuni
On Wed, Jan 29, 2020 at 3:43 AM Masahiko Sawada wrote: > Thank you for the suggestion. I like your suggestion. We can do an > integrity check of the user input wrapped key by using HMAC when > unwrapping. Regarding the output format you meant to use aes-256 > rather than aes-256-key-wrap? I think

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2020-01-29 Thread Masahiko Sawada
On Sun, 26 Jan 2020 at 01:35, Sehrope Sarkuni wrote: > > Hi, > > I took a look at this patch. With some additions I think the feature > itself is useful but the patch needs more work. It also doesn't have > any of its own automated tests yet so the testing below was done > manually. > > The

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2020-01-25 Thread Sehrope Sarkuni
Hi, I took a look at this patch. With some additions I think the feature itself is useful but the patch needs more work. It also doesn't have any of its own automated tests yet so the testing below was done manually. The attached file, kms_v2.patch, is a rebased version of the kms_v1.patch that

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2020-01-09 Thread cary huang
On Mon, Jan 6, 2020 at 4:43 AM Masahiko Sawada < masahiko.saw...@2ndquadrant.com> wrote: > On Sat, 4 Jan 2020 at 15:11, cary huang wrote: > >> > >> Hello Sawada and all > >> > >> I would like to elaborate more on Sehrope and Sawada's discussion on > passing NULL IV in "pg_cipher_encrypt/decrypt"

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2020-01-06 Thread Masahiko Sawada
On Sat, 4 Jan 2020 at 15:11, cary huang wrote: > > Hello Sawada and all > > I would like to elaborate more on Sehrope and Sawada's discussion on passing > NULL IV in "pg_cipher_encrypt/decrypt" functions during kmgr_wrap_key and > kmgr_unwrap_key routines in kmgr_utils.c. Openssl implements key

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2020-01-03 Thread cary huang
Hello Sawada and all I would like to elaborate more on Sehrope and Sawada's discussion on passing NULL IV in "pg_cipher_encrypt/decrypt" functions during kmgr_wrap_key and kmgr_unwrap_key routines in kmgr_utils.c. Openssl implements key wrap according to RFC3394 as Sawada mentioned and passing

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-12-31 Thread Ibrar Ahmed
On Tue, Dec 31, 2019 at 1:05 PM Masahiko Sawada wrote: > On Sun, Dec 1, 2019 at 12:03 PM Michael Paquier > wrote: > > > > On Fri, Nov 01, 2019 at 09:38:37AM +0900, Moon, Insung wrote: > > > Of course, I may not have written the excellent quality code > > > correctly, so I will make an interim

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-12-31 Thread Masahiko Sawada
On Sun, Dec 1, 2019 at 12:03 PM Michael Paquier wrote: > > On Fri, Nov 01, 2019 at 09:38:37AM +0900, Moon, Insung wrote: > > Of course, I may not have written the excellent quality code > > correctly, so I will make an interim report if possible. > > The last patch has rotten, and does not apply

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-30 Thread Michael Paquier
On Fri, Nov 01, 2019 at 09:38:37AM +0900, Moon, Insung wrote: > Of course, I may not have written the excellent quality code > correctly, so I will make an interim report if possible. The last patch has rotten, and does not apply anymore. A rebase would be nice, so I am switching the patch as

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-11 Thread Antonin Houska
Robert Haas wrote: > On Sat, Nov 2, 2019 at 8:23 AM Antonin Houska wrote: > > Change to hint bits does not result in LSN change in the case I described > > here > > > > https://www.postgresql.org/message-id/28452.1572443058%40antos > > > > but I consider this a bug (BTW, I discovered this

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-10 Thread shawn wang
Hi hackers, By arrange, I will complete the modification of the front-end tool to support TDE. Now I have completed the modification of the pg_waldump, pg_resetwal, and pg_rewind tools. My design: 1. Add two options, -D and -c, to the front-end tools. You can use -c to get a password of the user

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-06 Thread Bruce Momjian
On Sat, Nov 2, 2019 at 01:34:41PM +0100, Antonin Houska wrote: > Robert Haas wrote: > > > On Tue, Aug 6, 2019 at 10:36 AM Bruce Momjian wrote: > > > Seems reasonable (not that I am an encryption expert). > > > > > For WAL, we effectively create a 16MB bitstream, though we can create it > > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-02 Thread Masahiko Sawada
On Sat, 2 Nov 2019 at 21:33, Antonin Houska wrote: > Robert Haas wrote: > > > On Tue, Aug 6, 2019 at 10:36 AM Bruce Momjian wrote: > > > Seems reasonable (not that I am an encryption expert). > > > > > For WAL, we effectively create a 16MB bitstream, though we can create > it > > > in parts as

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-02 Thread Antonin Houska
Robert Haas wrote: > On Tue, Aug 6, 2019 at 10:36 AM Bruce Momjian wrote: > Seems reasonable (not that I am an encryption expert). > > > For WAL, we effectively create a 16MB bitstream, though we can create it > > in parts as needed. (Creating it in parts is easier in CTR mode.) The > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-02 Thread Antonin Houska
Robert Haas wrote: > On Mon, Aug 5, 2019 at 8:44 PM Bruce Momjian wrote: > > Right. The 8k page LSN changes each time the page is modified, and the > > is part of the page nonce. > > What about hint bit changes? > > I think even with wal_log_hints=on, it's not the case that *every* > change

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-01 Thread Robert Haas
On Tue, Aug 6, 2019 at 10:36 AM Bruce Momjian wrote: > OK, I think you are missing something. Let me go over the details. > First, I think we are all agreed we are using CTR for heap/index pages, > and for WAL, because CTR allows byte granularity, it is faster, and > might be more secure. > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-11-01 Thread Robert Haas
On Mon, Aug 5, 2019 at 8:44 PM Bruce Momjian wrote: > Right. The 8k page LSN changes each time the page is modified, and the > is part of the page nonce. What about hint bit changes? I think even with wal_log_hints=on, it's not the case that *every* change to hint bits results in an LSN

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-10-31 Thread Moon, Insung
Hello. On Thu, Oct 31, 2019 at 11:25 PM Masahiko Sawada wrote: > > On Fri, Sep 6, 2019 at 3:34 PM Smith, Peter > wrote: > > > > -Original Message- > > From: Masahiko Sawada Sent: Thursday, 15 August > > 2019 7:10 PM > > > > > BTW I've created PoC patch for cluster encryption feature.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-10-31 Thread Masahiko Sawada
On Fri, Sep 6, 2019 at 3:34 PM Smith, Peter wrote: > > -Original Message- > From: Masahiko Sawada Sent: Thursday, 15 August 2019 > 7:10 PM > > > BTW I've created PoC patch for cluster encryption feature. Attached patch > > set has done some items of TODO list and some of them can be

RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-09-06 Thread Smith, Peter
-Original Message- From: Masahiko Sawada Sent: Thursday, 15 August 2019 7:10 PM > BTW I've created PoC patch for cluster encryption feature. Attached patch set > has done some items of TODO list and some of them can be used even for finer > granularity encryption. Anyway, the

RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-26 Thread Smith, Peter
Greetings, (Apologies for any naïve thoughts below. Please correct my misunderstandings) I am trying to understand the background for the ideas proposed and/or already decided, but it is increasingly difficult to follow. I’ve been watching the TDE list for several months and over that time

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-26 Thread Joe Conway
On 8/26/19 2:53 AM, Masahiko Sawada wrote: > I guess that this depends on the number of encryption keys we use. If > we have encryption keys per tablespace or database the number of keys > would be at most several dozen or several hundred. It's enough to have > them in flat-file format on the disk

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-26 Thread Masahiko Sawada
On Mon, Aug 26, 2019 at 7:49 PM Joe Conway wrote: > > On 8/26/19 2:53 AM, Masahiko Sawada wrote: > > I guess that this depends on the number of encryption keys we use. If > > we have encryption keys per tablespace or database the number of keys > > would be at most several dozen or several

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-26 Thread Masahiko Sawada
On Fri, Aug 23, 2019 at 11:35 PM Stephen Frost wrote: > > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Fri, Aug 23, 2019 at 07:45:22AM -0400, Stephen Frost wrote: > > > Having listed out the feature set of each of the other major databases > > > when it comes to TDE is exactly

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-25 Thread Moon, Insung
Dear Hackers. > Specifically in this case- I went back and tried to figure out what > other database systems have an "encrypt EVERYTHING" option. I didn't > have much luck finding one though. So I think we need to ask ourselves- > the "check box" that we're trying to check off with TDE, do the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-25 Thread Moon, Insung
Dear Hackers. It's been a long time since I sent a mail. On Sat, Aug 24, 2019 at 9:27 AM Bruce Momjian wrote: > On Fri, Aug 23, 2019 at 10:35:17AM -0400, Stephen Frost wrote: > > > Agreed. The features of other databases are a clear source for what we > > > should consider and run through the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-24 Thread Bruce Momjian
On Fri, Aug 23, 2019 at 10:04:13PM -0400, Stephen Frost wrote: > > Well, I think they might do that to reduce encryption overhead. I think > > tests have shown that is not an issue, but we will need to test further. > > I seriously doubt that's why and I don't think there's actually much > value

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-23 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Aug 23, 2019 at 10:35:17AM -0400, Stephen Frost wrote: > > Following on from that- when other databases don't have something that > > we're thinking about implementing, maybe we should be contemplating if > > it really makes sense as

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-23 Thread Bruce Momjian
On Fri, Aug 23, 2019 at 10:35:17AM -0400, Stephen Frost wrote: > > Agreed. The features of other databases are a clear source for what we > > should consider and run through the useful/reasonable filter. > > Following on from that- when other databases don't have something that > we're thinking

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-23 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Aug 23, 2019 at 07:45:22AM -0400, Stephen Frost wrote: > > Having listed out the feature set of each of the other major databases > > when it comes to TDE is exactly how we objectively look at what is being > > done in the industry,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-23 Thread Bruce Momjian
On Fri, Aug 23, 2019 at 07:45:22AM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Sat, Aug 17, 2019 at 01:52:17PM -0400, Stephen Frost wrote: > > > Being PostgreSQL, I would expect us to shoot for as much flexibility as > > > we possible, similar to

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-23 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Sat, Aug 17, 2019 at 01:52:17PM -0400, Stephen Frost wrote: > > Being PostgreSQL, I would expect us to shoot for as much flexibility as > > we possible, similar to what we've done for our ACL system where we > > support down to a

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-23 Thread Bruce Momjian
On Sat, Aug 17, 2019 at 01:52:17PM -0400, Stephen Frost wrote: > Being PostgreSQL, I would expect us to shoot for as much flexibility as > we possible, similar to what we've done for our ACL system where we > support down to a column-level (and row level with RLS). > > That's our target end-goal.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-19 Thread Ahsan Hadi
I have shared a calendar invite for TDE/KMS weekly meeting with the members who expressed interest of joining the meeting in this chain. Hopefully I haven't missed anyone. I am not aware of everyone's timezone but I have tried to setup a time that's not very inconvenient. It won't be ideal for

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-19 Thread Joe Conway
On 8/19/19 8:51 AM, Ahsan Hadi wrote: > I have shared a calendar invite for TDE/KMS weekly meeting with the > members who expressed interest of joining the meeting in this chain. > Hopefully I haven't missed anyone. > > I am not aware of everyone's timezone but I have tried to setup a time >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-18 Thread Ahsan Hadi
On Mon, 19 Aug 2019 at 6:23 AM, Smith, Peter wrote: > > From: Ibrar Ahmed Sent: Sunday, 18 August 2019 > 2:43 AM > > +1 for voice call, bruce we usually have a weekly TDE call. I will > include you in > > If you don't mind, please also add me to that TDE call list. > Sure will do. >

RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-18 Thread Smith, Peter
> -Original Message- > From: Stephen Frost Sent: Friday, 16 August 2019 11:01 AM > Having direct integration with a KMS would certainly be valuable, and I don't > see a reason to deny users that option if someone would like to spend time > implementing it- in addition to a simpler

RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-18 Thread Smith, Peter
> From: Ibrar Ahmed Sent: Sunday, 18 August 2019 2:43 AM > +1 for voice call, bruce we usually have a weekly TDE call. I will include > you in If you don't mind, please also add me to that TDE call list. Thanks/Regards, --- Peter Smith Fujitsu Australia

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-18 Thread Peter Eisentraut
On 2019-08-17 08:16, Antonin Houska wrote: > One problem that occurs to me is that PG may need to send some sort of > credentials to the KMS. If it runs a separate process to execute the command, > it needs to pass those credentials to it. Whether it does so via parameters or > environment

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Stephen Frost
Greetings, On Sat, Aug 17, 2019 at 18:30 Ahsan Hadi wrote: > The current calendar entry for TDE weekly call will not work for EST > timezone. I will change the invite so we can accommodate people from > multiple time zones. > I appreciate the thought but at least for my part, I already have

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Ahsan Hadi
The current calendar entry for TDE weekly call will not work for EST timezone. I will change the invite so we can accommodate people from multiple time zones. Stay tuned. On Sun, 18 Aug 2019 at 2:29 AM, Sehrope Sarkuni wrote: > On Sat, Aug 17, 2019 at 12:43 PM Ibrar Ahmed > wrote: > >> +1

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Sehrope Sarkuni
On Sat, Aug 17, 2019 at 12:43 PM Ibrar Ahmed wrote: > +1 for voice call, bruce we usually have a weekly TDE call. > Please add me to the call as well. Thanks! Regards, -- Sehrope Sarkuni Founder & CEO | JackDB, Inc. | https://www.jackdb.com/

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > I will state whet I have already told some people privately, that for > this feature, we have many people understanding 40% of the problem, but > thinking they understand 90%. I do agree we should plan for our > eventual full feature set,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Sat, Aug 17, 2019 at 08:16:06AM +0200, Antonin Houska wrote: > > Bruce Momjian wrote: > > > > > On Thu, Aug 15, 2019 at 09:01:05PM -0400, Stephen Frost wrote: > > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > > Why would it not

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Stephen Frost
Greetings, * Ibrar Ahmed (ibrar.ah...@gmail.com) wrote: > On Sat, Aug 17, 2019 at 3:04 AM Bruce Momjian wrote: > > +1 for voice call, bruce we usually have a weekly TDE call. I will include > you in > that call. Currently, in that group are > moon_insung...@lab.ntt.co.jp, >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Ibrar Ahmed
On Sat, Aug 17, 2019 at 3:04 AM Bruce Momjian wrote: > On Fri, Aug 16, 2019 at 07:47:37PM +0200, Antonin Houska wrote: > > Bruce Momjian wrote: > > > > > I have seen no one present a clear description of how anything beyond > > > all-cluster encryption would work or be secure. Wishing that

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Bruce Momjian
On Sat, Aug 17, 2019 at 08:16:06AM +0200, Antonin Houska wrote: > Bruce Momjian wrote: > > > On Thu, Aug 15, 2019 at 09:01:05PM -0400, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > Why would it not be simpler to have the cluster_passphrase_command run > > > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Bruce Momjian
On Fri, Aug 16, 2019 at 06:04:39PM -0400, Bruce Momjian wrote: > I suggest we schedule a voice call and I will go over all the issues and > explain why I came to the conclusions listed. It is hard to know what > level of detail to explain that in an email, beyond what I have already > posted on

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-17 Thread Antonin Houska
Bruce Momjian wrote: > On Thu, Aug 15, 2019 at 09:01:05PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > Why would it not be simpler to have the cluster_passphrase_command run > > > whatever command-line program it wants? If you don't want to use a > > > shell

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Bruce Momjian
On Fri, Aug 16, 2019 at 07:47:37PM +0200, Antonin Houska wrote: > Bruce Momjian wrote: > > > I have seen no one present a clear description of how anything beyond > > all-cluster encryption would work or be secure. Wishing that were not > > the case doesn't change things. > > Since this email

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Antonin Houska
Bruce Momjian wrote: > I have seen no one present a clear description of how anything beyond > all-cluster encryption would work or be secure. Wishing that were not > the case doesn't change things. Since this email thread has grown a lot and is difficult to follow, it might help if we

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Bruce Momjian
On Fri, Aug 16, 2019 at 05:58:59PM +0500, Ibrar Ahmed wrote: > > > On Thu, Aug 15, 2019 at 8:21 PM Bruce Momjian wrote: > > On Thu, Aug 15, 2019 at 11:24:46AM +0200, Antonin Houska wrote: > > > I think there are several directions we can go after all-cluster > > > encryption, >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Bruce Momjian
On Thu, Aug 15, 2019 at 09:01:05PM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > I assume you are talking about my option #1. I can see if you only need > > a few tables encrypted, e.g., credit card numbers, it can be excessive > > to encrypt the entire cluster. (I

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-16 Thread Ibrar Ahmed
On Thu, Aug 15, 2019 at 8:21 PM Bruce Momjian wrote: > On Thu, Aug 15, 2019 at 11:24:46AM +0200, Antonin Houska wrote: > > > I think there are several directions we can go after all-cluster > > > encryption, > > > > I think I misunderstood. What you summarize in > > > > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-15 Thread Masahiko Sawada
On Fri, Aug 16, 2019 at 10:01 AM Stephen Frost wrote: > > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Thu, Aug 15, 2019 at 06:10:24PM +0900, Masahiko Sawada wrote: > > > On Thu, Aug 15, 2019 at 10:19 AM Bruce Momjian wrote: > > > > > > > > On Wed, Aug 14, 2019 at 04:36:35PM

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-15 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Aug 15, 2019 at 06:10:24PM +0900, Masahiko Sawada wrote: > > On Thu, Aug 15, 2019 at 10:19 AM Bruce Momjian wrote: > > > > > > On Wed, Aug 14, 2019 at 04:36:35PM +0200, Antonin Houska wrote: > > > > I can work on it right away but

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-15 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Aug 15, 2019 at 11:24:46AM +0200, Antonin Houska wrote: > > > I think there are several directions we can go after all-cluster > > > encryption, > > > > I think I misunderstood. What you summarize in > > > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-15 Thread Bruce Momjian
On Thu, Aug 15, 2019 at 11:24:46AM +0200, Antonin Houska wrote: > > I think there are several directions we can go after all-cluster > > encryption, > > I think I misunderstood. What you summarize in > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-15 Thread Bruce Momjian
On Thu, Aug 15, 2019 at 06:10:24PM +0900, Masahiko Sawada wrote: > On Thu, Aug 15, 2019 at 10:19 AM Bruce Momjian wrote: > > > > On Wed, Aug 14, 2019 at 04:36:35PM +0200, Antonin Houska wrote: > > > I can work on it right away but don't know where to start. > > > > I think the big open question

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-15 Thread Bruce Momjian
On Wed, Aug 14, 2019 at 09:19:44PM -0400, Bruce Momjian wrote: > I think there are several directions we can go after all-cluster > encryption, and it does matter because we would want minimal API > breakage. The options are: > > 1) Allow per-table encyption control to limit encryption

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-15 Thread Antonin Houska
Bruce Momjian wrote: > On Wed, Aug 14, 2019 at 04:36:35PM +0200, Antonin Houska wrote: > > I can work on it right away but don't know where to start. > > I think the big open question is whether there will be acceptance of an > all-cluster encyption feature. I guess if no one objects, we can

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-15 Thread Masahiko Sawada
On Thu, Aug 15, 2019 at 10:19 AM Bruce Momjian wrote: > > On Wed, Aug 14, 2019 at 04:36:35PM +0200, Antonin Houska wrote: > > I can work on it right away but don't know where to start. > > I think the big open question is whether there will be acceptance of an > all-cluster encyption feature. I

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-14 Thread Bruce Momjian
On Wed, Aug 14, 2019 at 04:36:35PM +0200, Antonin Houska wrote: > I can work on it right away but don't know where to start. I think the big open question is whether there will be acceptance of an all-cluster encyption feature. I guess if no one objects, we can move forward. > First, I think we

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-14 Thread Antonin Houska
Bruce Momjian wrote: > On Sat, Aug 10, 2019 at 08:06:17AM -0400, Bruce Momjian wrote: > > So, I just had an indea if we use separate encryption keys for > > heap/index and for WAL --- we already know we will have an offline tool > > that can rotate the passphrase or encryption keys. If we allow

RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-13 Thread Smith, Peter
On Sat, Aug 10, 2019 at 1:19 AM Tomas Vondra wrote: > We can of course support "forced" re-encryption, but I think it's acceptable > if that's fairly expensive as long as it can be throttled and executed in the > background (kinda similar to the patch to enable checksums in the background).

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-13 Thread Masahiko Sawada
On Sat, Aug 10, 2019 at 12:18 AM Tomas Vondra wrote: > > On Fri, Aug 09, 2019 at 11:51:23PM +0900, Masahiko Sawada wrote: > >On Fri, Aug 9, 2019 at 10:25 AM Bruce Momjian wrote: > >> > >> On Thu, Aug 8, 2019 at 06:31:42PM -0400, Stephen Frost wrote: > >> > > >Crash recovery doesn't happen "all

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Sat, Aug 10, 2019 at 08:06:17AM -0400, Bruce Momjian wrote: > So, I just had an indea if we use separate encryption keys for > heap/index and for WAL --- we already know we will have an offline tool > that can rotate the passphrase or encryption keys. If we allow the > encryption keys to be

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > On Mon, Aug 5, 2019 at 9:02 PM Bruce Momjian wrote: > I don't think we want to add a MAC at this point since the MAC for 8k > pages seems unattainable. > > Even without a per-page MAC, a MAC at some level for WAL has its

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Wed, Jul 31, 2019 at 09:43:00AM -0400, Sehrope Sarkuni wrote: > On Wed, Jul 31, 2019 at 2:32 AM Masahiko Sawada wrote: > For WAL encryption,  before flushing WAL we encrypt whole 8k WAL page > and then write only the encrypted data of the new WAL record using > pg_pwrite() rather

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Tue, Jul 30, 2019 at 04:48:31PM -0400, Bruce Momjian wrote: > I am not even clear if pg_upgrade preserving relfilenode is possible --- > when we wrap the relfilenode counter, does it start at 1 or at the > first-user-relation-oid? If the former, it could conflict with oids > assigned to new

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Thu, Jul 25, 2019 at 11:30:55PM -0400, Alvaro Herrera wrote: > On 2019-Jul-25, Alvaro Herrera wrote: > > On the other hand if the Key and IV are reused between messages then > > the same plaintext will lead to the same ciphertext, so you can > > potentially decrypt a message using a

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Bruce Momjian
On Wed, Jul 10, 2019 at 08:07:49PM -0400, Bruce Momjian wrote: > On Thu, Jul 11, 2019 at 12:18:47AM +0200, Tomas Vondra wrote: > > On Wed, Jul 10, 2019 at 06:04:30PM -0400, Stephen Frost wrote: > > > Greetings, > > > > > > * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > > > > On Wed, Jul

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-12 Thread Antonin Houska
Masahiko Sawada wrote: > Attached the draft version patch sets of per tablespace transparent > data at rest encryption. The patch doesn't support full functionality, > it includes: > > * Per tablespace encryption > * Encryption and decryption buffer data when disk I/O. > * 2 tier key hierarchy

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-10 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > On Mon, Aug 5, 2019 at 9:02 PM Bruce Momjian wrote: > I was thinking the WAL would use the same key since the nonce is unique > between the two.  What value is there in using a different key? > Never having to worry about

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-10 Thread Bruce Momjian
On Fri, Aug 9, 2019 at 10:54:51PM -0400, Bruce Momjian wrote: > On Thu, Aug 8, 2019 at 10:17:53PM -0400, Sehrope Sarkuni wrote: > > On Thu, Aug 8, 2019 at 2:16 PM Bruce Momjian wrote: > > > > On Wed, Aug  7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > > > Simplest approach for

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Bruce Momjian
On Thu, Aug 8, 2019 at 10:17:53PM -0400, Sehrope Sarkuni wrote: > On Thu, Aug 8, 2019 at 2:16 PM Bruce Momjian wrote: > > On Wed, Aug  7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > > Simplest approach for derived keys would be to use immutable attributes > of the > > WAL

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Bruce Momjian
On Fri, Aug 9, 2019 at 05:01:47PM +0200, Tomas Vondra wrote: > I know there were proposals to keep it encrypted in shared buffers, but > I'm not sure that's what we'll end up doing (I have not followed the > recent discussion all that closely, though). There is no plan to encrypt shared buffers.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Bruce Momjian
On Thu, Aug 8, 2019 at 10:34:26PM -0400, Sehrope Sarkuni wrote: > On Thu, Aug 8, 2019 at 6:31 PM Stephen Frost wrote: > > Strictly speaking, that isn't actually crash recovery, it's physical > replication / HA, and while those are certainly nice to have it's no > guarantee that

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Bruce Momjian
On Fri, Aug 9, 2019 at 11:51:23PM +0900, Masahiko Sawada wrote: > I agree that cluster-wide is more simpler but I'm not sure that it > meets real needs from users. One example is re-encryption; when the > key leakage happens, in cluster-wide encryption we end up with doing > re-encrypt whole

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Tomas Vondra
On Fri, Aug 09, 2019 at 11:51:23PM +0900, Masahiko Sawada wrote: On Fri, Aug 9, 2019 at 10:25 AM Bruce Momjian wrote: On Thu, Aug 8, 2019 at 06:31:42PM -0400, Stephen Frost wrote: > > >Crash recovery doesn't happen "all the time" and neither does vacuum > > >freeze, and autovacuum processes

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Tomas Vondra
On Thu, Aug 08, 2019 at 06:31:42PM -0400, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: On Thu, Aug 08, 2019 at 03:07:59PM -0400, Stephen Frost wrote: >* Bruce Momjian (br...@momjian.us) wrote: >>On Tue, Jul 9, 2019 at 11:09:01AM -0400, Bruce Momjian

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-09 Thread Masahiko Sawada
On Fri, Aug 9, 2019 at 10:25 AM Bruce Momjian wrote: > > On Thu, Aug 8, 2019 at 06:31:42PM -0400, Stephen Frost wrote: > > > >Crash recovery doesn't happen "all the time" and neither does vacuum > > > >freeze, and autovacuum processes are independent of individual client > > > >backends- we

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Sehrope Sarkuni
On Thu, Aug 8, 2019 at 6:31 PM Stephen Frost wrote: > Strictly speaking, that isn't actually crash recovery, it's physical > replication / HA, and while those are certainly nice to have it's no > guarantee that they're required or that you'd want to have the same keys > for them- conceptually,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Sehrope Sarkuni
On Thu, Aug 8, 2019 at 2:16 PM Bruce Momjian wrote: > On Wed, Aug 7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > > Simplest approach for derived keys would be to use immutable attributes > of the > > WAL files as an input to the key derivation. Something like HKDF(MDEK, > "WAL:" | > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
On Thu, Aug 8, 2019 at 06:31:42PM -0400, Stephen Frost wrote: > > >Crash recovery doesn't happen "all the time" and neither does vacuum > > >freeze, and autovacuum processes are independent of individual client > > >backends- we don't need to (and shouldn't) have the keys in shared > > >memory. >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
On Thu, Aug 8, 2019 at 03:07:59PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Tue, Jul 9, 2019 at 11:09:01AM -0400, Bruce Momjian wrote: > > > On Tue, Jul 9, 2019 at 10:59:12AM -0400, Stephen Frost wrote: > > > > * Bruce Momjian

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > On Thu, Aug 08, 2019 at 03:07:59PM -0400, Stephen Frost wrote: > >* Bruce Momjian (br...@momjian.us) wrote: > >>On Tue, Jul 9, 2019 at 11:09:01AM -0400, Bruce Momjian wrote: > >>> On Tue, Jul 9, 2019 at 10:59:12AM -0400, Stephen

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Tomas Vondra
On Thu, Aug 08, 2019 at 03:07:59PM -0400, Stephen Frost wrote: Greetings, * Bruce Momjian (br...@momjian.us) wrote: On Tue, Jul 9, 2019 at 11:09:01AM -0400, Bruce Momjian wrote: > On Tue, Jul 9, 2019 at 10:59:12AM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > >

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jul 9, 2019 at 11:09:01AM -0400, Bruce Momjian wrote: > > On Tue, Jul 9, 2019 at 10:59:12AM -0400, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > I agree that all of that isn't necessary for an initial

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
On Tue, Jul 9, 2019 at 11:09:01AM -0400, Bruce Momjian wrote: > On Tue, Jul 9, 2019 at 10:59:12AM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > I agree that all of that isn't necessary for an initial implementation, > > I was rather trying to lay out how we could

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 07:40:05PM -0400, Sehrope Sarkuni wrote: > With the provisos above, yes I think that would work though I don't think it's > a good idea. Better to start off using the functions directly and then look > into optimizing only if they're a bottleneck. As a first pass I'd break

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-08 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 08:56:18AM -0400, Sehrope Sarkuni wrote: > On Mon, Aug 5, 2019 at 9:02 PM Bruce Momjian wrote: > > On Wed, Jul 31, 2019 at 09:25:01AM -0400, Sehrope Sarkuni wrote: > > Even if we do not include a separate per-relation salt or things like > > relfilenode when

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-07 Thread Sehrope Sarkuni
On Wed, Aug 7, 2019 at 1:39 PM Bruce Momjian wrote: > On Wed, Aug 7, 2019 at 11:41:51AM -0400, Sehrope Sarkuni wrote: > > On Wed, Aug 7, 2019 at 7:19 AM Bruce Momjian wrote: > > > > On Wed, Aug 7, 2019 at 05:13:31PM +0900, Masahiko Sawada wrote: > > > I understood. IIUC in your

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-07 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 11:41:51AM -0400, Sehrope Sarkuni wrote: > On Wed, Aug 7, 2019 at 7:19 AM Bruce Momjian wrote: > > On Wed, Aug  7, 2019 at 05:13:31PM +0900, Masahiko Sawada wrote: > > I understood. IIUC in your approach postgres processes encrypt WAL > > records when

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-07 Thread Sehrope Sarkuni
On Wed, Aug 7, 2019 at 7:19 AM Bruce Momjian wrote: > On Wed, Aug 7, 2019 at 05:13:31PM +0900, Masahiko Sawada wrote: > > I understood. IIUC in your approach postgres processes encrypt WAL > > records when inserting to the WAL buffer. So WAL data is encrypted > > even on the WAL buffer. > I

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-07 Thread Sehrope Sarkuni
On Mon, Aug 5, 2019 at 9:02 PM Bruce Momjian wrote: > On Wed, Jul 31, 2019 at 09:25:01AM -0400, Sehrope Sarkuni wrote: > > Even if we do not include a separate per-relation salt or things like > > relfilenode when generating a derived key, we can still include other > types of > > immutable

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-07 Thread Bruce Momjian
On Wed, Aug 7, 2019 at 05:13:31PM +0900, Masahiko Sawada wrote: > I understood. IIUC in your approach postgres processes encrypt WAL > records when inserting to the WAL buffer. So WAL data is encrypted > even on the WAL buffer. > > It works but I think the implementation might be complex; For

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-08-07 Thread Masahiko Sawada
On Wed, Aug 7, 2019 at 2:55 AM Bruce Momjian wrote: > > On Wed, Aug 7, 2019 at 12:31:58AM +0900, Masahiko Sawada wrote: > > Well, so you mean that for example we encrypt only 100 bytes WAL > > record when append 100 bytes WAL records? > > > > For WAL encryption, if we encrypt the entire 8k WAL

  1   2   3   4   5   >