Would PostgreSQL 16 native transparent data encryption support database level encryption?

[Tony Xu]

Hi There,

The FAQ (copied below) mentioned that native transparent data encryption
might be included in 16. Is it fair to assume that it will support database
level encryption, that is, we can use two encryption keys for two databases
in the same server, respectively? How can one verify that?

Thanks
Tony




*https://www.postgresql.org/about/press/faq/
*

*Q: What features will PostgreSQL 16 have?A: As always, we can't be certain
what will go in and what won't; the project has strict quality standards
that not all patches can make before deadline. All we can tell you is
what's currently being worked on, which includes native transparent data
encryption support, continued improvements to logical replication,
parallelism, partitioning, and vacuuming, and many more features. By the
time 16 is released, though, this list may have changed considerably.*

Re: Would PostgreSQL 16 native transparent data encryption support database level encryption?

[Tony Xu]

Ok, thanks for the clarification. Any particular reason for the change of
plans? Would it come in 17?

Thanks
Tony

On Wed, May 17, 2023, 5:49 PM Bruce Momjian  wrote:

> On Wed, May 17, 2023 at 03:35:39PM -0700, Tony Xu wrote:
> > Hi There,
> >
> > The FAQ (copied below) mentioned that native transparent data encryption
> might
> > be included in 16. Is it fair to assume that it will support database
> level
> > encryption, that is, we can use two encryption keys for two databases in
> the
> > same server, respectively? How can one verify that?
>
> It will not be in PG 16.
>
> --
>   Bruce Momjian  https://momjian.us
>   EDB  https://enterprisedb.com
>
>   Embrace your flaws.  They make you human, rather than perfect,
>   which you will never be.
>

Re: Would PostgreSQL 16 native transparent data encryption support database level encryption?

[Tony Xu]

Thanks for the information,  Andreas, Stephen.

Our use-case is for a multi-tenancy scenario - we are considering using
different databases to store different customer's data, however, for
cost-efficiency, we want to host them in the same server (to reduce the
CPU/mem idle time and to reduce the server management efforts). Now there
is a compliance related feature that we need to let our customer control
the KEK for their databases so they can rotate their KEKs independently, so
we cannot use one KEK for the whole PG server. Conceptually, different
databases are independent of each other, it also makes sense to allow them
to have completely independent encryption facilities?

Thanks,
Tony



On Thu, May 18, 2023 at 8:54 AM Stephen Frost  wrote:

> Greetings,
>
> * Tony Xu (tony...@rubrik.com) wrote:
> > The FAQ (copied below) mentioned that native transparent data encryption
> > might be included in 16. Is it fair to assume that it will support
> database
> > level encryption, that is, we can use two encryption keys for two
> databases
> > in the same server, respectively? How can one verify that?
>
> The current work to include TDE in PG isn't contemplating a per-database
> key option.  What's the use-case for that?  Why do you feel that you'd
> need two independent keys?  Also, the general idea currently is that
> we'll have one key provided by the user which will be a KEK and then
> multiple DEKs (different ones for relation data vs. temporary data vs.
> the WAL).
>
> If you're interested in TDE in PG, we could certainly use more folks
> being involved and working to push it forward.
>
> Thanks,
>
> Stephen
>

Re: Would PostgreSQL 16 native transparent data encryption support database level encryption?

[Tony Xu]

Thanks all for the discussions. New to PostgreSQL so don't have much
context here.

Regarding the multiple clusters idea, how does that work? Assume we can
store one customer's data in one cluster, is it possible to have separate
KEK for different clusters?

Why not using multiple clusters then?

Better isolation of the customers, but still on one server.


On Thu, May 18, 2023 at 3:53 PM Stephen Frost  wrote:

> Greetings,
>
> Please don't top-post on these lists.
>
> * Tony Xu (tony...@rubrik.com) wrote:
> > Our use-case is for a multi-tenancy scenario - we are considering using
> > different databases to store different customer's data, however, for
> > cost-efficiency, we want to host them in the same server (to reduce the
> > CPU/mem idle time and to reduce the server management efforts). Now there
> > is a compliance related feature that we need to let our customer control
> > the KEK for their databases so they can rotate their KEKs independently,
> so
> > we cannot use one KEK for the whole PG server. Conceptually, different
> > databases are independent of each other, it also makes sense to allow
> them
> > to have completely independent encryption facilities?
>
> This really isn't currently in the plans and while it might be something
> added later, as pointed out farther down on this thread, it wouldn't be
> possible for the shared catalogs or the WAL to have separate keys for
> those things which are relevant to a database, so it's not like each
> tenant would actually have control over the key for "all" of their data
> (consider that roles are stored in a shared PG catalog and then shared
> among databases...).
>
> To meet this compliance requirement, you'd certainly be much more able
> to blanket claim that everything is independent by having a separate PG
> instance for each client.  This would also allow rather useful things
> like being able to do a file-based restore on a per-client basis in the
> event something happens, rather than having to roll back an entire
> cluster to some point in time just because one client did something
> bad..  You'd also be able to scale the number of systems supporting a
> given client independently.
>
> Thanks,
>
> Stephen
>

Re: Would PostgreSQL 16 native transparent data encryption support database level encryption?

[Tony Xu]

Thanks all for the discussions. It sounds like there are different
questions to clear before we can get to a conclusion on if per-database KEK
is possible or not.

First question - do we, as a community, see the value of the proposal and
do we believe that value is big enough for us to make any necessary changes
to PostgreSQL?  Another way to ask is, do we want/need to keep the parity
with other database solutions that already have such ability?

If the answer to the first question is no, then I will stop here.
However, if the answer is yes or "it depends", then the second question is
on how - It seems that per-cluster KEK can be a a good alternative to
achieve the same as per-db KEK, however, there are still some shared area
between clusters. Is it possible to further split this shared area? Are
there more shared memory/storage between clusters?


> In the proposed TDE work, yes, each cluster (which is an entier
>> PostgreSQL system) would be able to have its own KEK.
>>
>> There's a bit of overhead from each cluster and each would have their
>> own shared buffers pool of memory and such.
>>
>
Thanks
Tony

Re: Would PostgreSQL 16 native transparent data encryption support database level encryption?

[Tony Xu]

Thanks Christophe for the clarification.

That's not quite right.  A PostgreSQL cluster (in the traditional sense,
> which means one PostgreSQL server handling a particular endpoint) is
> isolated from any other clusters on the same machine.
>

Thanks. I think I had a misunderstanding over the "cluster" eariler. So a
PostgreSQL cluster is a PostgreSQL server that we're talking about in
general. This means per-DB KEY within a cluster/server is impossible
without splitting the whole shared storage among databases, which is indeed
a significant amount of work.


> And, to be somewhat blunt, is Rubrik prepared to pay for it?  This
> engineering effort needs to be funded by someone, since engineers have to
> eat.  This is not to say that any feature someone is willing to pay for
> will make it into PostgreSQL, but an effort of this size, to have any hope
> of reaching the point of being accepted, will need someone to pay the
> salaries of the people working on it.
>
I cannot speak on behalf of Rubrik - I'm just exploring PostgreSQL as a
solution for one of my team's projects thus the questions. Nothing is
finalized yet but we prefer a managed solution provided by some cloud infra
providers so the context I'm getting here will allow me to better
understand our provider's solution and their road map.

Thanks
Tony