Hi all, The built-in encryption is designed to protect tables in any storage backend, including untrusted storage. So I also think the storage credentials and KMS credentials are best treated separately, as they don't have the same purpose/scope.
Regarding the current PR (13225) - it doesn't handle credentials directly; instead, it focuses on building the scaffolding for table encryption in the REST catalog client - similarly to the Hive catalog client (and other catalogs in the future). Cheers, Gidon On Tue, May 5, 2026 at 3:40 AM Chris Lu <[email protected]> wrote: > Hi Prashant, > > My own preference would be to keep storage credentials and encryption/KMS > credentials separate in the REST contract. > > Storage credentials and KMS/Vault credentials have different scopes, > lifetimes, providers, and failure modes. > > > - Storage credentials authorize access to object locations. > - > > KMS/Vault credentials authorize key wrap/unwrap operations. > > Object-store SSE-KMS is a storage-layer feature, but Iceberg table > encryption is not necessarily tied to the object store. Vault or external > KMS support would not naturally fit inside a storage credential object. So > mixing KMS credentials into the storage credentials object may work for > some S3/SSE cases, but it makes the REST model harder to extend. > > A cleaner model may be to have separate sections for storage credentials > and encryption credentials. Another option is a generic typed credential > model, where each credential has a type such as storage or encryption, and > a provider such as S3, Vault, or a cloud KMS. > > The typed model may be easier to extend later if we expect more credential > types. > > For backward compatibility, I think the current PR can still move forward, > but the client should not assume that storage credentials are the permanent > place for KMS credentials. If we need an interim implementation, maybe it > should be documented as catalog-level KMS configuration support, not > table-level encryption credential vending. > > I also think the catalog should have a clear capability or failure signal. > If a table requires Iceberg encryption and the catalog supports vended > credentials, then the catalog should either return the required encryption > credentials or fail early during table load or credential refresh with a > clear error. Failing later during file IO or key unwrap would be much > harder to debug. > > So my suggested direction would be: > > > - Keep object-store SSE credentials/config separate from Iceberg table > encryption credentials. > - > > Add a dedicated encryption credential model, or a generic typed > credential model. > - > > Add a capability or requirement signal so clients can fail early when > required credentials are missing. > > Avoid baking in the assumption that KMS credentials always live under > storage credentials. This should leave room for catalogs that vend > table-scoped KMS/Vault credentials in the future, instead of only > supporting catalog-level KMS client configuration. > > Thanks, > Chris > On Tue, May 5, 2026 at 3:22 AM Sreesh Maheshwar <[email protected]> wrote: > Hey Prashant, > > Thank you for kicking off this discussion! Some thoughts: > > > The current PR advocates for including KMS credentials within the > storage credentials object. > > This isn't true - the KMS client is currently configured with just the > catalog config client-side (though that includes the server's `getConfig` > response) [1]; there is no per-table vending of KMS credentials as such. > > > Separation of KMS/Vault Creds from Storage Creds > > I agree, if/when it supports this, the AWS KMS client could choose to > introduce `kms.access-key-id` and co. instead of reusing `s3.access-key-id` > and co. (similar to how we have `rest.access-key-id` and co. [2]) to > differentiate from storage credentials. And similarly for vending: if we'd > like to support the vending of KMS credentials [3], we can discuss its > format to enable separation. > > (Specifically on the current PR [4], I don't think that the concerns > described affect it due to my first point above. I think > catalog-initialised KMS clients, as implemented, are useful to support now > and in the future.) > > Thanks, > Sreesh Maheshwar > > [1] > https://github.com/apache/iceberg/pull/13225/files#diff-86450612dbe323d6d06cbc3846aa1913f042eaedadc0ca027c36bfbe08d3a46cR284 > [2] > https://github.com/apache/iceberg/blob/2d54125734ddc9b9fb87db147ff255918108fa2c/aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java#L193 > [3] https://github.com/apache/iceberg/issues/16194 > [4] https://github.com/apache/iceberg/pull/13225 > > On Monday, May 4th, 2026 at 9:03 AM, Prashant Singh < > [email protected]> wrote: > > Hi everyone, > > I’d like to start a discussion regarding how we handle credentials for > encryption (like KMS or Vault (recently being discussed)) in the REST > catalog. > > As we know, unlike other catalogs, the REST catalog mints credentials at > the table level for the client to use in subsequent operations, and we > already have the dedicated /credentials endpoint in place to handle > refreshing these. > > While reviewing the recent encryption PR, a few architectural concerns > came up that I believe we need to conclude on before we mark the REST > catalog as "ready" for supporting encryption: > > - > > *Separation of KMS/Vault Creds from Storage Creds:* How should we > handle external key managers like Vault? The current PR [1] advocates for > including KMS credentials within the storage credentials object. If we end > up supporting Vault, this can't be mixed with storage cred. *(Side > note: catalogs have historically mixed KMS creds with this object for > things like SSE, but that is entirely an object-store-level concept).* > We need a clear path forward for how REST will return per-table credentials > specifically for Vault/KMS stores. > - > > *Catalog Awareness & Client-Side Assertions:* If the catalog returns > credentials, it overrides the client-side credentials. This means a naive > catalog that is unaware of encryption and just treats metadata as-is will > have no clue it needs to vend these specific KMS credentials and if it > forgerts to do that there is no way for the client to know this (for object > store cases) except to fail during runtime ? Should the catalog fail such > requests *as part of their contract of supporting v3* (we don't need > this in spec), should the catalog send some signal, hey i send you creds > for encryption too, and if the client doesn't find it fails early ? > - > > *Backward Compatibility Risks:* If we release the client now with the > expectation that "storage credentials" will always contain KMS credentials. > If we later introduce a dedicated field for encryption credentials in the > loadTable response, we will be forced to maintain backward > compatibility to support both ways of returning credentials. > > To be clear, I do not want to block the progress on the current PR. I > really appreciate all the hard work that has gone into it! However, I think > it is crucial that we align on these design points for the REST catalog's > encryption architecture before finalizing it. > > I would appreciate any thoughts or feedback on how we should structure > this. > > [1] https://github.com/apache/iceberg/pull/13225 > > Thanks, > > Prashant Singh > > >
