Hi Prashant, My own preference would be to keep storage credentials and encryption/KMS credentials separate in the REST contract.
Storage credentials and KMS/Vault credentials have different scopes, lifetimes, providers, and failure modes. - Storage credentials authorize access to object locations. - KMS/Vault credentials authorize key wrap/unwrap operations. Object-store SSE-KMS is a storage-layer feature, but Iceberg table encryption is not necessarily tied to the object store. Vault or external KMS support would not naturally fit inside a storage credential object. So mixing KMS credentials into the storage credentials object may work for some S3/SSE cases, but it makes the REST model harder to extend. A cleaner model may be to have separate sections for storage credentials and encryption credentials. Another option is a generic typed credential model, where each credential has a type such as storage or encryption, and a provider such as S3, Vault, or a cloud KMS. The typed model may be easier to extend later if we expect more credential types. For backward compatibility, I think the current PR can still move forward, but the client should not assume that storage credentials are the permanent place for KMS credentials. If we need an interim implementation, maybe it should be documented as catalog-level KMS configuration support, not table-level encryption credential vending. I also think the catalog should have a clear capability or failure signal. If a table requires Iceberg encryption and the catalog supports vended credentials, then the catalog should either return the required encryption credentials or fail early during table load or credential refresh with a clear error. Failing later during file IO or key unwrap would be much harder to debug. So my suggested direction would be: - Keep object-store SSE credentials/config separate from Iceberg table encryption credentials. - Add a dedicated encryption credential model, or a generic typed credential model. - Add a capability or requirement signal so clients can fail early when required credentials are missing. Avoid baking in the assumption that KMS credentials always live under storage credentials. This should leave room for catalogs that vend table-scoped KMS/Vault credentials in the future, instead of only supporting catalog-level KMS client configuration. Thanks, Chris On Monday, May 4th, 2026 at 9:03 AM, Prashant Singh <[email protected]> wrote: > Hi everyone, > > I’d like to start a discussion regarding how we handle credentials for > encryption (like KMS or Vault (recently being discussed)) in the REST catalog. > > As we know, unlike other catalogs, the REST catalog mints credentials at the > table level for the client to use in subsequent operations, and we already > have the dedicated /credentials endpoint in place to handle refreshing these. > > While reviewing the recent encryption PR, a few architectural concerns came > up that I believe we need to conclude on before we mark the REST catalog as > "ready" for supporting encryption: > > - > > Separation of KMS/Vault Creds from Storage Creds: How should we handle > external key managers like Vault? The current PR [1] advocates for including > KMS credentials within the storage credentials object. If we end up > supporting Vault, this can't be mixed with storage cred. (Side note: catalogs > have historically mixed KMS creds with this object for things like SSE, but > that is entirely an object-store-level concept). We need a clear path forward > for how REST will return per-table credentials specifically for Vault/KMS > stores. > > - > > Catalog Awareness & Client-Side Assertions: If the catalog returns > credentials, it overrides the client-side credentials. This means a naive > catalog that is unaware of encryption and just treats metadata as-is will > have no clue it needs to vend these specific KMS credentials and if it > forgerts to do that there is no way for the client to know this (for object > store cases) except to fail during runtime ? Should the catalog fail such > requests as part of their contract of supporting v3 (we don't need this in > spec), should the catalog send some signal, hey i send you creds for > encryption too, and if the client doesn't find it fails early ? > > - > > Backward Compatibility Risks: If we release the client now with the > expectation that "storage credentials" will always contain KMS credentials. > If we later introduce a dedicated field for encryption credentials in the > loadTable response, we will be forced to maintain backward compatibility to > support both ways of returning credentials. > > To be clear, I do not want to block the progress on the current PR. I really > appreciate all the hard work that has gone into it! However, I think it is > crucial that we align on these design points for the REST catalog's > encryption architecture before finalizing it. > > I would appreciate any thoughts or feedback on how we should structure this. > > [1] https://github.com/apache/iceberg/pull/13225 > > Thanks, > > Prashant Singh
