Hi All, Just a gentle ping on this thread.
TLDR Due to the potentially complex implementation it's common practice to have a 1-1 mapping between encryption keys and encrypted files. Because of this I think we should revisit the 'key-id' + 'encryption-keys' list approach. For V3 manifest list files we can't change this but for V4 root manifests and statistics files we could introduce a simpler approach by storing 'encryption-key' directly in the encrypted file's metadata. As an example see how this could look like for statistics files in the spec: https://github.com/apache/iceberg/pull/17533 Xander, I prefer to conclude on the design first, and right after that will provide the Java implementation proposal. Thanks, Gabor Xander Bailey <[email protected]> ezt írta (időpont: 2026. szept. 24., Cs, 15:36): > I think these arguments are compelling arguments from my perspective. > > Did you get as far as creating a draft implementation for this change on > the Java side? Would be keen to see how this might look for rust also. > > Best, > Xander > > On Thu, 24 Sept 2026 at 13:39, Gábor Kaszab <[email protected]> > wrote: > >> I get your point, and agree that whichever design we go with, we can >> implement the 1-1 mapping between encryption keys and encrypted files. >> >> If we look for the long term, keeping the existing 'encryption-keys' >> based approach might cause some headaches: >> - When adding a new file type that is encrypted this way, then we have to >> remember to implement the encryption key cleanup for that file type too. >> E.g. for indexes in the future when we dereference an index file we have to >> implement a trigger to also clean up the relevant encryption key. With the >> 'in-place encryption key' it's not an issue. >> - The length of the 'encryption-keys' list matters. We have to find stuff >> there by ID, so even if we build a map from the list internally, there is a >> cost somewhere that grows with the size of the list. Now V3 manifest list >> encryption keys are there, we can't change that, but I would be careful >> adding keys for stat files, V4 root manifests, etc. into the list. >> >> As I understood, the motivation for the 'encryption-keys' list was to >> implement a way to reuse encryption keys across files. If reusal is not an >> option, the list lost its purpose. >> >> Any feedback is appreciated! >> Gabor >> >> Xander Bailey <[email protected]> ezt írta (időpont: 2026. szept. 24., >> Cs, 12:56): >> >>> Hi Gabor, >>> >>> I understand the cleanup benefit of storing the encrypted key metadata >>> alongside the file reference. I’m still weighing that against following the >>> existing manifest-list pattern. >>> >>> >>> With a fresh DEK and a dedicated encryption-keys entry for each >>> statistics file, the existing pattern would (does?) also support the >>> one-file-per-DEK model. The additional responsibility would be removing >>> that entry when the statistics metadata is removed or replaced, but this is >>> a lifecycle problem we already need to handle for manifest lists. We would >>> still need to manage shared KEK lifetimes in either design. >>> >>> >>> I agree on generating fresh DEKs per file, and your proposal makes >>> ownership of the key metadata more explicit. My hesitation is that >>> statistics and manifest lists would then use different representations for >>> essentially the same operation. Following the existing pattern would let >>> implementations extend the same key lookup and lifecycle handling to >>> statistics. >>> >>> >>> Do you think extending that handling to statistics would be >>> significantly more complicated than supporting the inline representation >>> alongside the existing manifest-list approach? I can see the cleanup >>> advantage in isolation, but I’m less convinced that it simplifies >>> implementations overall. >>> >>> >>> Thanks, >>> Xander >>> >>> On Thu, 24 Sept 2026 at 10:28, Gábor Kaszab <[email protected]> >>> wrote: >>> >>>> Thanks for the answers Gidon and Xander! >>>> >>>> This is how I think, a new design could look like for statistics files >>>> in terms of the spec: https://github.com/apache/iceberg/pull/17533 >>>> Here the encryption key is stored directly in the metadata for the file >>>> it encrypts, meaning there is 1-1 mapping between the encryption key and >>>> the encrypted files. KEKs remain in the `encryption-keys` list and we still >>>> refer to them by `key-id`. >>>> The same design could be used for V4 root manifests, but let's conclude >>>> on this first. Let me know if this makes sense! >>>> >>>> About "retaining the existing key-id / encryption-keys pattern": I >>>> think this is unwanted complexity, especially when it comes to cleaning up >>>> encryption keys. With the proposed design cleanup is coming out of the box: >>>> when a snapshot is gone, the root manifest's encryption key is also gone, >>>> no need to look elsewhere. Same for other file types like stats. >>>> >>>> Please share your opinions! >>>> Gabor >>>> >>>> >>>> Xander Bailey <[email protected]> ezt írta (időpont: 2026. szept. >>>> 23., Sze, 16:01): >>>> >>>>> Hi Gidon, Gabor, >>>>> >>>>> I agree that we should generate a fresh DEK for each file - in fact I >>>>> would personally want this to be made explicit. This came up when >>>>> reviewing >>>>> https://github.com/apache/iceberg/pull/16353 >>>>> >>>>> My initial preference reading this was to retain the existing key-id >>>>> / encryption-keys pattern so implementations could use the same >>>>> approach across these file types. The point about the metadata being >>>>> specific to the file, and simplifying its cleanup, makes sense though. >>>>> >>>>> I’d like us to preserve a common encrypted key metadata structure and >>>>> wrapping/unwrapping model as we make this change. We should be able to >>>>> share that implementation whether the metadata is resolved through a V3 >>>>> key >>>>> ID or obtained directly from a V4 per-file structure. >>>>> >>>>> With that, I can see the benefit of keeping the encrypted DEK metadata >>>>> alongside the file reference and retaining shared KEKs in >>>>> encryption-keys. The per-file DEK entry would no longer need its own >>>>> key-id, but would still need to identify the KEK used to wrap it. >>>>> >>>>> Thanks, >>>>> Xander >>>>> >>>>> >>>>> On Wed, 23 Sept 2026 at 14:42, Gidon Gershinsky <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Gabor, >>>>>> >>>>>> I agree that in V4, the DEK metadata is best moved into per-file >>>>>> structures (such as snapshots for manifest list files). This will make it >>>>>> easier to manage/clean-up these keys. The `key-d` field indeed >>>>>> becomes obsolete then. >>>>>> >>>>>> The key encryption keys would stay as they are in the shared >>>>>> `encryption keys` list in TableMetadata, since these keys are shared >>>>>> across >>>>>> the table. >>>>>> >>>>>> Technically, it is possible to re-use DEKs for multiple files, but >>>>>> this requires careful management to prevent breaking the AES GCM cipher. >>>>>> The simple and practical solution is to generate a random DEK per file. >>>>>> Moreover, Iceberg key metadata includes a unique file ID and, in some >>>>>> cases, the file length - so even if a key can be reused, the key metadata >>>>>> cannot. >>>>>> >>>>>> (as for the other technical details - the nonces/IVs are not visible >>>>>> at this level, they are an internal part of PME >>>>>> <https://parquet.apache.org/docs/file-format/data-pages/encryption/> >>>>>> and GCM Stream <https://iceberg.apache.org/gcm-stream-spec/> >>>>>> mechanisms) >>>>>> >>>>>> Cheers, Gidon >>>>>> >>>>>> >>>>>> On Wed, Sep 23, 2026 at 3:18 PM Gábor Kaszab <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hey Iceberg Community, >>>>>>> >>>>>>> >>>>>>> >>>>>>> I've been recently involved in conversations around encryption keys >>>>>>> and the current design of how we encrypt manifest list files with the >>>>>>> purpose of extending the design to additional file types like statistics >>>>>>> files and V4 root manifest files. >>>>>>> >>>>>>> I think there are some assumptions with the current design that we >>>>>>> should revisit now. >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Context* >>>>>>> >>>>>>> 1. Manifest list file encryption >>>>>>> >>>>>>> The `encryption-keys` list in table metadata (spec >>>>>>> <https://iceberg.apache.org/spec/#table-metadata-fields>) contains: >>>>>>> >>>>>>> - Encryption keys used for encrypting manifest list files (DEK) >>>>>>> - Key encryption keys (KEK) used for encrypting DEKs. >>>>>>> >>>>>>> The format for them is described here >>>>>>> <https://iceberg.apache.org/spec/#encryption-keys>. Each of them >>>>>>> has a `key-id` and for manifest list files we store a `key-id` in the >>>>>>> snapshot to refer to a DEK in the list. The DEK in turn refers to the >>>>>>> KEK >>>>>>> used for encrypting that particular DEK. >>>>>>> >>>>>>> 2. Manifest, data and delete file encryption >>>>>>> >>>>>>> The raw encryption keys are stored directly for these files as >>>>>>> `key-metadata`. See this >>>>>>> <https://iceberg.apache.org/spec/#manifest-lists> or this >>>>>>> <https://iceberg.apache.org/spec/#data-file-fields>. >>>>>>> >>>>>>> >>>>>>> >>>>>>> We'll focus on 1) now. >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Assumption* >>>>>>> >>>>>>> The assumption is that a particular DEK in the list can be reused >>>>>>> for encrypting multiple files (manifest list files now, other files also >>>>>>> later on) reducing the space required for storing the DEKs. >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Reality check* >>>>>>> >>>>>>> While I'm not an expert of this area, I did some research, and I >>>>>>> think while it's theoretically feasible, in practice it's overly >>>>>>> complicated to implement such a DEK sharing approach. Here is what I >>>>>>> found: >>>>>>> >>>>>>> - It's not just a DEK what we need for encryption but other >>>>>>> auxiliary, generated information like AAD prefix, nonce and other >>>>>>> auxiliary >>>>>>> information baked into `key-metadata` >>>>>>> - It's a cryptographical requirement that if DEK is reused then >>>>>>> the some of the generated auxiliary information MUST differ. More >>>>>>> particular: >>>>>>> - Some source says reusing the same DEK + nonce pair for >>>>>>> multiple files is "Catastrophic" Link >>>>>>> >>>>>>> <https://neilmadden.blog/2024/05/23/galois-counter-mode-and-random-nonces/> >>>>>>> - Even if nonce is generated, there is a theoretical chance of >>>>>>> reusing the same for the same DEK >>>>>>> - Implementation-wise a tracking information is required to keep >>>>>>> which nones are used for each DEK. This information have to be kept >>>>>>> in a >>>>>>> persisted way that we don't loose after a restart. Multi-writer >>>>>>> scenarios >>>>>>> would make keeping this information even harder >>>>>>> - It's common practice to use a different DEK for different files >>>>>>> >>>>>>> >>>>>>> >>>>>>> Based on the above, I'd be against reusing such keys across multiple >>>>>>> files. >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Way forward* >>>>>>> >>>>>>> For V3 manifest lists we already use the approach with a `key-id` >>>>>>> in snapshot referring to an encrypted key metadata in `encryption-keys` >>>>>>> list. This has been out there and I don’t think we can change this now. >>>>>>> >>>>>>> >>>>>>> >>>>>>> For future file types, like V4 root manifests and statistics files, >>>>>>> I think we can consider using a different approach. If we conclude on >>>>>>> not >>>>>>> reusing key metadata across files, then I don’t see any point of having >>>>>>> the >>>>>>> indirection of storing a `key-id` that refers to an item in the list. >>>>>>> >>>>>>> Instead we can follow a more direct approach, and simply store the >>>>>>> encrypted key metadata of such a file instead of a `key-id`. For >>>>>>> instance >>>>>>> for V4 root manifest files we can store `encryption-key` (spec >>>>>>> <https://iceberg.apache.org/spec/#encryption-keys>) directly >>>>>>> instead of `key-id`. >>>>>>> >>>>>>> >>>>>>> Note, this is the structure of an `encryption-key`: >>>>>>> >>>>>>> - Key-id >>>>>>> - Encrypted-key-metadata >>>>>>> - Encrypted-by-id >>>>>>> - Properties >>>>>>> >>>>>>> In this design `key-id` in the structure is unnecessary. (might be >>>>>>> optional?) >>>>>>> >>>>>>> >>>>>>> We could still keep the KEKs in the `encryption-keys` list. >>>>>>> >>>>>>> >>>>>>> What do you think? >>>>>>> >>>>>>> Gabor >>>>>>> >>>>>>
