Hi all, During the review of the following Polaris pull request, I was asked to bring this topic to the dev mailing list for visibility:
https://github.com/apache/polaris/pull/5060 The discussion spans several layers: engine-side file writes, REST metadata commits, KMS access, and server-side operations such as purge. Based on the current Iceberg behavior, REST Catalog semantics, and the related implementation work, the responsibilities and related work can be organized as follows. Iceberg native encryption ------------------------- Iceberg native encryption protects data files, delete files, manifest files, and manifest-list files. The table metadata JSON itself remains plaintext. It contains the encryption metadata and wrapped keys required to open the encrypted files. Engine/client-side writes ------------------------- The engine or client writes the encrypted data files, manifests, and manifest lists. It obtains the required KMS access and commits the resulting metadata updates, including wrapped encryption keys, through the catalog. For a REST Catalog, Polaris receives and persists those metadata updates through the normal commit path. Polaris does not write manifests and does not need KMS access for this path. The remaining Iceberg client work for performing encrypted reads and writes through RESTCatalog is tracked in: https://github.com/apache/iceberg/pull/13225 Server-side purge ----------------- Server-side purge is a separate read and delete path. When a table is dropped with purgeRequested=true, Polaris is responsible for deleting the files referenced by the table. The data and delete file locations are stored in manifests, and the manifest locations are obtained from manifest lists. For an encrypted table, the cleanup worker therefore needs to decrypt the manifest lists and manifests before it can enumerate the files to delete. This requires KMS access and the wrapped table keys on the Polaris side. This does not require Polaris to write, rewrite, or re-encrypt manifests. The server uses encryption only to read the files required for purge and then delegates deletion to the underlying FileIO. The Polaris implementation work for this path is in: https://github.com/apache/polaris/pull/5060 That PR keeps encryption handling out of the normal LocalIcebergCatalog write path. The encryption context is used by asynchronous cleanup tasks to create an encryption-aware FileIO for reading encrypted manifest lists and manifests. HiveCatalog comparison ---------------------- HiveCatalog can make the two paths look less distinct because the catalog is loaded in the same process as the engine. The same catalog implementation can therefore provide encryption support for engine-side writes and for purge. With a REST Catalog, the engine and the Polaris server are separate processes. The engine performs encrypted writes, while Polaris participates only in REST metadata commits and server-side operations assigned to the catalog service. Related work ------------ The overall Polaris encryption discussion is tracked in: https://github.com/apache/polaris/issues/2829 REST KMS credential vending is being discussed separately in: https://github.com/apache/iceberg/pull/17155 That work concerns credentials returned by a REST catalog to a client so that the client can access KMS directly. It is separate from the credentials used internally by a Polaris cleanup worker for server-side purge. Server-side scan planning is another server-side read path related to encrypted table metadata. Polaris does not currently implement the general REST server-side scan-planning endpoint, so this is outside the scope of the current purge work. Metadata integrity, encryption-key retirement, and other server-side maintenance operations are also adjacent topics recorded in the tracking issue, but are separate from the existing purgeRequested behavior. These topics can therefore be tracked as separate but related work streams: client-side encryption, server-side purge, credential vending, and possible future server-side operations. If anything in this summary is incomplete or does not match your understanding, please point it out.
