+1 on Dmitri's point. I think 5060 goes in the right direction. Yufei
On Fri, Aug 7, 2026 at 7:22 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Robert, > > You raise good points and I agree that the Polaris project should follow up > on those items. > > However, as far as PR [5060] goes, I believe it represents a valid > incremental change. If the Polaris owner is comfortable with the security > of object storage linked to Polaris, I do not see a reason for not > permitting Polaris to use encryption key IDs from the metadata file. > > [5060] https://github.com/apache/polaris/pull/5060 > > Cheers, > Dmitri. > > On Thu, Jul 23, 2026 at 8:44 AM Robert Stupp <[email protected]> wrote: > > > Hi all, > > > > Thanks for laying out the different work streams. > > > > I think one part of the classification needs tightening: metadata > integrity > > is not merely adjacent work. > > > > Iceberg documents two catalog security requirements [1] for table > > encryption: > > > > * `encryption.key-id` must not be modified or removed during the table's > > lifetime; > > * when `metadata.json` is kept in storage vulnerable to tampering, the > > catalog > > must obtain the metadata from a trusted store or verify its integrity > > using > > information kept in separate trusted storage. > > > > These requirements apply before Polaris can safely rely on > `metadata.json` > > for encrypted-table operations, including server-side purge. > > > > The earlier RFC discussed both requirements, but I do not currently see > > where the active work in #5060 and #5127 will enforce them. > > Could we make these two invariants explicit requirements of the combined > > Polaris design and identify which changes will own them? > > > > It may be useful for the authors and reviewers of #5060 and #5127 to walk > > through this together at the next Polaris community call (in 14 days) and > > converge on one plan. > > > > Cheers, > > Robert > > > > [1] > > > > > https://iceberg.apache.org/docs/nightly/encryption/#catalog-security-requirements > > > > On Tue, Jul 21, 2026 at 10:39 AM Hiroaki Kawai <[email protected]> > > wrote: > > > > > Hi all, > > > > > > During the review of the following Polaris pull request, I was asked to > > > bring > > > this topic to the dev mailing list for visibility: > > > > > > https://github.com/apache/polaris/pull/5060 > > > > > > The discussion spans several layers: engine-side file writes, REST > > metadata > > > commits, KMS access, and server-side operations such as purge. Based on > > the > > > current Iceberg behavior, REST Catalog semantics, and the related > > > implementation > > > work, the responsibilities and related work can be organized as > follows. > > > > > > Iceberg native encryption > > > ------------------------- > > > > > > Iceberg native encryption protects data files, delete files, manifest > > > files, > > > and manifest-list files. > > > > > > The table metadata JSON itself remains plaintext. It contains the > > > encryption > > > metadata and wrapped keys required to open the encrypted files. > > > > > > Engine/client-side writes > > > ------------------------- > > > > > > The engine or client writes the encrypted data files, manifests, and > > > manifest > > > lists. It obtains the required KMS access and commits the resulting > > > metadata > > > updates, including wrapped encryption keys, through the catalog. > > > > > > For a REST Catalog, Polaris receives and persists those metadata > updates > > > through the normal commit path. Polaris does not write manifests and > does > > > not > > > need KMS access for this path. > > > > > > The remaining Iceberg client work for performing encrypted reads and > > writes > > > through RESTCatalog is tracked in: > > > > > > https://github.com/apache/iceberg/pull/13225 > > > > > > Server-side purge > > > ----------------- > > > > > > Server-side purge is a separate read and delete path. > > > > > > When a table is dropped with purgeRequested=true, Polaris is > responsible > > > for > > > deleting the files referenced by the table. The data and delete file > > > locations > > > are stored in manifests, and the manifest locations are obtained from > > > manifest > > > lists. > > > > > > For an encrypted table, the cleanup worker therefore needs to decrypt > the > > > manifest lists and manifests before it can enumerate the files to > delete. > > > This > > > requires KMS access and the wrapped table keys on the Polaris side. > > > > > > This does not require Polaris to write, rewrite, or re-encrypt > manifests. > > > The > > > server uses encryption only to read the files required for purge and > then > > > delegates deletion to the underlying FileIO. > > > > > > The Polaris implementation work for this path is in: > > > > > > https://github.com/apache/polaris/pull/5060 > > > > > > That PR keeps encryption handling out of the normal LocalIcebergCatalog > > > write > > > path. The encryption context is used by asynchronous cleanup tasks to > > > create > > > an encryption-aware FileIO for reading encrypted manifest lists and > > > manifests. > > > > > > HiveCatalog comparison > > > ---------------------- > > > > > > HiveCatalog can make the two paths look less distinct because the > catalog > > > is > > > loaded in the same process as the engine. The same catalog > implementation > > > can > > > therefore provide encryption support for engine-side writes and for > > purge. > > > > > > With a REST Catalog, the engine and the Polaris server are separate > > > processes. > > > The engine performs encrypted writes, while Polaris participates only > in > > > REST > > > metadata commits and server-side operations assigned to the catalog > > > service. > > > > > > Related work > > > ------------ > > > > > > The overall Polaris encryption discussion is tracked in: > > > > > > https://github.com/apache/polaris/issues/2829 > > > > > > REST KMS credential vending is being discussed separately in: > > > > > > https://github.com/apache/iceberg/pull/17155 > > > > > > That work concerns credentials returned by a REST catalog to a client > so > > > that > > > the client can access KMS directly. It is separate from the credentials > > > used > > > internally by a Polaris cleanup worker for server-side purge. > > > > > > Server-side scan planning is another server-side read path related to > > > encrypted > > > table metadata. Polaris does not currently implement the general REST > > > server-side scan-planning endpoint, so this is outside the scope of the > > > current > > > purge work. > > > > > > Metadata integrity, encryption-key retirement, and other server-side > > > maintenance operations are also adjacent topics recorded in the > tracking > > > issue, but are separate from the existing purgeRequested behavior. > > > > > > These topics can therefore be tracked as separate but related work > > streams: > > > client-side encryption, server-side purge, credential vending, and > > possible > > > future server-side operations. > > > > > > If anything in this summary is incomplete or does not match your > > > understanding, > > > please point it out. > > > > > >
