I have prepared two draft patches for the missing catalog-side prerequisites and summarized the proposed model below.
## Background Iceberg table encryption is already part of Iceberg's table model. In particular, Iceberg defines the `encryption.key-id` table property and specifies security requirements for catalogs that handle encrypted tables: https://iceberg.apache.org/docs/nightly/encryption/#catalog-security-requirements This discussion should therefore start from those catalog requirements, rather than from whether Polaris itself performs encrypted file I/O or server-side purge. An Iceberg client can already use a KMS and encryption-aware FileIO while using Polaris as its REST Catalog. In that arrangement, the client writes encrypted data files, delete files, manifests, and manifest lists, while `metadata.json` remains plaintext. Polaris can store the table's metadata pointer and serve the table without using the KMS itself. This means the catalog security requirements existed independently of apache/polaris#5060. That PR adds a Polaris-side consumer of encrypted metadata by allowing asynchronous server-side purge to read encrypted manifest lists and manifests. It makes the consequences of trusting storage metadata more visible, but it does not create the underlying catalog responsibilities. ## Related work and discussion This proposal continues the encryption discussion tracked in apache/polaris#2829. The digest is a first integrity step, not a claim that plaintext metadata becomes confidential. The two draft implementation PRs are: - apache/polaris#5185: https://github.com/apache/polaris/pull/5185 - apache/polaris#5186: https://github.com/apache/polaris/pull/5186 The related work has separate responsibilities: | Work | Responsibility and current status | | --- | --- | | apache/polaris#5185 | Pin the expected table key ID, including its absence | | apache/polaris#5186 | Pin and verify the current encrypted metadata revision; stacked on #5185 | | apache/polaris#5060 | Let asynchronous server-side purge read encrypted manifests | | apache/polaris#5127 | Propose a catalog-level representation and persistence model for KMS configuration; its current change does not connect that model to `LocalIcebergCatalog` or encrypted FileIO | | apache/iceberg#13225 | Wire client-side encrypted reads and writes through `RESTCatalog` | | apache/iceberg#17155 | Define REST KMS credential vending | | apache/polaris#3786 | Explore a delegation-service architecture for server-side work such as purge and scan planning | #5185 and #5186 are catalog invariants and do not technically depend on #13225 or #17155. Likewise, #5127 is related to how KMS configuration may be represented, but the current proposal does not make it a prerequisite for #5060. #5060 already has runtime wiring based on catalog properties; what remains unowned is connecting #5127's structured model and secret handling to that wiring. Although the two requirements are logically independent, #5185 and #5186 are not proposed as two independently selectable protection modes for untrusted metadata storage. #5186 is stacked on #5185, and the proposed rollout treats them as one unit. A protected encrypted table admitted under this model must atomically have a pin marker, the trusted key ID, the metadata digest, and the digest version. A state with a trusted key ID but no digest is inconsistent and fails closed. There is no supported path that enrolls a table under #5185 alone and later derives or backfills its digest from storage. Both draft PRs target `apache/polaris` `main` because their head branches are in an external fork. The #5186 branch contains the #5185 commit followed by one metadata-integrity commit. Its incremental second-layer diff is: https://github.com/hkwi/polaris/compare/upstream/enforce-immutable-encryption-key-id...upstream/enforce-encrypted-metadata-integrity After #5185 is merged, the #5186 head branch will need to be updated against the resulting `main` history so that the upstream PR contains only the metadata-integrity layer. apache/iceberg#15314 is an unresolved proposal for forward-only rotation to a different master-key ID. This is distinct from rotation performed by a KMS behind a stable key ID, and from Iceberg's internal KEK rotation; neither of those changes `encryption.key-id` and both remain compatible with key-ID pinning. If a master-key-ID transition is standardized in the future, the current table-lifetime invariant would need a narrowly defined catalog-authorized transition rather than an ordinary property update. That design is outside the scope of the current patches. ## Two independent catalog requirements Iceberg defines two distinct catalog responsibilities. ### 1. The table key ID must be immutable A catalog must ensure that `encryption.key-id` is not modified or removed during a table's lifetime. This requirement does not depend on the storage trust model. Even if `metadata.json` is kept in trusted or tamper-proof storage, an otherwise valid catalog update must not be allowed to add, change, or remove the table's master-key ID. A catalog that accepts Iceberg tables therefore has two meaningful choices: 1. Explicitly reject tables that use Iceberg encryption. 2. Accept them and enforce the key-ID invariant. Polaris currently accepts metadata containing Iceberg encryption properties. Unless Polaris decides to reject encrypted tables, key-ID pinning is consequently a general catalog responsibility, not a feature needed only by server-side purge. apache/polaris#5185 addresses this requirement by recording trusted key-ID state when a table is created. The trusted value may either be a particular key ID or the explicit absence of a key ID. ### 2. Plaintext table metadata may require integrity protection Iceberg metadata files are not encrypted. If `metadata.json` is stored in storage that may be modified by an attacker, a catalog must not treat the file as authoritative without an independent integrity mechanism. Iceberg permits several ways to satisfy this requirement: - keep metadata in an independent trusted object store; - keep `metadata.json` in tamper-proof storage; or - verify it with a checksum stored in separate trusted catalog state. Unlike key-ID immutability, this requirement is conditional on the storage trust model. A checksum is not required when the metadata storage is trusted or tamper-proof. If normal object storage is treated as untrusted, however, Polaris needs a mechanism equivalent to apache/polaris#5186. That conditional statement describes the alternatives allowed by Iceberg, not a Polaris configuration switch in this proposal. Under this proposal, every protected encrypted table has a trusted digest; deployments do not disable it by declaring their metadata storage trusted. #5186 stores the canonical SHA-256 of the current encrypted table metadata in trusted Polaris state. The digest is updated atomically with the current metadata pointer and is checked when the metadata is loaded and before asynchronous purge traverses it. ## Why the two requirements are independent The two protections overlap, but neither replaces the other. A trusted metadata digest detects storage-side modification of `encryption.key-id`. It does not prevent an authorized catalog commit from replacing both the metadata and its digest with a different key ID. Conversely, key-ID pinning prevents the key ID from being added, changed, or removed, but it does not protect snapshots, manifest references, schemas, wrapped encryption keys, or other metadata fields from storage-side modification or rollback. The two catalog requirements can therefore be summarized as follows: | Requirement | Proposed change | | --- | --- | | Table-lifetime `encryption.key-id` invariant | apache/polaris#5185 | | Encrypted metadata integrity when metadata storage is untrusted | apache/polaris#5186 | Client-side KMS use does not remove the need for #5185 or, when storage is untrusted, #5186. ## Table protection states Polaris distinguishes a table's protection state using trusted, internal entity properties, rather than public entity properties or values inferred from `metadata.json`. Polaris-owned names such as `polaris.encryption.key-id` and `polaris.encryption.metadata-hash` are deliberately distinct from Iceberg table properties supplied by clients. | State | Pin marker | Trusted key ID | Trusted metadata digest | | --- | --- | --- | --- | | Legacy | Absent | Unknown | None | | Protected plaintext | Present | Pinned absent | None | | Protected encrypted | Present | Pinned value | Present under #5186 | The digest record also carries an explicit canonicalization version (`iceberg-canonical-json-sha256-v1`). An unknown version fails closed instead of silently recomputing the digest with different semantics. The absence of `encryption.key-id` in a protected plaintext table is an explicitly pinned state. It is different from a legacy table, where the expected key-ID state is unknown. In compact form: ```text marker absent -> legacy table; expected key-ID state is unknown marker present, key ID absent -> protected plaintext table; key-ID absence is pinned marker present, key ID present -> protected encrypted table; the exact key ID is pinned and #5186 stores the current metadata digest ``` ## State transitions New tables immediately enter one of the protected states: ```text CREATE without encryption.key-id -> protected plaintext CREATE with encryption.key-id = K -> protected encrypted(K) ``` A protected plaintext table may commit new metadata only while the key ID remains absent: ```text protected plaintext -- commit without encryption.key-id --> protected plaintext ``` A protected encrypted table may commit new metadata only while the same key ID is retained. Under #5186, each successful commit replaces the trusted digest with the digest of the new current metadata: ```text protected encrypted(K, digest H1) -- commit with the same key ID K --> protected encrypted(K, digest H2) ``` The following transitions are rejected: ```text protected plaintext -X-> protected encrypted protected encrypted(K1) -X-> protected encrypted(K2) protected encrypted -X-> protected plaintext ``` The checks apply both to catalog updates and to metadata loaded from storage. In particular, adding `encryption.key-id` directly to the stored metadata of a protected plaintext table is rejected even though that table has no metadata digest. Under this strict interpretation, encryption cannot be enabled later for an existing protected plaintext table. Enabling encryption requires a new table lifetime, for example by dropping and recreating the table, unless Iceberg defines a separate safe transition mechanism in the future. Dropping and recreating a table starts a new table lifetime, so the newly created table may independently be plaintext or encrypted. ## Legacy tables and upgrade behavior Tables created before these checks were introduced have no pin marker. Polaris has no trusted catalog record of either: - the expected value, including absence, of `encryption.key-id`; or - the expected digest of the current table metadata. Polaris cannot safely derive those values from the existing `metadata.json` when storage may already have been modified. Automatically recording its current contents would establish trust on first use at the boundary the integrity mechanism is intended to protect. For compatibility, an unpinned table remains in legacy mode: | Operation | Legacy behavior | | --- | --- | | Load and read | Allowed | | Commit or update | Allowed | | Register-table overwrite | Allowed | | Notification update | Allowed | | Automatic key-ID pinning | Not performed | | Automatic digest backfill | Not performed | | Drop without purge | Allowed | | Drop with server-side purge | Allowed with legacy behavior | These operations preserve existing behavior, but they do not provide the new key-ID or metadata-integrity guarantee. In particular, server-side purge of a legacy table may still consume metadata that has not been checked against trusted catalog state. Ordinary legacy operations must not implicitly move the table into a protected state. A successful commit, register-table overwrite, notification, or purge therefore does not create the marker or derive a trusted digest from storage. The security guarantees apply only to tables whose trusted state was established when they were created under the new model or through a future explicit attestation procedure. ## Future attestation A future attestation mechanism may explicitly move a legacy table into a protected state: ```text legacy -- trusted attestation of key-ID absence --> protected plaintext legacy -- trusted attestation of key ID K and metadata revision H --> protected encrypted(K, H) ``` Attestation must bind at least: - the target Polaris table identity; - the expected `encryption.key-id`, including its absence; and - the exact current metadata revision or its canonical digest. The values must come from a trusted source or an explicit decision by an authorized operator. Attestation must not silently treat the current contents of unprotected object storage as authoritative. ## Metadata admission boundaries For a normal REST create or commit, Polaris stores the trusted marker, key ID, metadata pointer, digest, and digest version in one catalog entity create or compare-and-swap update. Register and notification paths instead begin with metadata that already exists in storage. They therefore define explicit admission boundaries: | Path | Protected-table behavior | | --- | --- | | Normal create | Pin the candidate state produced by the catalog operation | | Normal commit | Validate the pinned key ID and atomically replace the pointer and digest | | Register new | Authorized trust-on-first-use admission; pin the exact revision read by Polaris | | Register overwrite | `TABLE_FULL_METADATA`-authorized re-admission; require the pinned key ID to match, then replace the trusted pointer and digest | | Notification create or update for a missing table | Authorized trust-on-first-use admission; pin the exact revision read by Polaris | | Notification update for a protected plaintext table | Allow only while the pinned key-ID absence remains unchanged | | Notification update for a protected encrypted table | Reject until the request can carry an authenticated expected digest | | Legacy register overwrite or notification update | Preserve legacy behavior without creating a pin or digest | | Refresh/load | Validate the key ID and digest before returning metadata | | Asynchronous cleanup | Validate the copied trusted state before traversing metadata | Iceberg permits encryption table properties only with format version 3 or later. Normal table creation already enforces this through Iceberg, but parsing existing metadata does not. Polaris therefore applies Iceberg's compatibility predicate whenever metadata is pinned or re-pinned, including protected-state admission, protected re-admission, and each normal commit to a pinned table. A pre-v3 revision containing any Iceberg encryption table property, including `encryption.key-id` or `encryption.data-key-length`, is rejected before trusted state is changed. Unpinned legacy tables remain outside this new check. Admission proves only that an authorized principal selected the exact revision read at that operation. It does not prove the revision's history or authenticity before admission. The integrity guarantee begins after admission and lasts until another explicitly authorized register overwrite re-admits a revision. The threat model therefore protects against a principal that can modify metadata storage but cannot perform a catalog-authorized admission or re-admission. The admission boundary includes both the register and notification paths listed above. A principal authorized to replace the metadata pointer, and for an encrypted table its trusted digest, through one of those paths is inside the trusted admission boundary for this guarantee. If such a principal must instead be treated as an attacker, Polaris needs a separate re-admission privilege or a broader redesign of the catalog-management privilege hierarchy; the digest cannot enforce that authorization boundary by itself. Polaris already emits before/after register-table events through its event-listener interface. A deployment that needs a durable audit trail for re-admission must configure a listener that persists those events and their request, principal, and table attributes. The events identify the actor, table, and candidate request, but do not carry both the displaced metadata location and its trusted digest; a complete before/after record must therefore correlate the event with previously persisted trusted state. This proposal does not add a second re-admission event or a metric counter. Notification update is intentionally stricter than register overwrite for encrypted tables. Register overwrite is an explicit re-admission operation authorized as `TABLE_FULL_METADATA`. That privilege is also satisfied by broader catalog-management privileges, so it is not necessarily a human-operator-only boundary. A notification update does not currently bind an expected digest supplied through an authenticated channel, so it must not silently replace an encrypted table's trusted digest. A protected plaintext table has no digest to replace and may continue to update while its pinned key-ID absence remains unchanged. Future notification support may bind at least the Polaris table identity, table UUID, previous metadata revision or sequence information, and expected digest. ## Digest compatibility and read boundary The digest is over Iceberg's canonical `TableMetadataParser` JSON representation, matching the existing HiveCatalog approach. It is not computed from the raw metadata-file bytes. Compression changes therefore do not affect the digest. More specifically, both implementations serialize the parsed `TableMetadata` through `TableMetadataParser.toJson`, compute the SHA-256 digest of the UTF-8 output, Base64-encode the result, and store it outside the metadata file in trusted catalog state. The compatibility policy is intentionally different. HiveCatalog falls back to checking encryption properties when an existing encrypted table has no stored digest. This proposal instead leaves unpinned legacy tables wholly outside the new guarantee, while requiring every protected encrypted table to have a supported digest version and a matching digest. Thus the construction follows HiveCatalog, but the protected-state and missing-digest behavior does not. The implementation hashes the complete canonical representation produced by the active Iceberg parser, rather than a hand-selected subset of fields. A round-trip test exercises snapshots, snapshot history, metadata history, statistics files, partition statistics files, v3 row-lineage fields, and `encryption-keys`. A storage-side change to `encryption-keys` is therefore detected as a digest mismatch. A separate compatibility test parses a frozen JSON fixture and compares its canonical digest with a hard-coded Base64 value. The fixture covers multiple schemas, a partition spec, a sort order, branch and tag references with retention settings, non-empty blob metadata, row-lineage fields, and encrypted-key entries. Unlike the round-trip test, this test fails if an Iceberg upgrade changes canonical output on either side of the write/read boundary. Such a failure requires an explicit compatibility decision; the expected digest must not be mechanically regenerated. This compatibility test must run before an Iceberg dependency upgrade is deployed. Deployment rollback is not assumed to be an available recovery mechanism, because a release may also contain persistence or operational changes that prevent returning to the previous binary. Other valid migration tools include retaining the old canonicalizer, validating by digest version, and performing trusted pre-upgrade verification. This choice has two limitations: - fields ignored by the active Iceberg parser are not included in the canonical digest; and - a future serializer change may require an explicit migration or re-attestation procedure. The stored digest version makes such changes detectable, but does not by itself perform migration. Polaris must retain the old canonicalizer or require trusted re-attestation before changing the version. An integrity mismatch raises a dedicated internal exception and is mapped to HTTP 422. This keeps the failure distinct from retryable storage or server failures and prevents the default Iceberg REST client policy from repeatedly retrying it as a 5xx response. The guarantee covers metadata that Polaris loads, verifies, and returns inline through the catalog path. Standard Iceberg REST clients use that inline metadata. An engine that deliberately ignores it and rereads `metadata-location` directly from object storage operates outside this guarantee. ## Relationship to server-side purge apache/polaris#5060 has an operational responsibility: it reconstructs a KMS client and encryption manager inside a Polaris cleanup worker so that server-side purge can read encrypted manifest lists and manifests. This places the purge worker inside the trusted computing base for each table it purges: the worker must be able to obtain effective unwrap permission for that table. It does not necessarily require one Polaris service identity to access every master key; the effective scope depends on the KMS implementation, credentials, and key policy. The worker should receive only the permissions required for the target table. The current #5060 implementation serializes the catalog property map into durable cleanup tasks so a custom KMS implementation can be reconstructed. Raw secrets are not necessarily present in that map, but copying the whole map expands exposure when a deployment places credentials there. A structured configuration and secret-reference model, resolved when the worker runs, is preferable to a simple property allowlist. This should be coordinated with #5127's proposed configuration model; it is a separate #5060 runtime-wiring concern rather than part of #5185 or #5186. For a protected encrypted table, the intended order is: 1. Load the current metadata referenced by trusted Polaris state. 2. Verify its pinned key ID and trusted metadata digest. 3. Preserve the verified encryption context in the cleanup task. 4. Verify the metadata again in the asynchronous worker. 5. Only then traverse encrypted manifests and select files for deletion. Both verification points fail closed. If the initial catalog load fails, Polaris rejects the drop before removing the table or scheduling cleanup. If asynchronous verification fails after the catalog entity has been dropped, the worker does not traverse or delete metadata, manifests, or data files. The integrity verdict is non-retryable, a terminal error is logged, and the cleanup task remains for operator diagnosis rather than being reported as successful. The old trusted state is present in that task, but no longer in a live table entity; registering the name again would be a new admission, not automatic recovery of the failed purge. For a legacy table, this verification is skipped because no trusted pin or digest exists. Purge continues with legacy behavior and remains outside the new guarantee. Thus #5060, #5185, and #5186 have complementary but distinct scopes. #5060 makes encrypted purge functional; #5185 and #5186 define whether the metadata used by that function is trusted. The delegation-service work in #3786 is relevant to where this server-side operation should run. An asynchronous purge worker can be conceptually compatible with that push-mode architecture; the community has not decided that #5060 is temporary or that the work must move to a separate service. ## Encryption-key entry referential integrity Iceberg also allows entries to be removed from `encryption-keys`. Removing an entry still needed by a retained snapshot can make time-travel reads and purge permanently fail. A complete check must preserve the transitive references from each retained snapshot to its manifest-list key and from that entry to the KEK that encrypted it. The #5186 digest detects an out-of-band storage modification to this list after a revision has been admitted. It does not decide whether an authorized commit that removes an entry is semantically valid, because a successful commit also installs the digest of the new revision. This is therefore still a real metadata referential-integrity and availability concern, but it is not one of the two catalog security requirements above and is not Polaris-specific. Unreferenced keys must remain removable for garbage collection. The validation belongs in a separate Iceberg/Polaris correctness patch rather than in #5185 or #5186. ## Proposed direction The proposed model is: 1. New tables are protected from creation. 2. New plaintext tables pin the absence of `encryption.key-id` but do not store a metadata digest. 3. New encrypted tables pin the exact key ID, and #5186 records the canonical metadata SHA-256 in the same trusted catalog state. 4. Protected tables fail closed on key-ID mismatch and, for encrypted tables, metadata-digest mismatch. 5. Existing unpinned tables remain fully operational in legacy mode and are never enrolled automatically. 6. A separate explicit attestation mechanism may be designed later. 7. Register new, notification admission for a missing table, and register overwrite are explicit authorized admission or re-admission operations. 8. Notification update of an existing protected encrypted table is rejected until an authenticated expected digest can be bound to the request; protected plaintext updates remain allowed while key-ID absence is preserved. 9. The digest format is versioned, and direct storage rereads by non-standard clients are outside the guarantee. 10. A frozen metadata fixture and digest act as an upgrade gate for the v1 canonicalization contract. 11. Integrity failures use a dedicated exception mapped to non-retryable HTTP 422 rather than a generic 5xx response. 12. Storage-supplied metadata with encryption properties is admitted to protected state only when its Iceberg format version is compatible. 13. An asynchronous purge integrity failure is terminal and non-retryable; it is logged and leaves the cleanup task and storage files intact for operator diagnosis. This framing separates backward compatibility from the security guarantee. Polaris can preserve existing tables without claiming that unknown legacy state satisfies Iceberg's catalog security requirements, while providing the required invariants for tables created under the new model.
