Hi all, I’d like to get some feedback on how we should handle ambiguous persistence outcomes during Iceberg table metadata commits when Polaris uses the relational JDBC metastore.
Today, an Iceberg metadata update can be executed as a JDBC auto-commit statement. If the database applies the update but the connection fails before the JDBC driver receives the result, Polaris can’t tell whether the catalog update actually committed. A client retry can then use stale table state and get a normal optimistic concurrency conflict. For Iceberg clients, that distinction is important: CommitFailedException represents a definite commit failure and can trigger cleanup of newly written manifest files, even though the catalog may already point to metadata referencing those files. The proposal is: 1. Add a persistence signal for an ambiguous write outcome. 2. Have the JDBC implementation raise it only for connection/I/O failures from PreparedStatement.executeUpdate(), where the statement may have already reached the database. 3. Keep definite failures retryable, including serialization failures and connection-acquisition failures where no statement was sent. 4. In LocalIcebergCatalog table-like updates, catch the ambiguous signal and reload the persisted entity. 5. If the persisted metadata location matches the requested metadata location, treat the operation as successful. 6. Otherwise, return Iceberg’s CommitStateUnknownException(500) for which prevents the client from deleting the staged metadata files instead of a definite concurrency conflict. There’s no additional work on the normal commit path; reconciliation only happens after an ambiguous JDBC failure. The draft implementation uses default methods on BasePersistence and PolarisMetaStoreManager, so only JDBC opts into ambiguity detection. Transactional and non-JDBC persistence implementations keep their current behavior. The main question I’d like feedback on is whether this persistence-SPI approach makes sense, or whether maintainers would prefer a different mechanism for propagating a backend-specific “write outcome unknown” signal up to the Iceberg catalog layer. I also have a draft PR with JDBC unit coverage, existing optimistic-concurrency coverage, and an end-to-end test that simulates an update being persisted while the response is lost. The test verifies that the committed manifest list and manifests remain available. Bug Ticket: https://github.com/apache/polaris/issues/5459 PR: https://github.com/apache/polaris/pull/5458 Thanks. -- Arun Suri Staff Software Engineer He/him/his Engineering | Fivetran [email protected] fivetran.com <//fivetran.com> <http://www.fivetran.com> [image: facebook] <https://www.facebook.com/Fivetran/> [image: twitter] <https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor> [image: linkedin] <https://www.linkedin.com/company/fivetran> [image: instagram] <https://www.instagram.com/fivetran_ig/>
