Thanks for raising this, Dmitri. These are valid concerns, and they were already recognized when we introduced JDBC persistence to Polaris. At that time, we chose to use AtomicOperationMetaStoreManager for the JDBC due to the simplicity. I think most of the issues mentioned here can already be addressed by TransactionalMetaStoreManagerImpl.
For example, rename is already wrapped in a transaction in TransactionalMetaStoreManagerImpl [1]. Similarly, catalog creation, which involves reading and creating multiple objects, is also executed within a transaction [2]. I see two possible directions: 1. Modify AtomicOperationMetaStoreManager together with the persistence backends (such as JDBC) to provide the required consistency guarantees for specific operations, similar to what TransactionalMetaStoreManagerImpl does. 2. Migrate the persistence backends (such as JDBC) to use TransactionalMetaStoreManagerImpl directly. We may have to deal with transactional semantic mismatches across different persistence backends. For example, we would likely avoid using JDBC's `runWithinTransaction` for single row updates, which adds additional overhead and complexity without benefits. References: 1. https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 2. https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 Yufei On Thu, Jul 16, 2026 at 8:22 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi all, > > Ayush and Prithvi recently contributed a couple of interesting PRs: > [4939], [5035]. > > It looks like people are starting to encounter consistency issues in > JDBC persistence. > > The PRs provide valuable insight into the underlying issues. They offer > incremental fixes that can work. However, I believe it is time for the > Polaris community to review and improve this area of the codebase > holistically. > > By this, I mean finding a solution that can be applied to all > persistence backends (in-memory, JDBC, NoSQL) and addresses these > aspects: > > * Supporting concurrent and consistent changes where the service reads > and validates current catalog state, then commits a change (e.g. > name clashes during renames). > * Supporting consistent but independent changes to RBAC grants and > MetaStore entities. This independence is needed to support > external authorizers like OPA and Ranger. > * Supporting atomic changes across multiple similar entities. > * Supporting authorization-based filtering of list operations (cf. > [4831]). > * Supporting credential-vending decisions that are rooted in the > exact state of the catalog. > * Supporting server-side retries for transient persistence failures > (e.g. RDBMS Tx serializability failures). > > Please share your comments and ideas. > > [4831] https://github.com/apache/polaris/pull/4831 > > [4939] https://github.com/apache/polaris/pull/4939 > > [5035] https://github.com/apache/polaris/pull/5035 > > Thanks, > Dmitri >
