yuqi1129 opened a new pull request, #12456: URL: https://github.com/apache/gravitino/pull/12456
### What changes were proposed in this pull request? Add database-backed optimistic concurrency control and transaction boundaries for schema writes. - Advance the schema OCC version on every alter, and guard alter and drop with a compare-and-set on the observed version, classifying a failed CAS as either a stale-version conflict or an already-missing entity. - Make managed schema creation insert-only, and take a shared lock on the parent catalog row, so a concurrent same-name create cannot overwrite the winner and a schema cannot be created below a catalog that is being dropped. - Serialize hierarchical ancestor materialization and schema drops through the catalog row so overlapping cascades share one lock order. - Lock the parent schema row before writing a table, view, fileset, function, model, or topic. - Check views and functions before a non-cascade schema drop. **Accepted tradeoff:** a hierarchical schema create that materializes implicit ancestors takes an *exclusive* lock on the catalog row, so every other schema create under that catalog waits until that transaction ends, even when it touches a different ancestor path. The exclusive lock is needed because two concurrent creates can both find the same ancestor missing and both insert it, and a shared lock does not prevent that under MySQL REPEATABLE READ. Catalogs with heavy concurrent hierarchical schema creation will therefore serialize on this lock. If it becomes a bottleneck, a narrower fence — locking only the ancestor rows being created and relying on the unique constraint plus a retry — can be done in a follow-up. Third of three PRs replacing #12350. Stacked on the catalog PR; review the top commit only. This PR also restores the two cross-entity `TestMetalakeMetaService` tests that could not pass before schema writes took the catalog row lock. ### Why are the changes needed? Managed schema operations previously consisted of multiple independent reads and writes. Concurrent alter, create, and drop requests could overwrite newer metadata, create children below a deleted parent, leave view and function rows orphaned, or run partial cascade cleanup. Overlapping hierarchical schema drops could also acquire descendant row locks in different orders. Fix: #12453 ### Does this PR introduce _any_ user-facing change? Concurrent schema version conflicts are reported as HTTP 409. If the observed entity was deleted or renamed away, alter reports not found and drop preserves its idempotent false result. A managed schema create that loses a concurrent same-name create returns `SchemaAlreadyExistsException` instead of overwriting the winner. ### How was this patch tested? - `./gradlew :core:test :core:javadoc :catalogs:catalog-fileset:test :catalogs:catalog-kafka:test -PskipITs` (H2) - New and extended tests in `TestSchemaMetaService`, `TestMetalakeMetaService`, `TestFilesetCatalogOperations`, `TestKafkaCatalogOperations`. - MySQL and PostgreSQL coverage is left to CI (`-PskipDockerTests=false`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
