yuqi1129 opened a new issue, #12406:
URL: https://github.com/apache/gravitino/issues/12406
### Version
main branch
### Describe what's wrong
Schema cascade deletion is not fully coordinated with concurrent child
metadata writes across Gravitino server instances.
The existing `TreeLock` protects these operations only within one process.
At the relational storage layer, only some child inserts and cross-schema table
moves currently acquire a shared lock on the parent schema row. Existing entity
updates and model-version writes can therefore interleave with a schema cascade
after descendant cleanup has started.
Known unsafe paths include:
- `FunctionMetaService.updateFunction` inserts a new function-version row
and ignores a zero-row metadata update.
- A same-schema `TableMetaService.updateTable` can write a new table-version
row, commit the transaction, and only then report the zero-row metadata update
as a failure.
- A cross-schema table move protects only the destination schema, not the
source schema.
- `ModelVersionMetaService.insertModelVersion` and `updateModelVersion` do
not participate in the schema locking protocol.
- Other table, view, fileset, topic, function, and model update paths do not
consistently validate parent-schema liveness inside their write transactions.
As a result, a schema cascade can commit while an overlapping writer leaves
an active row in a child or version table whose schema or parent entity has
already been deleted.
This is a pre-existing cross-instance concurrency problem. It was identified
while reviewing #12350, but it is outside #12342's metalake/catalog/schema OCC
scope and should be fixed separately.
### Error message and/or stacktrace
The failure can be silent. Depending on the write path, the caller may
receive an `IOException`, a not-found error, or even success while the database
contains an active orphan row.
For example, `updateTable` can commit a new active table-version row and
then throw an `IOException` because its table metadata CAS affected zero rows.
### How to reproduce
One deterministic example is:
1. Create a schema and a table in that schema.
2. Start a table update and pause it after reading the existing table
metadata.
3. Cascade-delete the schema and commit the deletion.
4. Resume the table update.
5. The table metadata update affects zero rows, but the transaction still
writes and commits a new table-version row.
6. The caller receives an update failure while an active orphan version
remains in the database.
A similar race exists for function updates. Model-version insertion can
interleave between the cascade's model-version cleanup and model metadata
deletion.
### Additional context
Implement a complete database-backed schema-to-child write protocol:
- Acquire a shared lock on the active parent schema row as the first
operation in every child create, update, and version-write transaction.
- For cross-schema table moves, validate and lock both the source and
destination schemas.
- Cover table, view, fileset, topic, function, model, and model-version
write paths.
- Treat a zero-row child CAS/update as a transaction failure before any
transaction commit, so version rows, aliases, columns, and change-log writes
are rolled back.
- Preserve existing idempotent behavior for pure delete and
retention-cleanup operations that cannot create or reactivate child rows.
- Add deterministic service-level concurrency tests using the real write
paths rather than invoking the schema-lock helper directly.
- Test both race orderings:
- the writer obtains the shared schema lock first, causing the cascade to
wait;
- the cascade wins first, causing the writer to fail without leaving
active descendants.
- Assert that no affected metadata, version, alias, or column table contains
an active (`deleted_at = 0`) orphan row.
- Run the concurrency tests against H2, MySQL, and PostgreSQL.
The namespace OCC work remains in #12350. Catalog wrapper and classloader
lifetime management remains tracked separately by #12403.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
