jerryshao opened a new issue, #13303:
URL: https://github.com/apache/gravitino/issues/13303
### Describe the proposal
Tags, policies, owners and privileges are stored against a metadata object's
**id**. Several code paths break that model, so a tag's (or policy's) object
list and an object's own tag list can disagree, or governance metadata silently
disappears or lands on the wrong object.
Two user-visible symptoms on an Iceberg (lakehouse) catalog:
1. **Dropping a tagged column**: `GET /tags/{tag}/objects` still lists the
dropped column, while `GET /objects/column/{col}/tags` correctly returns
`NoSuchMetadataObjectException`.
2. **Renaming a tagged column**: the tag doesn't follow the new name, and
the tag still lists the old column name.
Root causes found while auditing `main` and `branch-1.3`:
- Column drop inserts a version row with `column_op_type = DELETE` and
`deleted_at = 0`. `MetadataObjectService.getColumnObjectsFullName` doesn't
filter these rows, and nothing cleans the column's relations at drop time.
- `TableOperationDispatcher.updateColumnsIfNecessary` matches columns by
name only, so a rename is stored as delete + add with a new column id.
- Several delete paths soft-delete the parent row before the relation
cleanup, whose SQL requires the parent to still be live (`deleted_at = 0`), so
the cleanup updates nothing.
- `branch-1.3` has no `OrphanedMetadataObjectRelationService`, so leaked
relation rows there are never collected.
Goals:
- A tag's or policy's object list never contains an object that no longer
exists, and agrees with the object's own tag or policy list.
- A rename or move that changes an object's full name carries its tags,
policies, owner and privileges.
- Recreating an object with the same name never inherits the old object's
metadata.
Fixes should land on `main` and be backported to `branch-1.3` where
applicable. The note at the end of each task says whether each item affects
`main`, `branch-1.3` or both.
### Task list
**Column identity and lifecycle**
- [ ] Dropped columns still appear in tag object lists. Filter `DELETE` rows
in `getColumnObjectsFullName`, soft-delete a column's tag, policy and owner
relations when the column is dropped, and make orphan relation GC treat a
column whose latest row is `DELETE` as dead. (main, 1.3)
- [ ] Column rename via `alterTable` on external catalogs loses tags. Use
`TableChange.RenameColumn` to keep the column id, handling chained renames and
a rename plus add of the old name in one request. (main, 1.3)
- [ ] `selectColumnIdByTableIdAndName` resolves a column's pre-rename name,
even after that column is dropped. It should match on each column's latest
version. This must be fixed together with the rename fix above. (main, 1.3)
- [ ] Re-importing a table (e.g. after an out-of-band rename) regenerates
every column id and drops the column-level relations. `importTable` should
reuse the existing column ids. (main, 1.3)
- [ ] Lance `VERSION_CHECK` schema refresh gives every column a new id on
each dataset version change (`replaceColumnsFromDataset`). (main, 1.3)
- [ ] Moving a table to another schema together with a column change leaves
the unchanged column rows under the old `schema_id`. A later cascade drop of
the old schema deletes those columns and their relations. (main, 1.3)
- [ ] The load path computes the column list before taking the lock
(`updateColumnsIfNecessaryWhenLoad`). A concurrent alter or load can mark live
columns deleted or allocate two ids for the same column. (main, 1.3)
- [ ] Managed and Lance tables treat nested-field changes (`fieldName.length
> 1`) as top-level column names. The Lance dataset changes but the Gravitino
metadata update fails. (main, 1.3)
**Relation cleanup on delete**
- [ ] Dropping a metalake doesn't clean tag relations or role securable
objects: the tags and roles are soft-deleted first. (main, 1.3)
- [ ] Dropping a metalake never calls
`softDeletePolicyMetadataObjectRelsByMetalakeId`. A non-cascade metalake drop
also leaves the policies and policy versions live. (main, 1.3)
- [ ] Deleting a tag doesn't clean its object relations (ordering bug), its
owner or its privileges. (1.3 only)
- [ ] Deleting a policy doesn't clean its object relations, owner or
privileges. (1.3 only)
- [ ] Deleting a job template or job doesn't clean their owner relations or
securable objects. (main, 1.3)
- [ ] Non-cascade schema drop doesn't check for views or functions before
dropping. (1.3)
**Rename, stale registration and caches**
- [ ] A stale table registration is inherited when a table with the same
name is recreated. On MySQL/H2 the upsert keeps the old id, so the old tags,
policies and privileges carry over. On PostgreSQL the insert fails and the
table becomes unloadable. (main (wider), 1.3)
- [ ] Renaming a table across schemas passes the old schema to the
authorization plugins (`TableHookDispatcher` TODO). (main, 1.3)
- [ ] For catalogs that don't persist the Gravitino id, an out-of-band
rename of a table, schema or view imports the object with a new id. The old
registration lingers and can't be dropped through the API. (main, 1.3)
- [ ] `put(overwrite)` doesn't invalidate the entity cache entry for the old
name after an import renames the row. (main, 1.3)
- [ ] Renaming or deleting a model, tag or policy doesn't call
`notifyEntityNameIdMappingChange`, so the authorizer's nameāid cache can go
stale. (main, 1.3)
**Tests**
- [ ] Add tests that assert the relation rows themselves after drop and
rename, not only name-based lookups, which hide leaked rows. (main, 1.3)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
