jerryshao opened a new pull request, #13307:
URL: https://github.com/apache/gravitino/pull/13307
### What changes were proposed in this pull request?
Keep column tags consistent when a column is dropped or renamed.
**Dropped columns (#13304)**
- `MetadataObjectService.getColumnObjectsFullName` returns `null` for a
column whose latest row is `DELETE`. A tag's object list therefore no longer
shows dropped columns. This also covers relations left over from before this
fix.
- `TableColumnMetaService.updateColumnPOsFromTableDiff` now soft-deletes the
tag and owner relations of dropped columns. This runs in the same transaction
as the table update, in batches of 1000 ids, using a new
`softDeleteTagMetadataObjectRelsByMetadataObjects` mapper method. Policies
cannot be attached to columns, so there is nothing to clean up for them.
- Orphan relation GC now treats a column as dead when its latest row is
`DELETE`. Before, any live row counted as live, so relations of dropped columns
were never collected.
**Renamed columns (#13305)**
- `TableOperationDispatcher.alterTable` replays the normalized
`TableChange`s (`resolveColumnNameChanges`) to work out each existing top-level
column's new name. It then uses that when matching stored columns to the
catalog's columns, so a renamed column keeps its id and its tags. The replay
handles:
- chained renames,
- swaps through a temporary name,
- a rename plus adding a new column with the old name,
- a drop plus re-adding the same name in one change (the new column gets a
new id).
Nested fields are ignored.
- `selectColumnIdByTableIdAndName` matches the name against each column's
latest row only. A renamed column's old name, and a dropped column's name, no
longer resolve to it.
- The load path (`updateColumnsIfNecessaryWhenLoad`) now matches columns
again against the entity passed to `store.update`, which is the latest stored
one. Before, it wrote a column list computed before taking the lock. A
concurrent load could therefore undo a rename that an in-flight `alterTable`
had just stored, which dropped the column's tags.
### Why are the changes needed?
- Dropping a tagged column left it in `GET /tags/{tag}/objects`, while the
column itself reported `NoSuchMetadataObjectException`. The two views disagreed.
- Renaming a tagged column through `alterTable` on an external catalog
(Hive, Iceberg, …) stored the rename as a drop plus a new column with a new id.
The column lost its tags, and the tag still listed the old name.
Fix: #13304, #13305
Part of #13303
### Does this PR introduce _any_ user-facing change?
No API or configuration change. Behaviour changes:
- A column renamed through Gravitino keeps its tags and owner.
- Dropped columns no longer appear in a tag's object list.
- The old name of a renamed column no longer resolves.
Columns renamed outside Gravitino, e.g. directly in Spark or Hive, still
can't be told apart from a drop plus an add. Their relations are now
soft-deleted instead of being left dangling.
### How was this patch tested?
- **Unit tests** (H2 locally):
- `TestTagMetaService.testTagRelationsFollowColumnDropAndRename`
- `TestTableColumnMetaService` (old-name lookup, dropped-column full name,
batched relation cleanup over more than one batch)
-
`TestOrphanedMetadataObjectRelationService.testSoftDeleteRelationsOfDroppedColumns`
- `TestTableOperationDispatcher`:
`testAlterTableKeepsColumnIdsAcrossRenames`, `testResolveColumnNameChanges`,
`testLoadDoesNotUndoConcurrentColumnRename`
- **Integration tests** (Hive, Docker):
`TagIT.testDroppedColumnIsRemovedFromTagObjects` and
`TagIT.testRenamedColumnKeepsItsTags` replay the reported REST steps.
- `./gradlew :core:test -PskipITs` passes locally. The MySQL/PostgreSQL
backends and `TagIT` still need to run in CI.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
