jerryshao opened a new pull request, #13320: URL: https://github.com/apache/gravitino/pull/13320
> **Depends on #13308 and #13316.** This branch is stacked on #13308 (the branch-1.3 backport of #13307), so the diff also shows its commits until #13308 is merged. Please review only the last 3 commits ([compare](https://github.com/jerryshao/gravitino/compare/backport-13304-13305-branch-1.3...backport-13309-13310-branch-1.3)). This PR stays a draft until #13308 and #13316 are merged; I'll then rebase it onto `branch-1.3` and mark it ready for review. ### What changes were proposed in this pull request? Backport of #13316 to `branch-1.3`, **except for the re-import fix** (see below). - **Cross-schema table move with a column change (#13309):** `TableColumnMetaService.updateColumnPOsFromTableDiff` moves all of the table's column rows to the new `schema_id` whenever the table moves. Before, it did this only when no column changed, so a later cascade drop of the old schema deleted the unchanged columns and their relations. - **Lance `VERSION_CHECK` schema refresh (#13309):** `LanceTableOperations.replaceColumnsFromDataset` reuses the id, comment and audit info of each column still in the dataset, so column tags and comments survive a dataset version change. - **Cross-schema table rename and authorization (#13310):** `TableHookDispatcher.alterTable` passes the new schema from `RenameTable.getNewSchemaName()` to the authorization plugins, through a new `AuthorizationUtils.authorizationPluginRenamePrivileges` overload that takes the new `NameIdentifier`. **Not backported: keeping column ids when a table is re-imported** (e.g. loaded after an out-of-band rename). On `branch-1.3`, `insertTable(overwrite = true)` resets the table version and `RelationalEntityStore.put(overwrite)` caches the incoming entity. Reusing column ids there would also need cache and column-query changes, which is too risky for this release. That fix stays on `main` only. The Lance change is a clean cherry-pick. So is the authorization change. The schema-move change is the corresponding part of the first commit of #13316, ported by hand. ### Why are the changes needed? - Moving a table to another schema and changing a column in one `alterTable`, then dropping the old schema, deleted the unchanged columns and their tags, owners and privileges. - On a Lance catalog with `lance.schema-refresh-mode=VERSION_CHECK`, every dataset version change gave all columns new ids and cleared their comments. - On a cross-schema rename, authorization plugins were told the table moved to `<old schema>.<new name>`. Fix: #13309, #13310 Part of #13303 ### Does this PR introduce _any_ user-facing change? No API or configuration change. Column tags, owners and privileges survive a table move with a column change and a Lance schema refresh. Lance columns keep their comments. Authorization plugins get the correct schema on a cross-schema table rename. ### How was this patch tested? - **Unit tests** (H2 locally; each fails without its fix): - `TestTableColumnMetaService.testMoveTableWithColumnChangeMovesAllColumns` - `TestLanceTableOperations.testVersionCheckRefreshKeepsExistingColumnIdsAndComments` - `TestTableHookDispatcher`: `testRenameAcrossSchemasPassesNewSchemaToAuthorization`, `testRenameTwiceKeepsSchemaFromEarlierRename` - `TestAuthorizationUtils.testRenameTableAcrossSchemasNotifiesAuthorizationPluginWithNewSchema` - **Integration tests**: - `CatalogGenericCatalogLanceIT.testVersionCheckRefreshKeepsColumnTagsAndComments` passes locally. - `TagIT.testMovedTableKeepsColumnTagsAfterOldSchemaIsDropped` needs Docker. - `./gradlew :core:test -PskipITs` and the `catalog-lakehouse-generic` tests pass locally on this branch. The MySQL/PostgreSQL backends and the Docker-based ITs still need to run in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
