nevzheng opened a new pull request, #14274:
URL: https://github.com/apache/gravitino/pull/14274
### What changes were proposed in this pull request?
- **HDFS catalog removal.** `RangerAuthorizationHDFSPlugin` now handles
`CATALOG` removal: it removes
the Gravitino-managed items of the policy on the catalog location, and
deletes the policy if nothing
else is left. It used to do nothing.
- **Shared-path guard.** The plugin keeps the policy when another Hive
catalog in the metalake uses the
same path (trailing `/` ignored). It reads catalog info from the entity
store and loads no catalog.
- **Only the dropped catalog's principals go.** One HDFS policy per path can
also carry grants of another
catalog's schema or fileset placed exactly there. A Gravitino role stays
when one of its securable
objects outside the dropped catalog is located at the path; the dropped
catalog's owner role is
removed and other catalogs' owner roles stay; metalake-level grants, the
metalake owner role and
object owners stay only when another object uses the path. A failed
location or role lookup refuses
the drop.
- **Strict lookup.** New `AuthorizationUtils.loadMetadataObjectLocation`
throws on failure.
`getMetadataObjectLocation` stays lenient on top of it, so other callers
are unchanged.
- **Plugins declare their need.** New SPI default method
`MetadataAuthorizationPlugin#requiresLocations()`,
`false` by default. The HDFS plugin returns `true`;
`ChainedAuthorizationPlugin` returns `true` if any
member does.
- **Schemas and tables of a force-dropped catalog.** When the HDFS plugin
removes a catalog whose
location no other catalog uses, it also removes the Gravitino-managed
items of the policies on the
locations of the catalog's schemas and tables, as dropping each would. All
of those locations are
looked up before any policy changes, and a failed listing or lookup
refuses the drop. A schema on the
catalog location (the default schema) shares the catalog's policy and is
skipped.
- **Refuse before side effects.** New
`AuthorizationUtils.removeCatalogPrivileges(ident)` does the
lookup itself. If the lookup failed and the plugin needs locations, it
throws
`AuthorizationPluginException` before any plugin call or delete.
`CatalogHookDispatcher#dropCatalog`
uses it.
### Why are the changes needed?
**Every Hive catalog drop with the Ranger HDFS plugin left HDFS path
policies behind.** The plugin
ignored catalog removals, and a failed location lookup returned an empty
list that the plugin read as
"nothing to remove". A force drop also removed the catalog's schemas and
tables without their drop
hooks, so their path policies stayed too. The grants outlived the catalog,
and a new catalog, schema or
table on the same path picked them up. Fixing this by clearing the whole
catalog-path policy would also
remove grants of other catalogs' objects placed at that path, so only the
dropped catalog's principals
are removed.
Fix: #14269
Fix: #14270
### Does this PR introduce _any_ user-facing change?
- **Catalog drop can now fail** with `AuthorizationPluginException` when the
catalog uses the HDFS plugin
(directly or in a chain) and its location cannot be looked up, for example
when the metastore is
down. `force=true` does not bypass this. To drop such a catalog without
cleanup, remove its
authorization provider first; the error message says so.
- **New SPI default method** `requiresLocations()`. Existing plugins keep
compiling.
### How was this patch tested?
**`TestRangerAuthorizationHDFSPluginCatalogDrop`** (23 cases) runs the full
chain:
`CatalogHookDispatcher` -> `AuthorizationUtils` -> the real HDFS plugin ->
an in-memory Ranger.
- **Catalog path:** removes Gravitino items and keeps admin items; deletes a
policy with only Gravitino
items; another Hive catalog on another path still lets the policy go; a
non-Hive sibling is not
loaded; a path shared with another catalog (trailing `/`) keeps its
policy; a default schema without
a location does not block the drop.
- **Refusals:** the drop is refused, with Ranger untouched, when the
catalog's location lookup fails or
another catalog's lookup fails.
- **Force drop:** removes the policies of schemas (including one outside the
warehouse) and tables;
refused when a schema location lookup, the schema listing or the table
listing fails; keeps the
schema policies of a shared warehouse.
- **Shared path:** keeps the grants of another catalog's schema or fileset
at the warehouse path, and
the owners when such a schema is granted; removes them when that schema is
on another path; keeps
other catalogs' owner roles and removes the dropped catalog's; keeps a
metalake grant only when the
path is shared; refused when a role object's location lookup fails.
- **On `main`:** 19 of the 23 fail. The 4 that pass are "keep" and "no
location" edges.
- **With the fix:** all 23 pass.
**Unit:** `TestAuthorizationUtils` (23) covers strict vs lenient lookup,
refusing when the plugin needs
locations, proceeding when it does not, and no plugin.
`TestChainedAuthorizationPlugin` (2) covers the
any-member rule. `:authorizations:authorization-ranger:test` and
`:authorizations:authorization-chain:test`
pass. No live Ranger IT was run.
`./gradlew -PskipITs :authorizations:authorization-ranger:test --tests
'*TestRangerAuthorizationHDFSPluginCatalogDrop' :core:test --tests
'*TestAuthorizationUtils'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
