LiJie20190102 commented on code in PR #13431: URL: https://github.com/apache/gravitino/pull/13431#discussion_r4072767221
########## design-docs/stale-registration-reconcile.md: ########## @@ -0,0 +1,296 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design: Reconcile Stale Registrations of Non-Managed Entities + +--- + +## 1. Background + +For non-managed catalogs (JDBC, Hive, Iceberg, Kafka, ...) the source system is +the source of truth. Gravitino keeps only a registration row per +schema/table/view/topic/fileset to attach owners, tags, policies, audit +information and properties. + +Registrations are kept in sync only on Gravitino's own write path. When an +object is created, renamed or dropped directly in the source, nothing +reconciles the registration: + +- A dropped schema stays live in `schema_meta`. Consumers that walk the + catalog (dashboard metrics, lineage, search sync) resolve the name from the + store and then fail on `loadSchema` with 404. See + [#13279](https://github.com/apache/gravitino/issues/13279) for the + drop-side symptom. +- `TableOperationDispatcher` and `SchemaOperationDispatcher` deliberately + preserve a registration when the source drop reports `false`, because a + `false` is ambiguous between "renamed" and "dropped out of band". A true + out-of-band drop therefore always leaves a stale row. + +A few code paths already delete registrations directly through +`EntityStore.delete` when they notice the source object is gone (e.g. +`IcebergTableHookDispatcher.deleteTableEntity`, +`SchemaEntityCleaner.deleteOrphanedSchemaEntities`). These ad-hoc deletes +bypass the dispatcher chain, so they skip secret cleanup, authorization-plugin +privilege removal, `Drop*Event` emission and orphan cleanup, and when run from +another process they race with concurrent creates because tree locks are per +JVM. + +This design proposes one server-side reconciliation mechanism that removes +stale registrations through the dispatcher chain, so a reconcile-triggered +removal behaves exactly like an explicit drop. + +--- + +## 2. Goals + +1. Define what "stale" means per entity type and how to confirm absence in the + source safely. +2. Add a server-side reconcile task for schemas in non-managed catalogs that + removes stale registrations through `SchemaDispatcher`, with events, + secret cleanup, authorization-plugin privilege removal and orphan cleanup. +3. Provide both a periodic trigger and an on-demand REST API, with a policy + switch between report-only and auto-remove. +4. Expose stale registrations via REST so administrators can inspect them + before removal. + +--- + +## 3. Non-Goals + +- Tables, views, topics and filesets: the mechanism is designed to extend to + them, but the first implementation covers schemas only. +- UI surfacing: the REST response carries everything a UI needs, but no web + page is added in this phase. +- Changing `list*` behavior for non-managed catalogs (tracked separately in + the epic). +- The reverse direction (object exists in source but is not registered): + already handled by lazy import on `loadSchema`/`loadTable`. +- Managed catalogs (fileset, model, generic-lakehouse): Gravitino owns the + storage there, no source drift is possible. + +--- + +## 4. Existing Architecture Overview + +### 4.1 Dispatcher chain + +``` +REST → EventDispatcher → NormalizeDispatcher → HookDispatcher → OperationDispatcher +``` + +Dropping a schema through `SchemaDispatcher` currently does all of the +following: + +- `SchemaOperationDispatcher.dropSchema`: deletes from the source catalog, + deletes the store registration, cleans write-through secrets via + `SecretManager.deleteSecretsFromProperties`, all under a WRITE tree lock on + the catalog node (`TreeLockUtils.doWithTreeLock`). (On cascade, filesets + are dropped via `FilesetDispatcher` first, before the catalog lock is + acquired, so each fileset cleans its own write-through secrets and no + nested tree locks are taken.) +- `SchemaHookDispatcher`: post-drop, calls + `authorizationPluginRemovePrivileges` so Ranger plugins drop privileges. +- `SchemaEventDispatcher`: emits `DropSchemaEvent` for audit, search index + and webhooks. + +### 4.2 What #13279 already added + +`SchemaOperationDispatcher.dropSchema(ident, cascade=true)` removes the +registration even when the source schema is already absent, reports +`dropped: true` if either side was removed, and cleans stored secrets. +Non-cascading drops keep the old conservative behavior. + +This gives reconcile an existing, fully-wired removal entry point: removing a +stale schema is a cascading drop where the source side is already gone. + +### 4.3 Existence probes + +Connectors expose explicit per-object existence probes +(`SupportsSchemas.schemaExists`, `TableCatalog.tableExists`, ...). These are +single-object probes, not listings, so they are not affected by listing +permission filters. + +### 4.4 Background task pattern + +There is no central scheduler. Server subsystems each own a +`ScheduledExecutorService` with `start()`/`close()`, wired in +`GravitinoEnv.initGravitinoServerComponents()`, with interval config keys in +`Configs`. `RelationalGarbageCollector` is the closest reference. + +--- + +## 5. Proposed Design + +### 5.1 Definition of stale + +A registration is stale for a given entity when all of the following hold: + +1. The owning catalog does not manage storage for that entity scope + (`catalog.capabilities().managedStorage(Capability.Scope.SCHEMA).supported()` + is false; same pattern as `CatalogManager.isManagedStorageCatalog`). +2. An explicit existence probe against the source reports the object absent + (`NoSuch*Exception` from the probe, never a missing entry in a listing). +3. The probe did not fail for infrastructural reasons (connection failure, + timeout, authentication): those mark the catalog "unreachable" and skip + all removals for that catalog in this round. + +### 5.2 StaleSchemaReconciler + +New class `StaleSchemaReconciler` in `core`, following the +`RelationalGarbageCollector` pattern: + +``` +every reconcileIntervalSecs (default: disabled): + for each non-managed catalog in each metalake: Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
