LiJie20190102 commented on code in PR #13431: URL: https://github.com/apache/gravitino/pull/13431#discussion_r4072748272
########## design-docs/stale-registration-reconcile.md: ########## @@ -0,0 +1,296 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design: Reconcile Stale Registrations of Non-Managed Entities + +--- + +## 1. Background + +For non-managed catalogs (JDBC, Hive, Iceberg, Kafka, ...) the source system is +the source of truth. Gravitino keeps only a registration row per +schema/table/view/topic/fileset to attach owners, tags, policies, audit +information and properties. + +Registrations are kept in sync only on Gravitino's own write path. When an +object is created, renamed or dropped directly in the source, nothing +reconciles the registration: + +- A dropped schema stays live in `schema_meta`. Consumers that walk the + catalog (dashboard metrics, lineage, search sync) resolve the name from the + store and then fail on `loadSchema` with 404. See + [#13279](https://github.com/apache/gravitino/issues/13279) for the + drop-side symptom. +- `TableOperationDispatcher` and `SchemaOperationDispatcher` deliberately + preserve a registration when the source drop reports `false`, because a + `false` is ambiguous between "renamed" and "dropped out of band". A true + out-of-band drop therefore always leaves a stale row. + +A few code paths already delete registrations directly through +`EntityStore.delete` when they notice the source object is gone (e.g. +`IcebergTableHookDispatcher.deleteTableEntity`, +`SchemaEntityCleaner.deleteOrphanedSchemaEntities`). These ad-hoc deletes +bypass the dispatcher chain, so they skip secret cleanup, authorization-plugin +privilege removal, `Drop*Event` emission and orphan cleanup, and when run from +another process they race with concurrent creates because tree locks are per +JVM. + +This design proposes one server-side reconciliation mechanism that removes +stale registrations through the dispatcher chain, so a reconcile-triggered +removal behaves exactly like an explicit drop. + +--- + +## 2. Goals + +1. Define what "stale" means per entity type and how to confirm absence in the + source safely. +2. Add a server-side reconcile task for schemas in non-managed catalogs that Review Comment: for the SCHEMA entity type -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
