mchades commented on code in PR #14378: URL: https://github.com/apache/gravitino/pull/14378#discussion_r4422743423
########## design-docs/gravitino-semantic-model-source-lookup-design.md: ########## @@ -0,0 +1,482 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Semantic Model Dataset Lookup by Source in Gravitino + +## Background + +A Semantic Model dataset references a table or logical View through `Dataset.source`. Sources may +belong to another catalog or schema in the same metalake. Users browsing a source cannot currently +list its associated Semantic Models without collecting and inspecting model definitions themselves. + +Gravitino should provide this lookup through its existing object relationship API pattern. The +result helps users and agents discover models built from a source and identify the datasets to +inspect when that source changes. + +## Goals + +1. **Discovery.** List fully qualified identifiers of Semantic Model datasets directly associated + with a table or View through REST, Java, and Python, across catalogs and schemas in the same + metalake. +2. **Visibility.** Require access to the source metadata and return datasets only from models the + caller can load. +3. **Consistency.** Maintain source associations in the same transaction as model definition writes. +4. **Integration.** Follow existing object relationship APIs, client capability interfaces, and + listener and audit conventions. + +## Non-Goals + +1. Transitive View dependencies, column or metric impact analysis, and validation of whether a + model can currently compile or execute. These checks require additional lineage or compiler + analysis beyond the source references explicitly declared by datasets. +2. Automatically changing model definitions, blocking source changes because models reference the + source, detecting mismatches between model references and current sources, and sending + notifications. These need separate decisions about detecting source changes, permissions to + modify affected models, and handling update or notification failures. +3. Independent association management, exposing internal source IDs, and historical association + queries. Associations are derived from model definitions and updated when those definitions are + written. Editing associations separately could make them disagree with the definitions. Discovery + of current datasets needs neither exposed source IDs nor retained association history. +4. Stable dataset IDs, dataset-specific privileges, and independent dataset lifecycle APIs. Dataset + identifiers locate datasets in the current model definition. Tracking the same dataset across + model versions or authorizing it separately requires rules for dataset identity and lifecycle. + +## Proposal + +### Association Contract + +Each dataset has one association to the registered Gravitino source ID and type resolved during the +latest successful model definition write (create, import, overwrite, or replacement). The model ID +and exact dataset name identify which dataset an association belongs to. Only tables or Views named +in `Dataset.source` are indexed; lookup does not follow the dependencies of a referenced View. + +`Dataset.name` is a case-sensitive alias unique within the model and need not match the table or +View name in `Dataset.source`. + +Definition updates and association updates must commit in the same transaction. Each association's +`dataset_name` must match the corresponding `Dataset.name` in that committed definition. + +Lookup returns one identifier per matching dataset whose model has not been deleted and can be +loaded by the caller. For example, model `analytics.semantic.employee_analysis` in metalake `example` +has two datasets referencing the same source: + +| Model | Dataset | Source | +| ------------------- | ---------- | --------------------- | +| `employee_analysis` | `employee` | `hr.public.employees` | +| `employee_analysis` | `manager` | `hr.public.employees` | + +Storing this model creates two association rows; looking up `hr.public.employees` returns both +dataset identifiers. Callers needing only models derive the containing model identifiers and remove +duplicates. Updates follow [Lifecycle Behavior](#lifecycle-behavior); drift and other constraints +are covered under [Known Limitations](#known-limitations). + +### API Contract + +#### Java API + +Add a read-only capability in the semantic API package: + +```java +/** Lists Semantic Model datasets associated with a source metadata object. */ +public interface SupportsSemanticModels { + + /** + * Lists datasets directly associated with this source from visible Semantic Models. + * + * @return The fully qualified dataset identifiers, or an empty array. + * @throws NoSuchMetadataObjectException If the source does not exist. + */ + NameIdentifier[] listSemanticModelDatasets() throws NoSuchMetadataObjectException; +} +``` + +Each result has five levels in total: namespace `[metalake, catalog, schema, semanticModel]` and the +exact dataset name as `name`, following the existing metric identifier convention. This identifies +a member of the current model definition and does not create a separate dataset metadata object. + +Construct identifiers from components with `NameIdentifier.of(namespace, datasetName)` and preserve +the structured `namespace` and `name` during serialization. Dataset names may contain literal dots. +Joining components with `.` and splitting that string again would treat dots in dataset names as +namespace separators. + +`Table` and `View` add the following default capability accessor, following `supportsTags()`: + +```java +/** + * Returns the Semantic Model dataset lookup operations for this source. + * + * @return The Semantic Model dataset lookup operations. + * @throws UnsupportedOperationException If the implementation does not support lookup. + */ +default SupportsSemanticModels supportsSemanticModels() { + throw new UnsupportedOperationException("Semantic Model dataset lookup is not supported"); +} +``` + +The Gravitino Java client's table and View implementations provide this capability through shared +object relationship request handling. Underlying catalog connectors do not implement the lookup. + +```java +NameIdentifier[] datasets = table.supportsSemanticModels().listSemanticModelDatasets(); +``` + +Results can belong to a different catalog from the source. Java and Python therefore return the same +fully qualified five-level identifiers as REST, including metalake and catalog in the namespace. +For a result `dataset`, `NameIdentifier.of(dataset.namespace().levels())` gives its containing model +identifier. Remove duplicates from these model identifiers to obtain the associated model list. + +#### REST API + +Add a route alongside the existing object tag and policy routes: + +```http +GET /api/metalakes/{metalake}/objects/{type}/{fullName}/semantic-model-datasets +``` + +`type` accepts `table` or `view`. `fullName` contains the source's catalog, schema, and name using the +existing `MetadataObject` format, such as `hr.public.employees`, with normal URL path encoding. +Limits on names containing literal dots are described under [Known Limitations](#known-limitations). + +The endpoint implements `listSemanticModelDatasets` and always returns the existing +`EntityListResponse`, with one identifier per matching dataset. For the employee example above: + +```json +{ + "code": 0, + "identifiers": [ + { + "namespace": ["example", "analytics", "semantic", "employee_analysis"], + "name": "employee" + }, + { + "namespace": ["example", "analytics", "semantic", "employee_analysis"], + "name": "manager" + } + ] +} +``` + +- **Uniqueness and ordering.** Return each dataset identifier once. Sort lexicographically by the + model namespace components (`metalake`, `catalog`, `schema`, `semanticModel`), then by the + case-sensitive dataset name. This order is independent of dataset order in the definition. +- **Empty results.** Return `identifiers: []` when the source exists but no matching datasets are + visible. +- **Errors.** Unsupported source types return `400`. Missing or inaccessible sources follow the + existing object relationship API error behavior, rather than becoming empty successful results. + Storage and source lookup failures fail the request without partial results. + +The response contains identifiers only. Callers select the result's catalog and use the existing +model load API to inspect the containing model and its datasets. No internal IDs, validity status, +hidden-model counts, or association mutation APIs are added. + +#### Python API + +The Python table and View APIs add `supports_semantic_models()`, returning a capability with +`list_semantic_model_datasets() -> list[NameIdentifier]`. It uses the same REST endpoint and full +four-level namespace plus dataset name as Java. Java and Python clients preserve identifier +components and server ordering and do not scan catalogs or load all models to perform lookup. + +### Authorization + +- **Source access.** The REST entry point uses `CAN_ACCESS_METADATA` for the source, as existing + object relationship APIs do. Source type, name normalization, existence checks, and internal ID + resolution reuse the existing table and View paths. Source existence is still checked when + authorization is disabled. +- **Model visibility.** For each distinct candidate model, apply + `LOAD_SEMANTIC_MODEL_AUTHORIZATION_EXPRESSION` in the caller's context using the containing model + identifier, not the dataset identifier. Include or exclude all matching datasets from that model + together. Apply this filtering in REST before constructing the response. +- **Catalog and schema access.** Apply the model load rules for ownership and use or modify + privileges, including access to that model's catalog and schema. Schema listing checks the schema + named in its request before filtering models. This lookup can return models from several schemas + and catalogs, so it must check each model's own parents using the full load expression. +- **Failure handling.** Reuse `MetadataAuthzHelper.filterByExpression`: exclude a model if its + authorization check denies access or throws an exception, and log evaluation exceptions. If the + caller fails the source access check, fail the entire request. When authorization is disabled, + skip model visibility filtering. +- **Association writes.** These are part of an already authorized model definition write and its + source validation. They do not introduce a new privilege or require the source owner to modify + other users' model definitions. + +Dataset-level privileges are not introduced, and owning a source does not grant access to its +associated models. Lookup grants no data access or model modification privileges. + +### Alternatives + +| Approach | Trade-offs | +| -------------------------- | ------------------------------------------------------------------------------------------------------- | +| Scan stored definitions | Avoids a new table, but reads and parses models across the metalake for each lookup. | +| Index source names | Names must be updated after a rename and can refer to different objects after name reuse. | +| Index source IDs and types | Keeps associations across source renames that preserve IDs; requires transactional association updates. | + +This proposal stores source IDs and types in the relation. Model load, export, and compilation +continue to use the source names in the stored definition. The relation records the source objects +resolved when that definition was written; it does not rewrite or override those names. + +A row containing only a model ID and source ID cannot identify which datasets reference that source. +Matching `Dataset.source` against the source's current name is unreliable after a source rename. +Store the dataset name in each association so lookup can return dataset identifiers. Callers can +derive the model list from those identifiers; no second model-level relation table is needed. + +### Internal Storage + +Add `semantic_model_source_rel` without changing the public definition or model snapshot format: Review Comment: The source index narrows the candidate associations; the joins are required to obtain current model identifiers. An index on `semantic_model_id` and `dataset_name` would not satisfy ordering by catalog/schema/model names. The API comparator now defines the response order independently of storage. Additional indexes or result limits need workload evidence. ########## design-docs/gravitino-semantic-model-source-lookup-design.md: ########## @@ -0,0 +1,482 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Semantic Model Dataset Lookup by Source in Gravitino + +## Background + +A Semantic Model dataset references a table or logical View through `Dataset.source`. Sources may +belong to another catalog or schema in the same metalake. Users browsing a source cannot currently +list its associated Semantic Models without collecting and inspecting model definitions themselves. + +Gravitino should provide this lookup through its existing object relationship API pattern. The +result helps users and agents discover models built from a source and identify the datasets to +inspect when that source changes. + +## Goals + +1. **Discovery.** List fully qualified identifiers of Semantic Model datasets directly associated + with a table or View through REST, Java, and Python, across catalogs and schemas in the same + metalake. +2. **Visibility.** Require access to the source metadata and return datasets only from models the + caller can load. +3. **Consistency.** Maintain source associations in the same transaction as model definition writes. +4. **Integration.** Follow existing object relationship APIs, client capability interfaces, and + listener and audit conventions. + +## Non-Goals + +1. Transitive View dependencies, column or metric impact analysis, and validation of whether a + model can currently compile or execute. These checks require additional lineage or compiler + analysis beyond the source references explicitly declared by datasets. +2. Automatically changing model definitions, blocking source changes because models reference the + source, detecting mismatches between model references and current sources, and sending + notifications. These need separate decisions about detecting source changes, permissions to + modify affected models, and handling update or notification failures. +3. Independent association management, exposing internal source IDs, and historical association + queries. Associations are derived from model definitions and updated when those definitions are + written. Editing associations separately could make them disagree with the definitions. Discovery + of current datasets needs neither exposed source IDs nor retained association history. +4. Stable dataset IDs, dataset-specific privileges, and independent dataset lifecycle APIs. Dataset + identifiers locate datasets in the current model definition. Tracking the same dataset across + model versions or authorizing it separately requires rules for dataset identity and lifecycle. + +## Proposal + +### Association Contract + +Each dataset has one association to the registered Gravitino source ID and type resolved during the +latest successful model definition write (create, import, overwrite, or replacement). The model ID +and exact dataset name identify which dataset an association belongs to. Only tables or Views named +in `Dataset.source` are indexed; lookup does not follow the dependencies of a referenced View. + +`Dataset.name` is a case-sensitive alias unique within the model and need not match the table or +View name in `Dataset.source`. + +Definition updates and association updates must commit in the same transaction. Each association's +`dataset_name` must match the corresponding `Dataset.name` in that committed definition. + +Lookup returns one identifier per matching dataset whose model has not been deleted and can be +loaded by the caller. For example, model `analytics.semantic.employee_analysis` in metalake `example` +has two datasets referencing the same source: + +| Model | Dataset | Source | +| ------------------- | ---------- | --------------------- | +| `employee_analysis` | `employee` | `hr.public.employees` | +| `employee_analysis` | `manager` | `hr.public.employees` | + +Storing this model creates two association rows; looking up `hr.public.employees` returns both +dataset identifiers. Callers needing only models derive the containing model identifiers and remove +duplicates. Updates follow [Lifecycle Behavior](#lifecycle-behavior); drift and other constraints +are covered under [Known Limitations](#known-limitations). + +### API Contract + +#### Java API + +Add a read-only capability in the semantic API package: + +```java +/** Lists Semantic Model datasets associated with a source metadata object. */ +public interface SupportsSemanticModels { + + /** + * Lists datasets directly associated with this source from visible Semantic Models. + * + * @return The fully qualified dataset identifiers, or an empty array. + * @throws NoSuchMetadataObjectException If the source does not exist. + */ + NameIdentifier[] listSemanticModelDatasets() throws NoSuchMetadataObjectException; +} +``` + +Each result has five levels in total: namespace `[metalake, catalog, schema, semanticModel]` and the +exact dataset name as `name`, following the existing metric identifier convention. This identifies +a member of the current model definition and does not create a separate dataset metadata object. + +Construct identifiers from components with `NameIdentifier.of(namespace, datasetName)` and preserve +the structured `namespace` and `name` during serialization. Dataset names may contain literal dots. +Joining components with `.` and splitting that string again would treat dots in dataset names as +namespace separators. + +`Table` and `View` add the following default capability accessor, following `supportsTags()`: + +```java +/** + * Returns the Semantic Model dataset lookup operations for this source. + * + * @return The Semantic Model dataset lookup operations. + * @throws UnsupportedOperationException If the implementation does not support lookup. + */ +default SupportsSemanticModels supportsSemanticModels() { + throw new UnsupportedOperationException("Semantic Model dataset lookup is not supported"); +} +``` + +The Gravitino Java client's table and View implementations provide this capability through shared +object relationship request handling. Underlying catalog connectors do not implement the lookup. + +```java +NameIdentifier[] datasets = table.supportsSemanticModels().listSemanticModelDatasets(); +``` + +Results can belong to a different catalog from the source. Java and Python therefore return the same +fully qualified five-level identifiers as REST, including metalake and catalog in the namespace. +For a result `dataset`, `NameIdentifier.of(dataset.namespace().levels())` gives its containing model +identifier. Remove duplicates from these model identifiers to obtain the associated model list. + +#### REST API + +Add a route alongside the existing object tag and policy routes: + +```http +GET /api/metalakes/{metalake}/objects/{type}/{fullName}/semantic-model-datasets +``` + +`type` accepts `table` or `view`. `fullName` contains the source's catalog, schema, and name using the +existing `MetadataObject` format, such as `hr.public.employees`, with normal URL path encoding. +Limits on names containing literal dots are described under [Known Limitations](#known-limitations). + +The endpoint implements `listSemanticModelDatasets` and always returns the existing +`EntityListResponse`, with one identifier per matching dataset. For the employee example above: + +```json +{ + "code": 0, + "identifiers": [ + { + "namespace": ["example", "analytics", "semantic", "employee_analysis"], + "name": "employee" + }, + { + "namespace": ["example", "analytics", "semantic", "employee_analysis"], + "name": "manager" + } + ] +} +``` + +- **Uniqueness and ordering.** Return each dataset identifier once. Sort lexicographically by the + model namespace components (`metalake`, `catalog`, `schema`, `semanticModel`), then by the + case-sensitive dataset name. This order is independent of dataset order in the definition. +- **Empty results.** Return `identifiers: []` when the source exists but no matching datasets are + visible. +- **Errors.** Unsupported source types return `400`. Missing or inaccessible sources follow the + existing object relationship API error behavior, rather than becoming empty successful results. + Storage and source lookup failures fail the request without partial results. + +The response contains identifiers only. Callers select the result's catalog and use the existing +model load API to inspect the containing model and its datasets. No internal IDs, validity status, +hidden-model counts, or association mutation APIs are added. + +#### Python API + +The Python table and View APIs add `supports_semantic_models()`, returning a capability with +`list_semantic_model_datasets() -> list[NameIdentifier]`. It uses the same REST endpoint and full +four-level namespace plus dataset name as Java. Java and Python clients preserve identifier +components and server ordering and do not scan catalogs or load all models to perform lookup. + +### Authorization + +- **Source access.** The REST entry point uses `CAN_ACCESS_METADATA` for the source, as existing + object relationship APIs do. Source type, name normalization, existence checks, and internal ID + resolution reuse the existing table and View paths. Source existence is still checked when + authorization is disabled. +- **Model visibility.** For each distinct candidate model, apply + `LOAD_SEMANTIC_MODEL_AUTHORIZATION_EXPRESSION` in the caller's context using the containing model + identifier, not the dataset identifier. Include or exclude all matching datasets from that model + together. Apply this filtering in REST before constructing the response. +- **Catalog and schema access.** Apply the model load rules for ownership and use or modify + privileges, including access to that model's catalog and schema. Schema listing checks the schema + named in its request before filtering models. This lookup can return models from several schemas + and catalogs, so it must check each model's own parents using the full load expression. +- **Failure handling.** Reuse `MetadataAuthzHelper.filterByExpression`: exclude a model if its + authorization check denies access or throws an exception, and log evaluation exceptions. If the + caller fails the source access check, fail the entire request. When authorization is disabled, + skip model visibility filtering. +- **Association writes.** These are part of an already authorized model definition write and its + source validation. They do not introduce a new privilege or require the source owner to modify + other users' model definitions. + +Dataset-level privileges are not introduced, and owning a source does not grant access to its +associated models. Lookup grants no data access or model modification privileges. + +### Alternatives + +| Approach | Trade-offs | +| -------------------------- | ------------------------------------------------------------------------------------------------------- | +| Scan stored definitions | Avoids a new table, but reads and parses models across the metalake for each lookup. | +| Index source names | Names must be updated after a rename and can refer to different objects after name reuse. | +| Index source IDs and types | Keeps associations across source renames that preserve IDs; requires transactional association updates. | + +This proposal stores source IDs and types in the relation. Model load, export, and compilation +continue to use the source names in the stored definition. The relation records the source objects +resolved when that definition was written; it does not rewrite or override those names. + +A row containing only a model ID and source ID cannot identify which datasets reference that source. +Matching `Dataset.source` against the source's current name is unreliable after a source rename. +Store the dataset name in each association so lookup can return dataset identifiers. Callers can +derive the model list from those identifiers; no second model-level relation table is needed. + +### Internal Storage + +Add `semantic_model_source_rel` without changing the public definition or model snapshot format: + +```sql +CREATE TABLE IF NOT EXISTS `semantic_model_source_rel` ( + `id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'auto increment id', + `metalake_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'containing metalake id', + `semantic_model_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'containing semantic model id', + `dataset_name` MEDIUMTEXT NOT NULL COMMENT 'exact dataset name in the current definition', Review Comment: `Dataset.name` currently has no explicit length limit. Using `VARCHAR(n)` or `TEXT` here could reject definitions accepted by existing storage. `MEDIUMTEXT` preserves that contract; the Dataset names paragraph already documents the rationale. A new name-length limit would need a separate API and compatibility decision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
