mchades commented on code in PR #12424: URL: https://github.com/apache/gravitino/pull/12424#discussion_r3797146663
########## design-docs/gravitino-semantic-model-design.md: ########## @@ -0,0 +1,670 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Semantic Model Support in Gravitino + +## Background + +Business definitions such as revenue, active users, dimensions, and dataset relationships are +shared semantic assets consumed by analytics, BI, and AI applications. When these definitions live +only in individual tools or project files, their identity, discovery, ownership, access control, +auditability, and reuse become fragmented. Gravitino needs a governed and engine-neutral metadata +object for managing these definitions alongside the data entities they reference. + +Unlike a logical View, which defines a fixed query and output schema, a semantic model defines +datasets, relationships, dimensions, and metrics that consumers combine at query time. Gravitino +therefore models it as a dedicated metadata object rather than a relational View. + +Apache Ossie, formerly Open Semantic Interchange (OSI), defines a vendor-neutral structured +`SemanticModel`. + +Gravitino's design direction for semantic metadata has evolved from preserving opaque YAML +documents, to exposing a strongly typed model, and now to managing it as an independent first-class +metadata object. This progression reflects the growing importance of semantics in Gravitino's +metadata and governance model. This design therefore adopts an Ossie-compatible analytical model as +a new schema-scoped Gravitino entity. + +## Goals + +1. **Lifecycle.** Manage Ossie-compatible Semantic Models as first-class, schema-scoped metadata + with stable identity and a dedicated lifecycle. +2. **Governance.** Apply authorization, ownership, audit, events, tags, and policies to Semantic + Models. +3. **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent + validation. +4. **Interoperability.** Define a stable Ossie-compatible contract that can evolve with Apache Ossie. +5. **User experience.** Support discovery and lifecycle management of Semantic Models as a distinct + schema-scoped object category in the Gravitino UI. + +## Non-Goals + +1. **Query compilation and execution.** Semantic query planning, SQL generation, engine execution, + and engine-specific query syntax are separate work. +2. **Non-Ossie native models.** Native dbt, Cube, Databricks, Snowflake, or other vendor-specific + semantic definitions are not modeled by this design. +3. **Document authoring and conversion.** YAML or JSON import, export, formatting, conversion, and + exact textual round trips are not server API contracts. +4. **Ontology management.** Apache Ossie Ontology definitions and mappings are separate metadata + concepts and require their own design. +5. **Materialization.** Metric caches, refresh policies, pre-aggregations, and materialized results + are not defined here. +6. **Member-level authorization.** Datasets, fields, relationships, and metrics are governed as + members of their enclosing Semantic Model rather than as independent securable objects. + +## Proposal + +### Object Model and Namespace + +`SemanticModel` is a new metadata entity under a schema, alongside Tables, Views, Functions, and +other schema-scoped objects. + +```text +metalake + catalog + schema + Table + View + SemanticModel +``` + +The implementation introduces `MetadataObject.Type.SEMANTIC_MODEL` and +`Entity.EntityType.SEMANTIC_MODEL`. Public Java types reside in a dedicated semantic package rather +than the relational View package. + +- **Identity.** The Semantic Model entity name maps to the Ossie `SemanticModel.name` field when + serialized. No separate nested model name or identity is stored. +- **Scope.** A Semantic Model belongs to one schema. Its fully qualified identity is + `metalake.catalog.schema.semanticModel`. +- **Namespace.** Semantic Models have an independent typed namespace. A Semantic Model may have the + same name as a Table, View, Function, or another entity type in the same schema, but two Semantic + Models with the same name cannot coexist. +- **Source of truth.** Semantic Models are always managed by Gravitino and are never persisted in an + underlying catalog. +- **One model per entity.** One Ossie `semantic_model` item maps to one Gravitino Semantic Model. An + external document containing multiple items maps to multiple entities. + +The term `SemanticModel` means an **analytical semantic model** compatible with the Ossie Core +specification. It is not an umbrella for every semantic artifact. The +[Ossie Ontology specification](https://github.com/apache/ossie/blob/88e0011148283302c9a04cd0287e00e0b9d87354/ontology/ontology.json) +defines Ontology as a separate document and maps logical semantic models to ontology concepts. +Gravitino should likewise introduce a separate `Ontology` entity and an explicit binding or mapping +contract if that capability is added later. Such a binding should reference the stable identities of +the Ontology and Semantic Model rather than embed a second copy of either definition. + +`SemanticModel` is a separate metadata type and lifecycle from the existing Gravitino `Model`, which +represents an ML model artifact. + +### Semantic Model Contract + +The API exposes a structured, immutable model. The initial contract follows the +[Apache Ossie Core schema pinned at commit `88e0011`](https://github.com/apache/ossie/blob/88e0011148283302c9a04cd0287e00e0b9d87354/core-spec/osi-schema.json), +whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other +fields are required. Language-specific APIs follow their normal naming conventions, while this +document uses Ossie field names for comparison with the upstream schema. + +The Gravitino entity combines common metadata with the Ossie-compatible definition: + +```text +SemanticModel + name: string + comment?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] + properties: map<string, string> + audit_info: AuditInfo +``` + +- `name` is the Gravitino entity name and maps to Ossie `SemanticModel.name`. +- `comment` follows the common Gravitino entity convention and maps to Ossie + `SemanticModel.description`. +- `properties` stores Gravitino-specific metadata and is not part of an exported Ossie model. +- `custom_extensions` is part of the semantic definition and is preserved for Ossie interchange. +- Collection order is preserved so consumers can produce stable serialized output. + +#### Datasets and Fields + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + datatype?: DataType + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- Dataset names are unique within a Semantic Model. Field names are unique within each Dataset. +- `source` is a three-part `NameIdentifier` in the form `catalog.schema.name`. The request already + identifies the metalake, so cross-catalog references within that metalake are allowed while + cross-metalake references are not. +- A source must resolve to either a Table or a logical View. The semantic definition does not need + to declare which of those two types it references; validation succeeds when either entity exists. +- Apache Ossie does not currently define cross-model dataset references. Therefore, a Semantic Model + is not a valid Dataset source; cross-model composition is deferred until an explicit compatible + reference contract is defined. +- Inline query sources are not supported. A query-backed source must first be created as a logical + View and then referenced by `NameIdentifier`. +- For Table and logical View sources, every column named by `primary_key`, `unique_keys`, + `from_columns`, or `to_columns` must exist in the source metadata. Validation rejects the + definition when column metadata is unavailable or a named column is missing; Gravitino does not + infer these references from free-form field or metric expressions. + +#### Relationships and Metrics + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + datatype?: DataType + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- Relationship and Metric names are unique within a Semantic Model. +- Each relationship endpoint names a Dataset in the same Semantic Model. +- `from_columns` and `to_columns` are non-empty, have equal lengths, and name columns exposed by + their respective Dataset sources. +- Metrics may reference fields and datasets in the same Semantic Model. Cross-model metric + references are not defined by this contract. + +#### Supporting Types + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | AIContextObject + +AIContextObject + instructions?: string + synonyms?: string[] + examples?: string[] + additional properties: allowed and retained losslessly + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" + | "DATABRICKS" | "MAQL" | "BIGQUERY" + +DataType = "String" | "Integer" | "Decimal" | "Float" | "Boolean" + | "Date" | "Time" | "DateTime" | "DateTimeTz" | "Opaque" +``` + +- Every Expression has at least one DialectExpression. +- Each dialect appears at most once in an Expression and each expression string is non-empty. +- Unknown model fields are rejected where the pinned Ossie schema sets `additionalProperties` to + `false`. +- Every supported `custom_extensions` array and every additional AI-context property is retained + losslessly. + +#### Validation + +Create and alter validate the complete candidate object before persistence. Validation is atomic: +if any check fails, no change is persisted. + +1. **Contract validation.** Check required fields, collection cardinality, enum values, string + constraints, and the pinned Ossie structure. +2. **Model-local validation.** Check name uniqueness, relationship endpoints, referenced fields, + key shapes, and dialect uniqueness without consulting a catalog. +3. **Catalog validation.** Resolve every Dataset source and validate explicitly named source + columns. Validation uses the caller's authorization context so it does not disclose metadata the + caller cannot access. + +Invalid definitions return `400`. Missing sources or source columns are invalid definitions. A +temporarily unavailable source catalog returns `503` so the caller can retry. Authorization failures +return `403`. + +Catalog changes do not automatically revalidate stored Semantic Models. A later source rename, +drop, or schema change can therefore leave a previously valid definition with an unavailable +reference. Automatic revalidation would require a durable dependency index and cross-catalog impact +analysis; it is outside this design. Loads continue to return the stored definition, while consumers +must handle unavailable sources when they compile or use it. + +Gravitino does not validate the execution semantics of expressions, transitive logical View +dependencies, fanout correctness, or engine compatibility. Those checks require a compiler or query +engine and remain outside the metadata write path. + +#### Ossie Compatibility + +The public API is a Gravitino contract derived from Ossie, not a stored YAML or JSON document. For +schema conformance, the server projects a candidate object into an in-memory Ossie document: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_model + description: Governed sales definitions + datasets: + - name: orders + source: sales.mart.orders +``` + +The projection maps the entity name to `SemanticModel.name`, `comment` to `description`, and each +source `NameIdentifier` to its three-part string. The request path supplies the metalake and is not +serialized into Ossie. + +The request path does not invoke the upstream Python validator or create an intermediate YAML +string. The implementation validates the in-memory projection against a bundled copy of the pinned +JSON Schema, then runs Gravitino-specific model-local and catalog checks. Upstream validation tools +are used in conformance tests for generated YAML and JSON. + +The entity carries no per-model Ossie specification version. Each Gravitino release pins one exact +upstream schema commit and defines the supported read and write contract. Updating that schema +requires a Gravitino code change, compatibility fixtures, and storage read tests. If a future +incompatible Ossie contract must coexist with the current one, explicit per-entity versioning +requires a separate design; it should not be inferred from `custom_extensions`. + +### API and Lifecycle + +`Catalog` exposes `asSemanticModelCatalog()`. Because definitions are stored by Gravitino, support +does not depend on whether the underlying connector implements a semantic-model capability. + +```text +SemanticModelCatalog + listSemanticModels(namespace): NameIdentifier[] + loadSemanticModel(identifier): SemanticModel + createSemanticModel(identifier, comment, definition, properties): SemanticModel + alterSemanticModel(identifier, changes...): SemanticModel + dropSemanticModel(identifier): boolean +``` + +Expected exceptions include `NoSuchSemanticModelException`, +`SemanticModelAlreadyExistsException`, and `InvalidSemanticModelException`. +`listSemanticModels` returns identifiers rather than complete definitions so listing a schema does +not transfer every model body. Callers use `loadSemanticModel` to retrieve selected models. + +#### Alter Semantics + +Supported `SemanticModelChange` operations are: + +- `rename`: Changes the entity name and therefore the Ossie name produced by future serialization. +- `updateComment`: Replaces the model comment. +- `setProperty` and `removeProperty`: Update Gravitino-specific properties. +- `replaceDefinition`: Atomically replaces AI context, datasets, relationships, metrics, and custom + extensions. + +Each alter request applies all changes atomically to the current Semantic Model. Rename retains the +stable entity ID. Owner, tag, and policy changes use their existing governance stores and remain +outside `SemanticModelChange`. + +Fine-grained member patch operations are not included. Datasets, relationships, fields, and metrics +have interdependent validation rules, so replacing the complete definition provides a clear atomic +contract. Additional typed changes can be added later without changing stored identity. + +#### Events + +Semantic Model operations use Gravitino's existing listener framework. List, load, create, alter, +and drop emit corresponding pre, success, and failure events. Rename is represented as an alter +event. Event payloads follow existing Gravitino conventions, and this design introduces no new +delivery or ordering guarantees. Tag and policy operations continue to use the existing Tag and +Policy event dispatchers. Association events identify the target metadata object as +`SEMANTIC_MODEL` and do not emit Semantic Model alter events. + +#### Java API + +```java +NameIdentifier ident = NameIdentifier.of("semantic", "sales_model"); + +Expression orderAmountExpression = + Expression.builder() + .withDialects( + new DialectExpression[] { + DialectExpression.builder() + .withDialect(Dialect.ANSI_SQL) + .withExpression("order_amount") + .build() + }) + .build(); + +Field orderAmount = + Field.builder() + .withName("order_amount") + .withExpression(orderAmountExpression) + .withDatatype(DataType.DECIMAL) + .build(); + +Dataset orders = + Dataset.builder() + .withName("orders") + .withSource(NameIdentifier.of("sales", "mart", "orders")) + .withFields(new Field[] {orderAmount}) + .build(); + +Expression totalRevenueExpression = + Expression.builder() + .withDialects( + new DialectExpression[] { + DialectExpression.builder() + .withDialect(Dialect.ANSI_SQL) + .withExpression("SUM(orders.order_amount)") + .build() + }) + .build(); + +Metric totalRevenue = + Metric.builder() + .withName("total_revenue") + .withExpression(totalRevenueExpression) + .withDescription("Total revenue across all orders") + .withDatatype(DataType.DECIMAL) + .build(); + +SemanticModelDefinition definition = + SemanticModelDefinition.builder() + .withDatasets(new Dataset[] {orders}) + .withMetrics(new Metric[] {totalRevenue}) + .build(); + +SemanticModel created = + catalog + .asSemanticModelCatalog() + .createSemanticModel( + ident, "Governed sales definitions", definition, Map.of()); + +SemanticModel loaded = catalog.asSemanticModelCatalog().loadSemanticModel(ident); +NameIdentifier[] models = + catalog.asSemanticModelCatalog().listSemanticModels(Namespace.of("semantic")); + +SemanticModel updated = + catalog + .asSemanticModelCatalog() + .alterSemanticModel( + ident, + SemanticModelChange.updateComment("Updated sales definitions"), + SemanticModelChange.replaceDefinition(definition)); + +boolean dropped = catalog.asSemanticModelCatalog().dropSemanticModel(ident); +``` + +`SemanticModelDefinition` is an immutable request value that groups the definition fields. It has no +name or independent lifecycle and is not another metadata object. The returned `SemanticModel` +exposes its definition fields directly. + +#### REST API + +Semantic Models use a dedicated resource and do not reuse `/views`: + +```http +POST /api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/semantic-models +{ + "name": "sales_model", + "comment": "Governed sales definitions", + "datasets": [ + { + "name": "orders", + "source": { + "namespace": ["sales", "mart"], + "name": "orders" + } + } + ], + "relationships": [], + "metrics": [], + "properties": {} +} +``` + +List, load, alter, and drop operations use the following resources: + +```http +GET /api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/semantic-models +GET /api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/semantic-models/{name} +PUT /api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/semantic-models/{name} +DELETE /api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/semantic-models/{name} +``` + +An alter request applies all changes atomically: + +```http +PUT /api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/semantic-models/sales_model +{ + "updates": [ + { + "@type": "updateComment", + "newComment": "Updated sales definitions" + }, + { + "@type": "replaceDefinition", + "definition": { + "datasets": [ + { + "name": "orders", + "source": { + "namespace": ["sales", "mart"], + "name": "orders" + } + } + ], + "relationships": [], + "metrics": [] + } + } + ] +} +``` + +#### Python API + +The Python client mirrors the Java lifecycle and structured types: + +```python +ident = NameIdentifier.of("semantic", "sales_model") +orders = Dataset("orders", NameIdentifier.of("sales", "mart", "orders")) +definition = SemanticModelDefinition(datasets=[orders]) + +created = catalog.as_semantic_model_catalog().create_semantic_model( + ident, + "Governed sales definitions", + definition, + {}, +) +loaded = catalog.as_semantic_model_catalog().load_semantic_model(ident) +models = catalog.as_semantic_model_catalog().list_semantic_models( + Namespace.of("semantic") +) +updated = catalog.as_semantic_model_catalog().alter_semantic_model( + ident, + SemanticModelChange.update_comment("Updated sales definitions"), +) +dropped = catalog.as_semantic_model_catalog().drop_semantic_model(ident) +``` + +### Storage and Parent Lifecycle + +Semantic Models are persisted through the Gravitino EntityStore as dedicated entities with stable +entity IDs. Their typed metadata is stored independently of Table and View entity types. Storage +follows the identity-and-version pattern used by View and Function: one table stores the stable +identity and current version pointer, while another stores complete version snapshots. + +#### Relational Schema + +The following MySQL-style DDL defines the logical schema. H2 and PostgreSQL use equivalent types and +indexes following the existing EntityStore conventions. + +```sql +CREATE TABLE IF NOT EXISTS `semantic_model_meta` ( + `semantic_model_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'semantic model id', + `semantic_model_name` VARCHAR(128) NOT NULL COMMENT 'semantic model name', + `metalake_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'metalake id', + `catalog_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'catalog id', + `schema_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'schema id', + `audit_info` MEDIUMTEXT NOT NULL COMMENT 'semantic model identity audit info', + `current_version` INT UNSIGNED NOT NULL DEFAULT 1 COMMENT 'current version', + `last_version` INT UNSIGNED NOT NULL DEFAULT 1 COMMENT 'last allocated version', + `deleted_at` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0 COMMENT 'semantic model deleted at', + PRIMARY KEY (`semantic_model_id`), + UNIQUE KEY `uk_sid_smn_del` (`schema_id`, `semantic_model_name`, `deleted_at`), + KEY `idx_smm_mid` (`metalake_id`), + KEY `idx_smm_cid` (`catalog_id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin + COMMENT 'semantic model metadata'; + +CREATE TABLE IF NOT EXISTS `semantic_model_version_info` ( + `id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'auto increment id', + `metalake_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'metalake id', + `catalog_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'catalog id', + `schema_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'schema id', + `semantic_model_id` BIGINT(20) UNSIGNED NOT NULL COMMENT 'semantic model id', + `version` INT UNSIGNED NOT NULL COMMENT 'semantic model version', + `semantic_model_name` VARCHAR(128) NOT NULL COMMENT 'semantic model name snapshot', + `semantic_model_comment` TEXT DEFAULT NULL COMMENT 'semantic model comment snapshot', + `semantic_model_definition` MEDIUMTEXT NOT NULL COMMENT 'structured definition snapshot (JSON)', + `properties` MEDIUMTEXT DEFAULT NULL COMMENT 'semantic model properties snapshot (JSON)', + `audit_info` MEDIUMTEXT NOT NULL COMMENT 'semantic model version audit info', + `deleted_at` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0 COMMENT 'version deleted at', + PRIMARY KEY (`id`), + UNIQUE KEY `uk_smid_ver_del` (`semantic_model_id`, `version`, `deleted_at`), + KEY `idx_smvi_mid` (`metalake_id`), + KEY `idx_smvi_cid` (`catalog_id`), + KEY `idx_smvi_sid` (`schema_id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin + COMMENT 'semantic model version information'; +``` + +`semantic_model_definition` contains AI context, datasets, relationships, metrics, and custom +extensions as structured JSON managed by typed converters. Keeping the definition as one snapshot +preserves collection order and allows create and alter to validate and persist the complete model +atomically. Child members are not stored in separate tables because they do not have independent +identity, lifecycle, or authorization. + +#### Version and Lifecycle Behavior + +- Create writes the identity row and version 1 snapshot in one transaction. +- Alter writes a complete new snapshot, increments `last_version`, and advances `current_version` in + the same transaction. Rename also updates `semantic_model_meta.semantic_model_name`; the version + snapshot retains the name at the time of the change. Review Comment: Semantic Model alters will compare-and-set the observed `current_version`. A concurrent update will return `409` with `OPTIMISTIC_LOCK_CONFLICT_CODE` instead of silently overwriting the newer version. I will make this explicit in the design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
