jerryshao commented on code in PR #12424: URL: https://github.com/apache/gravitino/pull/12424#discussion_r3766377092
########## design-docs/gravitino-semantic-model-design.md: ########## @@ -0,0 +1,670 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Semantic Model Support in Gravitino + +## Background + +Business definitions such as revenue, active users, dimensions, and dataset relationships are +shared semantic assets consumed by analytics, BI, and AI applications. When these definitions live +only in individual tools or project files, their identity, discovery, ownership, access control, +auditability, and reuse become fragmented. Gravitino needs a governed and engine-neutral metadata +object for managing these definitions alongside the data entities they reference. + +Unlike a logical View, which defines a fixed query and output schema, a semantic model defines +datasets, relationships, dimensions, and metrics that consumers combine at query time. Gravitino +therefore models it as a dedicated metadata object rather than a relational View. + +Apache Ossie, formerly Open Semantic Interchange (OSI), defines a vendor-neutral structured +`SemanticModel`. + +Gravitino's design direction for semantic metadata has evolved from preserving opaque YAML +documents, to exposing a strongly typed model, and now to managing it as an independent first-class +metadata object. This progression reflects the growing importance of semantics in Gravitino's +metadata and governance model. This design therefore adopts an Ossie-compatible analytical model as +a new schema-scoped Gravitino entity. + +## Goals + +1. **Lifecycle.** Manage Ossie-compatible Semantic Models as first-class, schema-scoped metadata + with stable identity and a dedicated lifecycle. +2. **Governance.** Apply authorization, ownership, audit, events, tags, and policies to Semantic + Models. +3. **Validation.** Define deterministic write-time checks and clear boundaries for catalog-dependent + validation. +4. **Interoperability.** Define a stable Ossie-compatible contract that can evolve with Apache Ossie. +5. **User experience.** Support discovery and lifecycle management of Semantic Models as a distinct + schema-scoped object category in the Gravitino UI. + +## Non-Goals + +1. **Query compilation and execution.** Semantic query planning, SQL generation, engine execution, + and engine-specific query syntax are separate work. +2. **Non-Ossie native models.** Native dbt, Cube, Databricks, Snowflake, or other vendor-specific + semantic definitions are not modeled by this design. +3. **Document authoring and conversion.** YAML or JSON import, export, formatting, conversion, and + exact textual round trips are not server API contracts. +4. **Ontology management.** Apache Ossie Ontology definitions and mappings are separate metadata + concepts and require their own design. +5. **Materialization.** Metric caches, refresh policies, pre-aggregations, and materialized results + are not defined here. +6. **Member-level authorization.** Datasets, fields, relationships, and metrics are governed as + members of their enclosing Semantic Model rather than as independent securable objects. + +## Proposal + +### Object Model and Namespace + +`SemanticModel` is a new metadata entity under a schema, alongside Tables, Views, Functions, and +other schema-scoped objects. + +```text +metalake + catalog + schema + Table + View + SemanticModel +``` + +The implementation introduces `MetadataObject.Type.SEMANTIC_MODEL` and +`Entity.EntityType.SEMANTIC_MODEL`. Public Java types reside in a dedicated semantic package rather +than the relational View package. + +- **Identity.** The Semantic Model entity name maps to the Ossie `SemanticModel.name` field when + serialized. No separate nested model name or identity is stored. +- **Scope.** A Semantic Model belongs to one schema. Its fully qualified identity is + `metalake.catalog.schema.semanticModel`. +- **Namespace.** Semantic Models have an independent typed namespace. A Semantic Model may have the + same name as a Table, View, Function, or another entity type in the same schema, but two Semantic + Models with the same name cannot coexist. +- **Source of truth.** Semantic Models are always managed by Gravitino and are never persisted in an + underlying catalog. +- **One model per entity.** One Ossie `semantic_model` item maps to one Gravitino Semantic Model. An + external document containing multiple items maps to multiple entities. + +The term `SemanticModel` means an **analytical semantic model** compatible with the Ossie Core +specification. It is not an umbrella for every semantic artifact. The +[Ossie Ontology specification](https://github.com/apache/ossie/blob/88e0011148283302c9a04cd0287e00e0b9d87354/ontology/ontology.json) +defines Ontology as a separate document and maps logical semantic models to ontology concepts. +Gravitino should likewise introduce a separate `Ontology` entity and an explicit binding or mapping +contract if that capability is added later. Such a binding should reference the stable identities of +the Ontology and Semantic Model rather than embed a second copy of either definition. + +`SemanticModel` is a separate metadata type and lifecycle from the existing Gravitino `Model`, which +represents an ML model artifact. + +### Semantic Model Contract + +The API exposes a structured, immutable model. The initial contract follows the +[Apache Ossie Core schema pinned at commit `88e0011`](https://github.com/apache/ossie/blob/88e0011148283302c9a04cd0287e00e0b9d87354/core-spec/osi-schema.json), +whose declared specification version is `0.2.0.dev0`. Fields marked with `?` are optional; all other +fields are required. Language-specific APIs follow their normal naming conventions, while this +document uses Ossie field names for comparison with the upstream schema. + +The Gravitino entity combines common metadata with the Ossie-compatible definition: + +```text +SemanticModel + name: string + comment?: string + ai_context?: AIContext + datasets: Dataset[1..*] + relationships?: Relationship[] + metrics?: Metric[] + custom_extensions?: CustomExtension[] + properties: map<string, string> + audit_info: AuditInfo +``` + +- `name` is the Gravitino entity name and maps to Ossie `SemanticModel.name`. +- `comment` follows the common Gravitino entity convention and maps to Ossie + `SemanticModel.description`. +- `properties` stores Gravitino-specific metadata and is not part of an exported Ossie model. +- `custom_extensions` is part of the semantic definition and is preserved for Ossie interchange. +- Collection order is preserved so consumers can produce stable serialized output. + +#### Datasets and Fields + +```text +Dataset + name: string + source: NameIdentifier + primary_key?: string[] + unique_keys?: string[][] + description?: string + ai_context?: AIContext + fields?: Field[] + custom_extensions?: CustomExtension[] + +Field + name: string + expression: Expression + dimension?: Dimension + label?: string + description?: string + datatype?: DataType + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- Dataset names are unique within a Semantic Model. Field names are unique within each Dataset. +- `source` is a three-part `NameIdentifier` in the form `catalog.schema.name`. The request already + identifies the metalake, so cross-catalog references within that metalake are allowed while + cross-metalake references are not. +- A source must resolve to either a Table or a logical View. The semantic definition does not need + to declare which of those two types it references; validation succeeds when either entity exists. +- Apache Ossie does not currently define cross-model dataset references. Therefore, a Semantic Model + is not a valid Dataset source; cross-model composition is deferred until an explicit compatible + reference contract is defined. +- Inline query sources are not supported. A query-backed source must first be created as a logical + View and then referenced by `NameIdentifier`. +- For Table and logical View sources, every column named by `primary_key`, `unique_keys`, + `from_columns`, or `to_columns` must exist in the source metadata. Validation rejects the + definition when column metadata is unavailable or a named column is missing; Gravitino does not + infer these references from free-form field or metric expressions. + +#### Relationships and Metrics + +```text +Relationship + name: string + from: string + to: string + from_columns: string[1..*] + to_columns: string[1..*] + ai_context?: AIContext + custom_extensions?: CustomExtension[] + +Metric + name: string + expression: Expression + description?: string + datatype?: DataType + ai_context?: AIContext + custom_extensions?: CustomExtension[] +``` + +- Relationship and Metric names are unique within a Semantic Model. +- Each relationship endpoint names a Dataset in the same Semantic Model. +- `from_columns` and `to_columns` are non-empty, have equal lengths, and name columns exposed by + their respective Dataset sources. +- Metrics may reference fields and datasets in the same Semantic Model. Cross-model metric + references are not defined by this contract. + +#### Supporting Types + +```text +Expression + dialects: DialectExpression[1..*] + +DialectExpression + dialect: Dialect + expression: string + +Dimension + is_time?: boolean + +AIContext = string | AIContextObject + +AIContextObject + instructions?: string + synonyms?: string[] + examples?: string[] + additional properties: allowed and retained losslessly + +CustomExtension + vendor_name: string + data: string + +Dialect = "ANSI_SQL" | "SNOWFLAKE" | "MDX" | "TABLEAU" + | "DATABRICKS" | "MAQL" | "BIGQUERY" + +DataType = "String" | "Integer" | "Decimal" | "Float" | "Boolean" + | "Date" | "Time" | "DateTime" | "DateTimeTz" | "Opaque" +``` + +- Every Expression has at least one DialectExpression. +- Each dialect appears at most once in an Expression and each expression string is non-empty. +- Unknown model fields are rejected where the pinned Ossie schema sets `additionalProperties` to + `false`. +- Every supported `custom_extensions` array and every additional AI-context property is retained + losslessly. + +#### Validation + +Create and alter validate the complete candidate object before persistence. Validation is atomic: +if any check fails, no change is persisted. + +1. **Contract validation.** Check required fields, collection cardinality, enum values, string + constraints, and the pinned Ossie structure. +2. **Model-local validation.** Check name uniqueness, relationship endpoints, referenced fields, + key shapes, and dialect uniqueness without consulting a catalog. +3. **Catalog validation.** Resolve every Dataset source and validate explicitly named source + columns. Validation uses the caller's authorization context so it does not disclose metadata the + caller cannot access. + +Invalid definitions return `400`. Missing sources or source columns are invalid definitions. A +temporarily unavailable source catalog returns `503` so the caller can retry. Authorization failures +return `403`. + +Catalog changes do not automatically revalidate stored Semantic Models. A later source rename, +drop, or schema change can therefore leave a previously valid definition with an unavailable +reference. Automatic revalidation would require a durable dependency index and cross-catalog impact +analysis; it is outside this design. Loads continue to return the stored definition, while consumers +must handle unavailable sources when they compile or use it. + +Gravitino does not validate the execution semantics of expressions, transitive logical View +dependencies, fanout correctness, or engine compatibility. Those checks require a compiler or query +engine and remain outside the metadata write path. + +#### Ossie Compatibility + +The public API is a Gravitino contract derived from Ossie, not a stored YAML or JSON document. For +schema conformance, the server projects a candidate object into an in-memory Ossie document: + +```yaml +version: 0.2.0.dev0 +semantic_model: + - name: sales_model + description: Governed sales definitions + datasets: + - name: orders + source: sales.mart.orders +``` + +The projection maps the entity name to `SemanticModel.name`, `comment` to `description`, and each +source `NameIdentifier` to its three-part string. The request path supplies the metalake and is not +serialized into Ossie. + +The request path does not invoke the upstream Python validator or create an intermediate YAML +string. The implementation validates the in-memory projection against a bundled copy of the pinned +JSON Schema, then runs Gravitino-specific model-local and catalog checks. Upstream validation tools +are used in conformance tests for generated YAML and JSON. + +The entity carries no per-model Ossie specification version. Each Gravitino release pins one exact +upstream schema commit and defines the supported read and write contract. Updating that schema +requires a Gravitino code change, compatibility fixtures, and storage read tests. If a future +incompatible Ossie contract must coexist with the current one, explicit per-entity versioning +requires a separate design; it should not be inferred from `custom_extensions`. + +### API and Lifecycle + +`Catalog` exposes `asSemanticModelCatalog()`. Because definitions are stored by Gravitino, support Review Comment: Only supports the relational catalog. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
