nevzheng opened a new issue, #12056: URL: https://github.com/apache/gravitino/issues/12056
### Describe the proposal Gravitino now has native representations for the five type families needed to expose Iceberg V3 schemas through its relational APIs: - Nanosecond `timestamp(9)` and `timestamp_tz(9)` - `variant` - Iceberg `unknown`, represented by Gravitino `NullType` - `geometry(crs)` - `geography(crs, edge_algorithm)` Adding these types to the unified model and Iceberg converter is only the first layer. Every engine and catalog connector must define a deliberate compatibility outcome when these types are created, altered, or loaded. For every connector and type-family combination, the connector must do exactly one of the following: 1. Convert to a semantically equivalent native type and prove a faithful round trip, including precision, timezone, CRS, and edge-algorithm metadata where applicable. 2. Reject with a clear `IllegalArgumentException` before connector-side catalog or database mutation. Silent coercion to a narrower timestamp, string, binary, JSON, or opaque spatial type is not acceptable. **Scope** - Cover all available relational engine, lakehouse, and JDBC connectors. - Exercise create, alter, and load/conversion paths exposed by each connector. - Cover top-level and nested fields where the connector supports complex types. - Verify rejected create and mixed alter requests leave no partial external metadata. - Keep one connector/type family per commit and one connector per review branch/PR. - Add focused tests, a reproducible case, and a connector documentation matrix. - For REST-exposed rejection paths, preserve the invalid-argument contract: HTTP 400 and error code 1001. **Completed foundation** - #11949 — Native Variant type - #11951 — Native Unknown/Null type - #11955 — Native Geometry and Geography types - #11962 — Native nanosecond timestamps - #11954 — Iceberg format-version 3 creation - Discussions #11929 and #11936 — Design and compatibility analysis **Out of scope** - Delta Lake support and newly discovered connector-native gaps. These will be tracked separately for later triage. - A general redesign of Hive timestamp semantics. The Hive compatibility subtask records the completed four-family work and links a separate timestamp investigation. ### Task list **Engine connectors** - [ ] Spark connector - [ ] Flink connector - [ ] Trino connector **Lakehouse and catalog connectors** - [ ] Hive catalog - [ ] AWS Glue catalog - [ ] Paimon catalog - [ ] Hudi catalog - [ ] Lance catalog **Shared JDBC validation** - [ ] JDBC preflight validation **JDBC catalogs** - [ ] MySQL catalog - [ ] PostgreSQL catalog - [ ] Doris catalog - [ ] StarRocks catalog - [ ] ClickHouse catalog - [ ] Hologres catalog - [ ] OceanBase catalog -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
