GitHub user voonhous edited a comment on the discussion: RFC-99: Hudi Type System
+1 for Option 1. I shared the same concern in RFC-88 here: https://github.com/apache/hudi/pull/12795#discussion_r1946516814 It's essential for **reliable interoperability and long-term data correctness**. The Core Problem: A "Many-to-Many" Mapping Nightmare Without a standardized table-level type system, every component in the stack is forced into a "many-to-many" mapping problem: Compute Engines (C): Flink, Trino, Spark, Hive, etc., all have their own internal logical type representations. File Formats (F): Parquet, ORC, Avro all have their own physical types. For example: The `Timestamp` type. We have: - **Flink**: `TIMESTAMP(3)` (millis), `TIMESTAMP(9)` (nanos), local vs. UTC semantics. - **Trino**: `TIMESTAMP(p)` with precision, `TIMESTAMP(p) WITH TIME ZONE`. - **Parquet**: Legacy `INT96` (which had ambiguous interpretations), `TIMESTAMP_MICROS`, `TIMESTAMP_MILLIS`. Going with Option 1 allows for a strict contract where there is a well defined "Hub-and-Spoke" abstraction. The type system acts as the central logical "hub". It breaks the C-to-F problem down into a much simpler C + F problem. `Engine Type (Flink)` <-> `Table Format Type (Hudi)` <-> `File Format Type (Parquet)` - For Writers (e.g., Flink): The Flink connector's job is to map `Flink.Timestamp` to the corresponding `Hudi.Timestamp` type. The Hudi format itself then takes responsibility for serializing that logical `Hudi.Timestamp` into the correct, unambiguous Parquet physical type (e.g., TIMESTAMP_MICROS with isAdjustedToUTC=true, timezone specifics). It is also up to Hudi to dictate which TIMESTAMP type(s) it wants to support. - For Readers (e.g., Trino): The Trino connector's job is to read the `Hudi.Timestamp` from table schema and map it to `Trino.Timestamp`. It doesn't need to know or care if the physical bytes on disk are Parquet INT96 or ORC struct { seconds, nanos }. The table format handles that abstraction. Option 2 feels fragile where we are replacing Avro with another Type system where we assume that , but we are not really solving the actual problem. Option 1 is the only reliable way to achieve the following: 1. Guarantee Semantic Consistency 2. Decouple Engines from Storage: It allows the file formats (like Parquet) to evolve (e.g., add new types) and the engines (like Flink) to evolve (e.g., add new precisions) independently. (as long as we have the specific implementations) GitHub link: https://github.com/apache/hudi/discussions/14253#discussioncomment-14965171 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
