Hi Ramin,
thank you for evolving the VARIANT type further. I fully support the FLIP.
The ask for a UUID data type came up a couple of times in the past
because UUIDs are in almost every data pipeline.
Also full timestamp and time support incl nano second precision is a
topic that always pops up and support for it avoids friction.
One question the statement: "time without time zone (us)"
Since Flink SQL only supports TIME(3), I guess we limit VARIANT also to
millis within the SQL runtime, right? For DataStream API we can support
9, right?
Cheers,
Timo
On 04.08.26 12:28, Ramin Gharib wrote:
Hi everyone,
I'd like to start a discussion on FLIP-604: Complete VARIANT
Primitive Coverage with UUID and Timestamps. [1]
FLIP-521 [2] shipped VARIANT in Flink 2.1, but it only implements Parquet
Variant primitive codes 0 through 16. The open Variant spec goes
through 20: TIME, nanosecond timestamps, and UUID. So a VARIANT
written by Spark, Iceberg, or any Parquet writer that holds one of
these values cannot be read by Flink today, which breaks the
lakehouse round-trip VARIANT was meant to support.
This FLIP completes the primitive set. TIME and the nanosecond
timestamps reuse types Flink already has. UUID needs a type to land
on, so the FLIP also adds a native 16-byte UUID type as a first-class
SQL type, with casts, ordering, a UUID literal, and UUID_V4 and
UUID_V7 generator functions.
One point worth noting: no Calcite upgrade is required. Calcite has
had a UUID type since 1.39, the same release FLIP-521 adopted for
VARIANT, and Flink already runs Calcite 1.41. The 16-byte layout also
matches every engine with a native UUID type, including PostgreSQL,
Trino, DuckDB, ClickHouse, Snowflake, and Iceberg.
Looking forward to your feedback.
Best,
Ramin Gharib
[1]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
[2]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/349637099/FLIP-521+Integrating+Variant+Type+into+Flink+Enabling+Efficient+Semi-Structured+Data+Processing
