Hi all, Thanks for the FLIP ! I'm +1 on this. Once users store data as VARIANT, changing one field or building an object from columns are among the first things they want to do, and writing a UDF for that is a big ask. I also like that SQL NULL is now handled the same way across all functions.
On the open question, I would keep NULL. Dremio has an IS_VARIANT_NULL that returns NULL for SQL NULL [1], and so do Snowflake's IS_NULL_VALUE and our own IS JSON. Spark is the exception. With NULL, one call also tells a missing field, a variant null and a value apart. It might be worth adding Dremio to the prior-art table. Best regards, Martijn [1] https://docs.dremio.com/dremio-cloud/sql/sql-functions/functions/IS_VARIANT_NULL Op vr 23 okt 2026 om 13:51 schreef Timo Walther <[email protected]>: > Hi everyone, > > thanks Ramin for updating the FLIP. It took us quite some iterations to > get it into a consistent and high quality shape. > > For context, these are items we put into consideration: > - Look at other vendors: BigQuery, Snowflake, Spark, DuckDB > - Look at existing Flink SQL built-in functions e.g. JSON_*, IS_* and > collection functions > - Look at use cases and find usability gaps > - Provide a reasonable set of building blocks > - What is required to make VARIANT in Flink a large success? > > The FLIP covers a lot of function at once. Some are not described in > every detail, but we will cover this as part of PR discussions and > writing docs. > > Overall, I hope we can make many users happy. > > Thanks, > Timo > > > On 23.10.26 13:32, Ramin Gharib wrote: > > Hi everyone, > > > > After a long discussion with Timo Walther, I updated the FLIP to > > change how the functions handle SQL NULL. > > > > The functions now handle SQL NULL and variant null consistently. A > > SQL NULL argument gives a SQL NULL result. > > > > To write a variant null on purpose, the FLIP adds two functions: > > > > - VARIANT_NULL() returns a variant null, the same as > > PARSE_JSON('null'). > > - TO_VARIANT(x) is the same as CAST(x AS VARIANT), except for SQL > > NULL. CAST(NULL AS VARIANT) returns NULL, so TO_VARIANT gives a > > variant null instead. > > > > For example, with discount = NULL: > > > > VARIANT_SET(payload, '$.discount', discount) > > -- NULL > > VARIANT_SET(payload, '$.discount', TO_VARIANT(discount)) > > -- {"discount":null,"order_id":8,"user":"bob"} > > > > Other changes: > > > > - VARIANT_OBJECT, VARIANT_ARRAY and VARIANT_MERGE return NULL for a > > SQL NULL argument. A NULL key in VARIANT_OBJECT is still an error. > > - VARIANT_OBJECT_AGG skips rows with a SQL NULL value. > > - VARIANT_DELETE and VARIANT_PICK still skip a NULL path. > > - The prior-art table has a new column that names the proposed > > function for each row. > > > > One point is still open. IS_VARIANT_NULL now returns NULL for SQL > > NULL. Returning FALSE instead would keep its result NOT NULL. The > > wiki has an example under Open Questions. I'm happy to hear your > > opinion on it. > > > > Wiki: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-618%3A+Edit%2C+Inspect+and+Build+VARIANT+Values > > > > Cheers, > > Ramin Gharib > > > > On Thu, Oct 22, 2026 at 5:36 PM Ramin Gharib <[email protected]> > wrote: > > > >> Hi everyone, > >> > >> I'd like to start a discussion on FLIP-618: Edit, Inspect and Build > >> VARIANT Values. > >> > >> Wiki: > >> > https://cwiki.apache.org/confluence/spaces/FLINK/pages/461570450/FLIP-618+Edit+Inspect+and+Build+VARIANT+Values > >> > >> Flink can store and read VARIANT values since 2.1. SQL users still > >> cannot inspect or change one field and keep the rest, ask what a value > >> contains, > >> or build an object from columns without going through JSON text. > >> Today each of these needs a UDF that rebuilds the whole value. > >> > >> The FLIP proposes built-in functions to edit (VARIANT_SET, > >> VARIANT_DELETE, VARIANT_MERGE, ...), inspect (IS_VARIANT_NULL, > >> VARIANT_KEYS, SCHEMA_OF_VARIANT, ...) and build (VARIANT_OBJECT, > >> VARIANT_OBJECT_AGG) VARIANT values, plus a Java API to edit a value > >> inside a UDF. > >> > >> Looking forward to your feedback. > >> > >> Best, > >> Ramin Gharib > >> > > > >
