Hi Martijn, Thanks for the review and the +1.
Agreed on keeping NULL. I updated the FLIP: - The prior-art section now mentions Dremio below the table. Its VARIANT functions only read values, so a column would be mostly empty. - The open question section now lists which engines return NULL for SQL NULL: Dremio, Snowflake and our own IS JSON. Spark returns FALSE. cheers, Ramin On Wed, Oct 28, 2026 at 11:30 AM Martijn Visser <[email protected]> wrote: > Hi all, > > Thanks for the FLIP ! I'm +1 on this. Once users store data as VARIANT, > changing one field or building an object from columns are among the first > things they want to do, and writing a UDF for that is a big ask. I also > like that SQL NULL is now handled the same way across all functions. > > On the open question, I would keep NULL. Dremio has an IS_VARIANT_NULL that > returns NULL for SQL NULL [1], and so do Snowflake's IS_NULL_VALUE and our > own IS JSON. Spark is the exception. With NULL, one call also tells a > missing field, a variant null and a value apart. It might be worth > adding Dremio to the prior-art table. > > Best regards, > > Martijn > > [1] > > https://docs.dremio.com/dremio-cloud/sql/sql-functions/functions/IS_VARIANT_NULL > > Op vr 23 okt 2026 om 13:51 schreef Timo Walther <[email protected]>: > > > Hi everyone, > > > > thanks Ramin for updating the FLIP. It took us quite some iterations to > > get it into a consistent and high quality shape. > > > > For context, these are items we put into consideration: > > - Look at other vendors: BigQuery, Snowflake, Spark, DuckDB > > - Look at existing Flink SQL built-in functions e.g. JSON_*, IS_* and > > collection functions > > - Look at use cases and find usability gaps > > - Provide a reasonable set of building blocks > > - What is required to make VARIANT in Flink a large success? > > > > The FLIP covers a lot of function at once. Some are not described in > > every detail, but we will cover this as part of PR discussions and > > writing docs. > > > > Overall, I hope we can make many users happy. > > > > Thanks, > > Timo > > > > > > On 23.10.26 13:32, Ramin Gharib wrote: > > > Hi everyone, > > > > > > After a long discussion with Timo Walther, I updated the FLIP to > > > change how the functions handle SQL NULL. > > > > > > The functions now handle SQL NULL and variant null consistently. A > > > SQL NULL argument gives a SQL NULL result. > > > > > > To write a variant null on purpose, the FLIP adds two functions: > > > > > > - VARIANT_NULL() returns a variant null, the same as > > > PARSE_JSON('null'). > > > - TO_VARIANT(x) is the same as CAST(x AS VARIANT), except for SQL > > > NULL. CAST(NULL AS VARIANT) returns NULL, so TO_VARIANT gives a > > > variant null instead. > > > > > > For example, with discount = NULL: > > > > > > VARIANT_SET(payload, '$.discount', discount) > > > -- NULL > > > VARIANT_SET(payload, '$.discount', TO_VARIANT(discount)) > > > -- {"discount":null,"order_id":8,"user":"bob"} > > > > > > Other changes: > > > > > > - VARIANT_OBJECT, VARIANT_ARRAY and VARIANT_MERGE return NULL for a > > > SQL NULL argument. A NULL key in VARIANT_OBJECT is still an error. > > > - VARIANT_OBJECT_AGG skips rows with a SQL NULL value. > > > - VARIANT_DELETE and VARIANT_PICK still skip a NULL path. > > > - The prior-art table has a new column that names the proposed > > > function for each row. > > > > > > One point is still open. IS_VARIANT_NULL now returns NULL for SQL > > > NULL. Returning FALSE instead would keep its result NOT NULL. The > > > wiki has an example under Open Questions. I'm happy to hear your > > > opinion on it. > > > > > > Wiki: > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-618%3A+Edit%2C+Inspect+and+Build+VARIANT+Values > > > > > > Cheers, > > > Ramin Gharib > > > > > > On Thu, Oct 22, 2026 at 5:36 PM Ramin Gharib <[email protected]> > > wrote: > > > > > >> Hi everyone, > > >> > > >> I'd like to start a discussion on FLIP-618: Edit, Inspect and Build > > >> VARIANT Values. > > >> > > >> Wiki: > > >> > > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/461570450/FLIP-618+Edit+Inspect+and+Build+VARIANT+Values > > >> > > >> Flink can store and read VARIANT values since 2.1. SQL users still > > >> cannot inspect or change one field and keep the rest, ask what a value > > >> contains, > > >> or build an object from columns without going through JSON text. > > >> Today each of these needs a UDF that rebuilds the whole value. > > >> > > >> The FLIP proposes built-in functions to edit (VARIANT_SET, > > >> VARIANT_DELETE, VARIANT_MERGE, ...), inspect (IS_VARIANT_NULL, > > >> VARIANT_KEYS, SCHEMA_OF_VARIANT, ...) and build (VARIANT_OBJECT, > > >> VARIANT_OBJECT_AGG) VARIANT values, plus a Java API to edit a value > > >> inside a UDF. > > >> > > >> Looking forward to your feedback. > > >> > > >> Best, > > >> Ramin Gharib > > >> > > > > > > > >
