Hi Ramin, One point on the aggregates.
Neither VARIANT_OBJECT_AGG nor SCHEMA_OF_VARIANT_AGG says what happens on an updating input, and FLINK-40582 does not either. Flink has two patterns for this: retract directly, as JSON_OBJECTAGG does by removing the key, or a WithRetract variant that AggFunctionFactory picks when aggCallNeedRetractions is set, as SUM, MIN, LISTAGG and FIRST_VALUE do. VARIANT_OBJECT_AGG looks like the first case. A least common type merge is not invertible, so SCHEMA_OF_VARIANT_AGG would need the accumulator to keep more than the merged type, the way MinWithRetractAggFunction keeps a MapView of counts. Worth saying which of the two it is, or that an updating input is out of scope. Best, Sepuri Sai Krishna On Thu, Oct 29, 2026 at 12:55 PM Ramin Gharib <[email protected]> wrote: > Hi Martijn, > > Thanks for the review and the +1. > > Agreed on keeping NULL. I updated the FLIP: > > - The prior-art section now mentions Dremio below the table. Its > VARIANT functions only read values, so a column would be mostly > empty. > - The open question section now lists which engines return NULL for SQL > NULL: Dremio, Snowflake and our own IS JSON. Spark returns FALSE. > > cheers, > Ramin > > On Wed, Oct 28, 2026 at 11:30 AM Martijn Visser <[email protected]> > wrote: > > > Hi all, > > > > Thanks for the FLIP ! I'm +1 on this. Once users store data as VARIANT, > > changing one field or building an object from columns are among the first > > things they want to do, and writing a UDF for that is a big ask. I also > > like that SQL NULL is now handled the same way across all functions. > > > > On the open question, I would keep NULL. Dremio has an IS_VARIANT_NULL > that > > returns NULL for SQL NULL [1], and so do Snowflake's IS_NULL_VALUE and > our > > own IS JSON. Spark is the exception. With NULL, one call also tells a > > missing field, a variant null and a value apart. It might be worth > > adding Dremio to the prior-art table. > > > > Best regards, > > > > Martijn > > > > [1] > > > > > https://docs.dremio.com/dremio-cloud/sql/sql-functions/functions/IS_VARIANT_NULL > > > > Op vr 23 okt 2026 om 13:51 schreef Timo Walther <[email protected]>: > > > > > Hi everyone, > > > > > > thanks Ramin for updating the FLIP. It took us quite some iterations to > > > get it into a consistent and high quality shape. > > > > > > For context, these are items we put into consideration: > > > - Look at other vendors: BigQuery, Snowflake, Spark, DuckDB > > > - Look at existing Flink SQL built-in functions e.g. JSON_*, IS_* and > > > collection functions > > > - Look at use cases and find usability gaps > > > - Provide a reasonable set of building blocks > > > - What is required to make VARIANT in Flink a large success? > > > > > > The FLIP covers a lot of function at once. Some are not described in > > > every detail, but we will cover this as part of PR discussions and > > > writing docs. > > > > > > Overall, I hope we can make many users happy. > > > > > > Thanks, > > > Timo > > > > > > > > > On 23.10.26 13:32, Ramin Gharib wrote: > > > > Hi everyone, > > > > > > > > After a long discussion with Timo Walther, I updated the FLIP to > > > > change how the functions handle SQL NULL. > > > > > > > > The functions now handle SQL NULL and variant null consistently. A > > > > SQL NULL argument gives a SQL NULL result. > > > > > > > > To write a variant null on purpose, the FLIP adds two functions: > > > > > > > > - VARIANT_NULL() returns a variant null, the same as > > > > PARSE_JSON('null'). > > > > - TO_VARIANT(x) is the same as CAST(x AS VARIANT), except for SQL > > > > NULL. CAST(NULL AS VARIANT) returns NULL, so TO_VARIANT gives a > > > > variant null instead. > > > > > > > > For example, with discount = NULL: > > > > > > > > VARIANT_SET(payload, '$.discount', discount) > > > > -- NULL > > > > VARIANT_SET(payload, '$.discount', TO_VARIANT(discount)) > > > > -- {"discount":null,"order_id":8,"user":"bob"} > > > > > > > > Other changes: > > > > > > > > - VARIANT_OBJECT, VARIANT_ARRAY and VARIANT_MERGE return NULL for a > > > > SQL NULL argument. A NULL key in VARIANT_OBJECT is still an error. > > > > - VARIANT_OBJECT_AGG skips rows with a SQL NULL value. > > > > - VARIANT_DELETE and VARIANT_PICK still skip a NULL path. > > > > - The prior-art table has a new column that names the proposed > > > > function for each row. > > > > > > > > One point is still open. IS_VARIANT_NULL now returns NULL for SQL > > > > NULL. Returning FALSE instead would keep its result NOT NULL. The > > > > wiki has an example under Open Questions. I'm happy to hear your > > > > opinion on it. > > > > > > > > Wiki: > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-618%3A+Edit%2C+Inspect+and+Build+VARIANT+Values > > > > > > > > Cheers, > > > > Ramin Gharib > > > > > > > > On Thu, Oct 22, 2026 at 5:36 PM Ramin Gharib <[email protected]> > > > wrote: > > > > > > > >> Hi everyone, > > > >> > > > >> I'd like to start a discussion on FLIP-618: Edit, Inspect and Build > > > >> VARIANT Values. > > > >> > > > >> Wiki: > > > >> > > > > > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/461570450/FLIP-618+Edit+Inspect+and+Build+VARIANT+Values > > > >> > > > >> Flink can store and read VARIANT values since 2.1. SQL users still > > > >> cannot inspect or change one field and keep the rest, ask what a > value > > > >> contains, > > > >> or build an object from columns without going through JSON text. > > > >> Today each of these needs a UDF that rebuilds the whole value. > > > >> > > > >> The FLIP proposes built-in functions to edit (VARIANT_SET, > > > >> VARIANT_DELETE, VARIANT_MERGE, ...), inspect (IS_VARIANT_NULL, > > > >> VARIANT_KEYS, SCHEMA_OF_VARIANT, ...) and build (VARIANT_OBJECT, > > > >> VARIANT_OBJECT_AGG) VARIANT values, plus a Java API to edit a value > > > >> inside a UDF. > > > >> > > > >> Looking forward to your feedback. > > > >> > > > >> Best, > > > >> Ramin Gharib > > > >> > > > > > > > > > > > > >
