Hi all, Following up on this FLIP to see if there are any comments or concerns.
Since the last mail I added tests for: - views using ARRAY_FILTER / TRANSFORM - materialized tables going through expand-and-replan Everything is green locally so far. Feedback on the planner integration, Calcite handling, missing edge cases, or the overall direction would be appreciated. As I’m still new to the FLIP process, I’d also appreciate some guidance on the next steps from here. Many thanks in advance. Best, Dominik From: Bünzli Dominik, SCS-INI-DNA-INF <[email protected]> Date: Friday, 24 July 2026 at 11:17 To: [email protected] <[email protected]> Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER) Hi Sergey, Good catch. I didn’t have these cases in mind. Quickly added test cases for them to my branch. The view test creates a view like "CREATE VIEW v AS SELECT id, ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and checks that the arrays come back right. Same idea for TRANSFORM and the map lambda (k, v) -> v * 100. The materialized table test does the equivalent: a CONTINUOUS table using ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, then a SELECT to confirm the output matches. So, both should (to my still limited knowledge), go through the expand-and-replan path and check real results. If Calcite mangles a lambda, these tests should fail. Looks good so far, tests are green locally. Best, Dominik From: Sergey Nuyanzin <[email protected]> Date: Friday, 24 July 2026 at 09:59 To: [email protected] <[email protected]> Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER) Be aware: This is an external email. thanks for the proposal and working in this direction I haven't checked yet all the things however I see at least one missing piece here. In FlinkSQL we have views and materialized tables. Both are relying on expanded ("rewritten" in Calcite terms) SQL. And we have already faced a number of issues around that and fixed them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6], [7], [8], [9], [10]) I'm pretty sure there will be something lambda related. For that reason I would suggest also having test with views(or materialized tables, does not matter) based on sql with lambda and check that SELECT from them still able to produce output [1] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519482673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=wy%2FmMDx7Bc%2Blkbjdyr1rWiEpGyOFweCNjHKHMhEGDYo%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660> [2] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519515485%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=jrSfxN8AhWe%2FY%2BeRtU7IWN9A5SIZrluz9a0jPCbrauE%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575> [3] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519540885%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2XqM%2FzWVAUM9M9v2BvhF2rG1Gxo2y12hPNJVV2OMbtU%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465> [4] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519565082%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0MUvjzrXcdaphlgigoN0j6xW5m5JiGYJBCe0pXpehiE%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480> [5] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519590890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=53w4cH9008310XzR343RfJKSpD%2BuiPJXL5sB4GivfMg%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471> [6] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519622955%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2FeYvSOwEryd6cE9JLIU%2Fyopj20WJzeu%2Fv8h2%2FSeGpUM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470> [7] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519653261%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PS6dyyL3YaK6Sefib5icMJZweVcQ3192NZI1owUwSIM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466> [8] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519678467%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=56Zea5qfcDszsps%2B4VOdVCvn2UeVCVx2fOYaUWlX%2FDw%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312> [9] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519700401%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=KJka%2Fs%2F8YPH1b%2F16FCwTgFtsb8vFyK9oY3DGeT%2BUzxU%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7217> [10] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519722120%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2B1HgC5SAvuBelfegMB0CLTCYdwcb6NxCBcTzdjC0Qlw%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944> On Fri, Jul 24, 2026 at 9:37 AM <[email protected]> wrote: > > Hi Gyula > > Thanks a lot for your reply and the +1! > > On a conceptual level, lambdas and scalar functions / UDFs are distinct. A > lambda isn't a stored/registered value. It's represented by the planning-only > FUNCTION logical type and only exists as an argument to higher-order > functions like ARRAY_FILTER and TRANSFORM. Essentially, it's the wrapper that > introduces the element variable x. > > Within the lambda you can freely use any scalar function or UDF: > ARRAY_FILTER(arr, x -> is_positive(x)) > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100) > > What currently doesn't work is passing a bare UDF name in the lambda position > (i.e. automatic expansion of is_positive into x -> is_positive(x)): > ARRAY_FILTER(arr, is_positive) -- not supported > > We intentionally left this out of scope for now, but it's a straightforward > follow-up if we see demand for it. > > Best, > Dominik > > From: Gyula Fóra <[email protected]> > Date: Thursday, 23 July 2026 at 16:07 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > (TRANSFORM and ARRAY_FILTER) > > > Be aware: This is an external email. > > > > Hey! > > Without a good background knowledge about the syntax in other systems, the > ARRAY_FILTER, TRANSFORM functions seem really useful, especially with > lambda functions. > Overall +1 for this idea from a very high level but it would be great with > someone with more deep context on SQL functions / runtime to chime in. > > One question I had is how do lambda functions relate to existing scalar > functions / UDFs? Would the user be able to use boolean valued scalar > functions where lambdas are shown in the proposal? > > Cheers > Gyula > > On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> wrote: > > > Hi everyone, > > > > I'd like to start a discussion on a FLIP that introduces higher-order > > functions > > (functions that take a lambda expression as an argument) to Flink SQL, > > together > > with the first two built-ins that use them: TRANSFORM and ARRAY_FILTER. > > > > FLIP: > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519743111%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MZQgSpOw4rNqYRG3Tm0mUigSQGXQi4dDreRhyV%2F4wZY%3D&reserved=0<https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519772568%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XNwMzQrplv6Pt7nGCO%2F0Jnk2YFD9v%2FJtP7b7iiyeGGQ%3D&reserved=0><https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing> > > > > Motivation > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake, DuckDB, and > > Presto/Trino expect to manipulate collections (arrays and maps) inline > > with a > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. The > > absence of > > higher-order collection functions forces verbose query rewrites during > > migration > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry points > > and, > > importantly, they can be built on lambda infrastructure that already > > exists in > > Calcite (CALCITE-3679), so the surface area we add on the Flink side is > > rather > > small. > > > > Summary > > > > • Introduce a new logical type FUNCTION that describes the type of a > > lambda > > (its argument types and its result type). This is the type-system > > foundation > > every higher-order function needs; it is a planning/translation helper > > type > > and is not a persisted column type. > > • Add ARRAY_FILTER(array, element -> predicate), which returns a new > > array > > containing only the elements for which the predicate holds. > > • Add TRANSFORM(collection, lambda), which applies a lambda to every > > element of > > an array (TRANSFORM(array, x -> expr)) or every entry of a map > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map. > > • Reuse Calcite's lambda parsing, validation and > > RexLambda/FunctionSqlType > > infrastructure (CALCITE-3679) rather than inventing a Flink-specific > > lambda > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is already > > parseable by the Calcite version Flink bundles (1.41.0). > > > > The functions are net-new syntax and are additive: no existing query > > changes > > behavior. There is no new configuration option — the functions are always > > available once the release ships. > > > > Examples: > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); -- > > [3, 4] > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); -- > > [10, 20, 30] > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s)); -- > > [1, 2, 3] > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); -- > > {a=100, b=200} > > > > Thanks, > > Dominik Bünzli > > Data, Analytics & AI Engineer > > -- Best regards, Sergey
