Hi Gustavo, Thanks for looking into this and the pointers. We have now included more details in the FLIP draft (link shared by Dominik in the previous e-mail: https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing ).
In short: - A UDF called from a lambda body — supported. - A UDF passed as the lambda, e.g. `ARRAY_TRANSFORM(a, my_udf)`, is not supported — this would need eta-expansion. The same call written as `ARRAY_TRANSFORM(a, x -> my_udf(x))` is supported. - Nesting: supported; arbitrary depth, any combination of built-in and user-defined, in both directions, with each level capturing the parameters of all enclosing lambdas plus outer columns. All three are covered by a new section in the FLIP draft called Public Interfaces / 6. Composition, which lists the supported compositions with an example for each. The mechanism behind it is in Proposed Changes / Lambda captures. Every one of those is covered by an IT case that executes the query and asserts the result. If you have other combinations in mind, we will add them. Best, Michal On Fri, Aug 7, 2026 at 4:15 PM Gustavo de Morais <[email protected]> wrote: > Hi Dominik, > > Thanks for the reply and for considering these. It's great that you're both > working on this. As mentioned, it's a very important feature, but there are > indeed some cases that might grow its scope - we should make sure we design > it in a complete way that fits the different parts of the project. > > I'll try to make some time next week to take a deeper look at what you > shared. In the meantime, here are a few more pointers you can already look > into: you mentioned lambda arguments for user-defined functions - what > about a UDF as an argument of a lambda function? Nesting is also worth > considering: multiple nested lambda functions, lambda function -> UDF or > regular function -> lambda function -> ... etc. > > Kind regards, > Gustavo > > On Fri, 7 Aug 2026 at 11:31, <[email protected]> wrote: > > > Hi Gustavo, > > > > Sorry for the late response. I had "a few" discussions with my colleague > > Michal Stutzmann (you might remember him from the Flink Forward > conference > > in Barcelona) about the FLIP and the features it should cover. Michal > > extended the proposal< > > > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing > > > > and implementation to also support the Table & Python APIs, as well as > > lambda arguments for user-defined functions. Additionally, we now support > > using columns within lambda expressions. > > > > We pushed the WIP changes to a branch in our fork, which you can find > here: > > > > > > > https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas > > > > The set of changes grew larger than expected, but in our opinion they > > significantly simplify the user experience by reducing the need to work > > with UDFs or custom code to handle collections. > > > > Thanks a lot in advance for taking a look! > > > > Best, > > Dominik > > > > From: Gustavo de Morais <[email protected]> > > Date: Friday, 31 July 2026 at 12:05 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > (TRANSFORM and ARRAY_FILTER) > > > > > > Be aware: This is an external email. > > > > > > > > Hey Dominik, > > > > Lambda functions are an important building block - not only for these > > built-in collection functions, but also for UDFs and PTFs whitin the > whole > > archictecture. So thanks for the proposal and for wanting to contribute. > > > > I'll take a look at the proposal soon. You mentioned a WIP branch a > couple > > of times in the thread - could you share the link so we can take a look > at > > that too? > > > > Kind regards, > > Gustavo > > > > On Thu, 30 Jul 2026 at 16:39, <[email protected]> wrote: > > > > > Hi all, > > > > > > Following up on this FLIP to see if there are any comments or concerns. > > > > > > Since the last mail I added tests for: > > > - views using ARRAY_FILTER / TRANSFORM > > > - materialized tables going through expand-and-replan > > > > > > Everything is green locally so far. > > > > > > Feedback on the planner integration, Calcite handling, missing edge > > cases, > > > or the overall direction would be appreciated. > > > > > > As I’m still new to the FLIP process, I’d also appreciate some guidance > > on > > > the next steps from here. Many thanks in advance. > > > > > > Best, > > > Dominik > > > > > > From: Bünzli Dominik, SCS-INI-DNA-INF <[email protected]> > > > Date: Friday, 24 July 2026 at 11:17 > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > > (TRANSFORM and ARRAY_FILTER) > > > > > > Hi Sergey, > > > > > > Good catch. I didn’t have these cases in mind. Quickly added test cases > > > for them to my branch. > > > > > > The view test creates a view like "CREATE VIEW v AS SELECT id, > > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and > checks > > > that the arrays come back right. Same idea for TRANSFORM and the map > > lambda > > > (k, v) -> v * 100. > > > > > > The materialized table test does the equivalent: a CONTINUOUS table > using > > > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, > then > > a > > > SELECT to confirm the output matches. > > > > > > So, both should (to my still limited knowledge), go through the > > > expand-and-replan path and check real results. If Calcite mangles a > > lambda, > > > these tests should fail. Looks good so far, tests are green locally. > > > > > > Best, > > > Dominik > > > > > > From: Sergey Nuyanzin <[email protected]> > > > Date: Friday, 24 July 2026 at 09:59 > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > Be aware: This is an external email. > > > > > > > > > > > > thanks for the proposal > > > > > > and working in this direction > > > > > > I haven't checked yet all the things > > > > > > however I see at least one missing piece here. > > > > > > In FlinkSQL we have views and materialized tables. Both are relying on > > > expanded ("rewritten" in Calcite terms) SQL. > > > And we have already faced a number of issues around that and fixed > > > them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6], > > > [7], [8], [9], [10]) > > > > > > I'm pretty sure there will be something lambda related. For that > > > reason I would suggest also having test with views(or materialized > > > tables, does not matter) based on sql with lambda > > > and check that SELECT from them still able to produce output > > > > > > [1] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7660> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7660>> > > > [2] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7575> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7575>> > > > [3] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7465> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7465>> > > > [4] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7480> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7480>> > > > [5] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7471> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7471>> > > > [6] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7470> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7470>> > > > [7] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7466> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7466>> > > > [8] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7312> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7312>> > > > [9] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7217> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-7217>> > > > [10] > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-6944> > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0 > > <https://issues.apache.org/jira/browse/CALCITE-6944>> > > > > > > On Fri, Jul 24, 2026 at 9:37 AM <[email protected]> wrote: > > > > > > > > Hi Gyula > > > > > > > > Thanks a lot for your reply and the +1! > > > > > > > > On a conceptual level, lambdas and scalar functions / UDFs are > > distinct. > > > A lambda isn't a stored/registered value. It's represented by the > > > planning-only FUNCTION logical type and only exists as an argument to > > > higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially, > it's > > > the wrapper that introduces the element variable x. > > > > > > > > Within the lambda you can freely use any scalar function or UDF: > > > > ARRAY_FILTER(arr, x -> is_positive(x)) > > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100) > > > > > > > > What currently doesn't work is passing a bare UDF name in the lambda > > > position (i.e. automatic expansion of is_positive into x -> > > is_positive(x)): > > > > ARRAY_FILTER(arr, is_positive) -- not supported > > > > > > > > We intentionally left this out of scope for now, but it's a > > > straightforward follow-up if we see demand for it. > > > > > > > > Best, > > > > Dominik > > > > > > > > From: Gyula Fóra <[email protected]> > > > > Date: Thursday, 23 July 2026 at 16:07 > > > > To: [email protected] <[email protected]> > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > > > > Be aware: This is an external email. > > > > > > > > > > > > > > > > Hey! > > > > > > > > Without a good background knowledge about the syntax in other > systems, > > > the > > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially with > > > > lambda functions. > > > > Overall +1 for this idea from a very high level but it would be great > > > with > > > > someone with more deep context on SQL functions / runtime to chime > in. > > > > > > > > One question I had is how do lambda functions relate to existing > scalar > > > > functions / UDFs? Would the user be able to use boolean valued scalar > > > > functions where lambdas are shown in the proposal? > > > > > > > > Cheers > > > > Gyula > > > > > > > > On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> > wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > I'd like to start a discussion on a FLIP that introduces > higher-order > > > > > functions > > > > > (functions that take a lambda expression as an argument) to Flink > > SQL, > > > > > together > > > > > with the first two built-ins that use them: TRANSFORM and > > ARRAY_FILTER. > > > > > > > > > > FLIP: > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0 > > < > > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0 > > < > > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > >< > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0 > > < > > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > > > > > > > > > > > > Motivation > > > > > > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake, > > > DuckDB, and > > > > > Presto/Trino expect to manipulate collections (arrays and maps) > > inline > > > > > with a > > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. > The > > > > > absence of > > > > > higher-order collection functions forces verbose query rewrites > > during > > > > > migration > > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and > > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry > > points > > > > > and, > > > > > importantly, they can be built on lambda infrastructure that > already > > > > > exists in > > > > > Calcite (CALCITE-3679), so the surface area we add on the Flink > side > > is > > > > > rather > > > > > small. > > > > > > > > > > Summary > > > > > > > > > > • Introduce a new logical type FUNCTION that describes the type > of > > a > > > > > lambda > > > > > (its argument types and its result type). This is the > type-system > > > > > foundation > > > > > every higher-order function needs; it is a planning/translation > > > helper > > > > > type > > > > > and is not a persisted column type. > > > > > • Add ARRAY_FILTER(array, element -> predicate), which returns a > > new > > > > > array > > > > > containing only the elements for which the predicate holds. > > > > > • Add TRANSFORM(collection, lambda), which applies a lambda to > > every > > > > > element of > > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a map > > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map. > > > > > • Reuse Calcite's lambda parsing, validation and > > > > > RexLambda/FunctionSqlType > > > > > infrastructure (CALCITE-3679) rather than inventing a > > > Flink-specific > > > > > lambda > > > > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is > > > already > > > > > parseable by the Calcite version Flink bundles (1.41.0). > > > > > > > > > > The functions are net-new syntax and are additive: no existing > query > > > > > changes > > > > > behavior. There is no new configuration option — the functions are > > > always > > > > > available once the release ships. > > > > > > > > > > Examples: > > > > > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); > > -- > > > > > [3, 4] > > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); > > -- > > > > > [10, 20, 30] > > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s)); > > -- > > > > > [1, 2, 3] > > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); > > -- > > > > > {a=100, b=200} > > > > > > > > > > Thanks, > > > > > Dominik Bünzli > > > > > Data, Analytics & AI Engineer > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Sergey > > > > > >
