Re: Will higher order functions in spark SQL be pushed upstream?

Hi Herman, That’s great and thanks for quick reply. The JIRA has an example of transform and refers to a presto doc with lots of functions. Do you know which functions will be supported. I am interested in using filter for example. Cheers Mick > On 19 Apr 2018, at 10:46, Herman van Hövell

Re: Will higher order functions in spark SQL be pushed upstream?

Hi, Regarding higher order functions > Yes, we intend to contribute this to open source. It doesn't look like this is in 2.3.0, at least I can't find it. Do you know when it might reach open source. Thanks Mick -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Will higher order functions in spark SQL be pushed upstream?

Hello, At Netflix's algorithm team, we work on ranking problems a lot where we naturally deal with the dataset with nested list of the structs. We built Scala APIs like map, filter, drop, withColumn that can work on the nested list of structs efficiently using SQL expression with codegen. Here

Re: Will higher order functions in spark SQL be pushed upstream?

Good news :) Thx Sameer. On Friday, June 9, 2017, Sameer Agarwal wrote: > * As a heavy user of complex data types I was wondering if there was >> any plan to push those changes upstream? >> > > Yes, we intend to contribute this to open source. > > >> * In addition, I was

Re: Will higher order functions in spark SQL be pushed upstream?

> > * As a heavy user of complex data types I was wondering if there was > any plan to push those changes upstream? > Yes, we intend to contribute this to open source. > * In addition, I was wondering if as part of this change it also tries > to solve the column pruning / filter pushdown issues

Re: Will higher order functions in spark SQL be pushed upstream?

+1 for the question 2017-06-07 19:50 GMT+02:00 Antoine HOM : > Hey guys, > > Databricks released higher order functions as part of their runtime > 3.0 beta (https://databricks.com/blog/2017/05/24/working-with- >

Will higher order functions in spark SQL be pushed upstream?

Hey guys, Databricks released higher order functions as part of their runtime 3.0 beta (https://databricks.com/blog/2017/05/24/working-with-nested-data-using-higher-order-functions-in-sql-on-databricks.html), which helps working with array within SQL statements. * As a heavy user of complex