[
https://issues.apache.org/jira/browse/SPARK-35580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-35580:
------------------------------------
Assignee: Apache Spark
> Support subexpression elimination for higher order functions
> ------------------------------------------------------------
>
> Key: SPARK-35580
> URL: https://issues.apache.org/jira/browse/SPARK-35580
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.1
> Reporter: Adam Binford
> Assignee: Apache Spark
> Priority: Major
>
> Currently higher order functions are not candidates for subexpression
> elimination. This is because all higher order functions have different
> semantic hashes, due to "exprId" and "value" in "NamedLambdaVariable". These
> always are unique, so the semanticHash of a NamedLambdaVariable is always
> unique. Also, [https://github.com/apache/spark/pull/32424] might throw a
> wrench in things some too, depending on how you define your expressions the
> name could be different.
> {code:java}
> scala> var d = transform($"a", x => x + 1)
> d: org.apache.spark.sql.Column = transform(a, lambdafunction((x_2 + 1), x_2))
> scala> var e = transform($"a", x => x + 1)
> e: org.apache.spark.sql.Column = transform(a, lambdafunction((x_3 + 1), x_3))
> scala> struct(d.alias("1"), d.alias("2")).expr
> res9: org.apache.spark.sql.catalyst.expressions.Expression =
> struct(NamePlaceholder, transform('a, lambdafunction((lambda 'x_2 + 1),
> lambda 'x_2, false)) AS 1#4, NamePlaceholder, transform('a,
> lambdafunction((lambda 'x_2 + 1), lambda 'x_2, false)) AS 2#5)
> scala> struct(d.alias("1"), e.alias("2")).expr
> res10: org.apache.spark.sql.catalyst.expressions.Expression =
> struct(NamePlaceholder, transform('a, lambdafunction((lambda 'x_2 + 1),
> lambda 'x_2, false)) AS 1#6, NamePlaceholder, transform('a,
> lambdafunction((lambda 'x_3 + 1), lambda 'x_3, false)) AS 2#7)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
