[
https://issues.apache.org/jira/browse/SPARK-39091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-39091:
------------------------------------
Assignee: (was: Apache Spark)
> SQL Expression traits don't compose due to nodePatterns being final.
> --------------------------------------------------------------------
>
> Key: SPARK-39091
> URL: https://issues.apache.org/jira/browse/SPARK-39091
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.2.0, 3.2.1
> Reporter: Huw
> Priority: Major
>
> In Spark 3.1 I have an expression which contains these parts:
> {code:scala}
> case class MyExploder(
> arrays: Expression, // Array[AnyDataType]
> asOfDate: Expression, // LambdaFunction[AnyDataType -> TimestampType]
> extractor: Expression, // TimestampType
> ) extends HigherOrderFunction with Generator with TimeZoneAwareExpression {
> override def arguments: Seq[Expression] =
> Seq(arrays, asOfDate)
> override def argumentTypes: Seq[AbstractDataType] =
> Seq(ArrayType, TimestampType)
> override def functions: Seq[Expression] =
> Seq(extractor)
> override def functionTypes =
> Seq(TimestampType)
> }{code}
>
> This is grossly simplified example. The extractor is a lambda which can
> gather information from a nested array, and explodes based on some business
> logic.
> When upgrading to Spark 3.2 however this can't work anymore, because they
> have conflicting final values for nodePatterns.
> {code:java}
> trait HigherOrderFunction extends Expression with ExpectsInputTypes {
> final override val nodePatterns: Seq[TreePattern] = Seq(HIGH_ORDER_FUNCTION)
> } {code}
>
> We get this errror.
> {noformat}
> value nodePatterns in trait TimeZoneAwareExpression of type
> Seq[org.apache.spark.sql.catalyst.trees.TreePattern.TreePattern] cannot
> override final member{noformat}
>
> This blocks us from upgrading. What's doubly annoying is that the actual
> value of the member appears to be the same.
>
> Thank you for your time.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
