[
https://issues.apache.org/jira/browse/SPARK-20122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944799#comment-15944799
]
Herman van Hovell commented on SPARK-20122:
-------------------------------------------
Yeah, I totally missed that. Anyway, you need to add a rule to
{{CheckAnalysis}} for this.
> Analyzer reports "unresolved operator 'Aggregate" for nonexistent window
> specifications
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-20122
> URL: https://issues.apache.org/jira/browse/SPARK-20122
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Jacek Laskowski
> Priority: Minor
>
> {{Analyzer}} should be more elaborate on the root cause of the
> {{org.apache.spark.sql.AnalysisException}} when {{WindowSpecReference}}
> cannot be resolved.
> I think the real issue is that {{AstBuilder}} creates a lookup table for
> {{WindowSpecDefinition}}s per name and when it cannot resolve
> {{WindowSpecReference}} it simply fails analysis (it's too early in the SQL
> parsing).
> {code}
> scala> val table = spark.emptyDataset[Int]
> table: org.apache.spark.sql.Dataset[Int] = [value: int]
> scala> table.selectExpr("count() OVER nonexistentWindowSpec")
> org.apache.spark.sql.AnalysisException: unresolved operator 'Aggregate
> [unresolvedwindowexpression(count(),
> WindowSpecReference(nonexistentWindowSpec)) AS
> unresolvedwindowexpression(count())#15];;
> 'Aggregate [unresolvedwindowexpression(count(),
> WindowSpecReference(nonexistentWindowSpec)) AS
> unresolvedwindowexpression(count())#15]
> +- LocalRelation <empty>, [value#11]
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:40)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:90)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$3.apply(CheckAnalysis.scala:408)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$3.apply(CheckAnalysis.scala:407)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:407)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:90)
> at
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:53)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:67)
> at
> org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2832)
> at org.apache.spark.sql.Dataset.select(Dataset.scala:1137)
> at org.apache.spark.sql.Dataset.selectExpr(Dataset.scala:1172)
> ... 48 elided
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
