[ https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539635#comment-16539635 ]
Saisai Shao commented on SPARK-24781: ------------------------------------- I see. I will wait for this before cutting a new 2.3.2 RC release > Using a reference from Dataset in Filter/Sort might not work. > ------------------------------------------------------------- > > Key: SPARK-24781 > URL: https://issues.apache.org/jira/browse/SPARK-24781 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.1 > Reporter: Takuya Ueshin > Priority: Blocker > > When we use a reference fromĀ {{Dataset}} in {{filter}} or {{sort}}, which was > not used in the prior {{select}}, an {{AnalysisException}} occurs, e.g., > {code:scala} > val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id") > df.select(df("name")).filter(df("id") === 0).show() > {code} > {noformat} > org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing > from name#5 in operator !Filter (id#6 = 0).;; > !Filter (id#6 = 0) > +- AnalysisBarrier > +- Project [name#5] > +- Project [_1#2 AS name#5, _2#3 AS id#6] > +- LocalRelation [_1#2, _2#3] > {noformat} > If we use {{col}} instead, it works: > {code:scala} > val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id") > df.select(col("name")).filter(col("id") === 0).show() > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org