[jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

2018-07-13 Thread Xiao Li (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543342#comment-16543342
 ] 

Xiao Li commented on SPARK-24781:
-

cc [~jerryshao] The fix is merged. Please start the next rc of Spark 2.3.2?

> Using a reference from Dataset in Filter/Sort might not work.
> -
>
> Key: SPARK-24781
> URL: https://issues.apache.org/jira/browse/SPARK-24781
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Takuya Ueshin
>Assignee: Liang-Chi Hsieh
>Priority: Blocker
> Fix For: 2.3.2, 2.4.0
>
>
> When we use a reference from {{Dataset}} in {{filter}} or {{sort}}, which was 
> not used in the prior {{select}}, an {{AnalysisException}} occurs, e.g.,
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(df("name")).filter(df("id") === 0).show()
> {code}
> {noformat}
> org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing 
> from name#5 in operator !Filter (id#6 = 0).;;
> !Filter (id#6 = 0)
>+- AnalysisBarrier
>   +- Project [name#5]
>  +- Project [_1#2 AS name#5, _2#3 AS id#6]
> +- LocalRelation [_1#2, _2#3]
> {noformat}
> If we use {{col}} instead, it works:
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(col("name")).filter(col("id") === 0).show()
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

2018-07-11 Thread Apache Spark (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539757#comment-16539757
 ] 

Apache Spark commented on SPARK-24781:
--

User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/21745

> Using a reference from Dataset in Filter/Sort might not work.
> -
>
> Key: SPARK-24781
> URL: https://issues.apache.org/jira/browse/SPARK-24781
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Takuya Ueshin
>Priority: Blocker
>
> When we use a reference from {{Dataset}} in {{filter}} or {{sort}}, which was 
> not used in the prior {{select}}, an {{AnalysisException}} occurs, e.g.,
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(df("name")).filter(df("id") === 0).show()
> {code}
> {noformat}
> org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing 
> from name#5 in operator !Filter (id#6 = 0).;;
> !Filter (id#6 = 0)
>+- AnalysisBarrier
>   +- Project [name#5]
>  +- Project [_1#2 AS name#5, _2#3 AS id#6]
> +- LocalRelation [_1#2, _2#3]
> {noformat}
> If we use {{col}} instead, it works:
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(col("name")).filter(col("id") === 0).show()
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

2018-07-11 Thread Saisai Shao (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539635#comment-16539635
 ] 

Saisai Shao commented on SPARK-24781:
-

I see. I will wait for this before cutting a new 2.3.2 RC release

> Using a reference from Dataset in Filter/Sort might not work.
> -
>
> Key: SPARK-24781
> URL: https://issues.apache.org/jira/browse/SPARK-24781
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Takuya Ueshin
>Priority: Blocker
>
> When we use a reference from {{Dataset}} in {{filter}} or {{sort}}, which was 
> not used in the prior {{select}}, an {{AnalysisException}} occurs, e.g.,
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(df("name")).filter(df("id") === 0).show()
> {code}
> {noformat}
> org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing 
> from name#5 in operator !Filter (id#6 = 0).;;
> !Filter (id#6 = 0)
>+- AnalysisBarrier
>   +- Project [name#5]
>  +- Project [_1#2 AS name#5, _2#3 AS id#6]
> +- LocalRelation [_1#2, _2#3]
> {noformat}
> If we use {{col}} instead, it works:
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(col("name")).filter(col("id") === 0).show()
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

2018-07-11 Thread Takuya Ueshin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539633#comment-16539633
 ] 

Takuya Ueshin commented on SPARK-24781:
---

Yes, I think so. This is a regression from 2.2 to 2.3.

> Using a reference from Dataset in Filter/Sort might not work.
> -
>
> Key: SPARK-24781
> URL: https://issues.apache.org/jira/browse/SPARK-24781
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Takuya Ueshin
>Priority: Blocker
>
> When we use a reference from {{Dataset}} in {{filter}} or {{sort}}, which was 
> not used in the prior {{select}}, an {{AnalysisException}} occurs, e.g.,
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(df("name")).filter(df("id") === 0).show()
> {code}
> {noformat}
> org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing 
> from name#5 in operator !Filter (id#6 = 0).;;
> !Filter (id#6 = 0)
>+- AnalysisBarrier
>   +- Project [name#5]
>  +- Project [_1#2 AS name#5, _2#3 AS id#6]
> +- LocalRelation [_1#2, _2#3]
> {noformat}
> If we use {{col}} instead, it works:
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(col("name")).filter(col("id") === 0).show()
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

2018-07-11 Thread Saisai Shao (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539630#comment-16539630
 ] 

Saisai Shao commented on SPARK-24781:
-

Thanks Felix. Does this have to be in 2.3.2? [~ueshin]

> Using a reference from Dataset in Filter/Sort might not work.
> -
>
> Key: SPARK-24781
> URL: https://issues.apache.org/jira/browse/SPARK-24781
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Takuya Ueshin
>Priority: Blocker
>
> When we use a reference from {{Dataset}} in {{filter}} or {{sort}}, which was 
> not used in the prior {{select}}, an {{AnalysisException}} occurs, e.g.,
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(df("name")).filter(df("id") === 0).show()
> {code}
> {noformat}
> org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing 
> from name#5 in operator !Filter (id#6 = 0).;;
> !Filter (id#6 = 0)
>+- AnalysisBarrier
>   +- Project [name#5]
>  +- Project [_1#2 AS name#5, _2#3 AS id#6]
> +- LocalRelation [_1#2, _2#3]
> {noformat}
> If we use {{col}} instead, it works:
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(col("name")).filter(col("id") === 0).show()
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

2018-07-11 Thread Felix Cheung (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539627#comment-16539627
 ] 

Felix Cheung commented on SPARK-24781:
--

[~jerryshao]

> Using a reference from Dataset in Filter/Sort might not work.
> -
>
> Key: SPARK-24781
> URL: https://issues.apache.org/jira/browse/SPARK-24781
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Takuya Ueshin
>Priority: Blocker
>
> When we use a reference from {{Dataset}} in {{filter}} or {{sort}}, which was 
> not used in the prior {{select}}, an {{AnalysisException}} occurs, e.g.,
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(df("name")).filter(df("id") === 0).show()
> {code}
> {noformat}
> org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing 
> from name#5 in operator !Filter (id#6 = 0).;;
> !Filter (id#6 = 0)
>+- AnalysisBarrier
>   +- Project [name#5]
>  +- Project [_1#2 AS name#5, _2#3 AS id#6]
> +- LocalRelation [_1#2, _2#3]
> {noformat}
> If we use {{col}} instead, it works:
> {code:scala}
> val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
> df.select(col("name")).filter(col("id") === 0).show()
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org