[jira] [Commented] (SPARK-34003) Rule conflicts between PaddingAndLengthCheckForCharVarchar and ResolveAggregateFunctions
[
https://issues.apache.org/jira/browse/SPARK-34003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262555#comment-17262555
]
Apache Spark commented on SPARK-34003:
--
User 'yaooqinn' has created a pull request for this issue:
https://github.com/apache/spark/pull/31129
> Rule conflicts between PaddingAndLengthCheckForCharVarchar and
> ResolveAggregateFunctions
>
>
> Key: SPARK-34003
> URL: https://issues.apache.org/jira/browse/SPARK-34003
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Kent Yao
>Assignee: Kent Yao
>Priority: Critical
> Fix For: 3.1.1
>
>
> ResolveAggregateFunctions is a hacky rule and it calls `executeSameContext`
> to generate a `resolved agg` to determine which unresolved sort attribute
> should be pushed into the agg. However, after we add the
> PaddingAndLengthCheckForCharVarchar rule which will rewrite the query output,
> thus, the `resolved agg` cannot match original attributes anymore.
> It causes some dissociative sort attribute to be pushed in and fails the query
> {code:java}
> [info] Failed to analyze query: org.apache.spark.sql.AnalysisException:
> expression 'testcat.t1.`v`' is neither present in the group by, nor is it an
> aggregate function. Add to group by or wrap in first() (or first_value) if
> you don't care which value you get.;
> [info] Project [v#14, sum(i)#11L]
> [info] +- Sort [aggOrder#12 ASC NULLS FIRST], true
> [info] +- !Aggregate [v#14], [v#14, sum(cast(i#7 as bigint)) AS
> sum(i)#11L, v#13 AS aggOrder#12]
> [info] +- SubqueryAlias testcat.t1
> [info]+- Project [if ((length(v#6) <= 3)) v#6 else if
> ((length(rtrim(v#6, None)) > 3)) cast(raise_error(concat(input string of
> length , cast(length(v#6) as string), exceeds varchar type length
> limitation: 3)) as string) else rpad(rtrim(v#6, None), 3, ) AS v#14, i#7]
> [info] +- RelationV2[v#6, i#7, index#15, _partition#16]
> testcat.t1
> [info]
> [info] Project [v#14, sum(i)#11L]
> [info] +- Sort [aggOrder#12 ASC NULLS FIRST], true
> [info] +- !Aggregate [v#14], [v#14, sum(cast(i#7 as bigint)) AS
> sum(i)#11L, v#13 AS aggOrder#12]
> [info] +- SubqueryAlias testcat.t1
> [info]+- Project [if ((length(v#6) <= 3)) v#6 else if
> ((length(rtrim(v#6, None)) > 3)) cast(raise_error(concat(input string of
> length , cast(length(v#6) as string), exceeds varchar type length
> limitation: 3)) as string) else rpad(rtrim(v#6, None), 3, ) AS v#14, i#7]
> [info] +- RelationV2[v#6, i#7, index#15, _partition#16]
> testcat.t1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34003) Rule conflicts between PaddingAndLengthCheckForCharVarchar and ResolveAggregateFunctions
[
https://issues.apache.org/jira/browse/SPARK-34003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262556#comment-17262556
]
Apache Spark commented on SPARK-34003:
--
User 'yaooqinn' has created a pull request for this issue:
https://github.com/apache/spark/pull/31129
> Rule conflicts between PaddingAndLengthCheckForCharVarchar and
> ResolveAggregateFunctions
>
>
> Key: SPARK-34003
> URL: https://issues.apache.org/jira/browse/SPARK-34003
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Kent Yao
>Assignee: Kent Yao
>Priority: Critical
> Fix For: 3.1.1
>
>
> ResolveAggregateFunctions is a hacky rule and it calls `executeSameContext`
> to generate a `resolved agg` to determine which unresolved sort attribute
> should be pushed into the agg. However, after we add the
> PaddingAndLengthCheckForCharVarchar rule which will rewrite the query output,
> thus, the `resolved agg` cannot match original attributes anymore.
> It causes some dissociative sort attribute to be pushed in and fails the query
> {code:java}
> [info] Failed to analyze query: org.apache.spark.sql.AnalysisException:
> expression 'testcat.t1.`v`' is neither present in the group by, nor is it an
> aggregate function. Add to group by or wrap in first() (or first_value) if
> you don't care which value you get.;
> [info] Project [v#14, sum(i)#11L]
> [info] +- Sort [aggOrder#12 ASC NULLS FIRST], true
> [info] +- !Aggregate [v#14], [v#14, sum(cast(i#7 as bigint)) AS
> sum(i)#11L, v#13 AS aggOrder#12]
> [info] +- SubqueryAlias testcat.t1
> [info]+- Project [if ((length(v#6) <= 3)) v#6 else if
> ((length(rtrim(v#6, None)) > 3)) cast(raise_error(concat(input string of
> length , cast(length(v#6) as string), exceeds varchar type length
> limitation: 3)) as string) else rpad(rtrim(v#6, None), 3, ) AS v#14, i#7]
> [info] +- RelationV2[v#6, i#7, index#15, _partition#16]
> testcat.t1
> [info]
> [info] Project [v#14, sum(i)#11L]
> [info] +- Sort [aggOrder#12 ASC NULLS FIRST], true
> [info] +- !Aggregate [v#14], [v#14, sum(cast(i#7 as bigint)) AS
> sum(i)#11L, v#13 AS aggOrder#12]
> [info] +- SubqueryAlias testcat.t1
> [info]+- Project [if ((length(v#6) <= 3)) v#6 else if
> ((length(rtrim(v#6, None)) > 3)) cast(raise_error(concat(input string of
> length , cast(length(v#6) as string), exceeds varchar type length
> limitation: 3)) as string) else rpad(rtrim(v#6, None), 3, ) AS v#14, i#7]
> [info] +- RelationV2[v#6, i#7, index#15, _partition#16]
> testcat.t1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34003) Rule conflicts between PaddingAndLengthCheckForCharVarchar and ResolveAggregateFunctions
[
https://issues.apache.org/jira/browse/SPARK-34003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258723#comment-17258723
]
Apache Spark commented on SPARK-34003:
--
User 'yaooqinn' has created a pull request for this issue:
https://github.com/apache/spark/pull/31027
> Rule conflicts between PaddingAndLengthCheckForCharVarchar and
> ResolveAggregateFunctions
>
>
> Key: SPARK-34003
> URL: https://issues.apache.org/jira/browse/SPARK-34003
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Kent Yao
>Priority: Critical
>
> ResolveAggregateFunctions is a hacky rule and it calls `executeSameContext`
> to generate a `resolved agg` to determine which unresolved sort attribute
> should be pushed into the agg. However, after we add the
> PaddingAndLengthCheckForCharVarchar rule which will rewrite the query output,
> thus, the `resolved agg` cannot match original attributes anymore.
> It causes some dissociative sort attribute to be pushed in and fails the query
> {code:java}
> [info] Failed to analyze query: org.apache.spark.sql.AnalysisException:
> expression 'testcat.t1.`v`' is neither present in the group by, nor is it an
> aggregate function. Add to group by or wrap in first() (or first_value) if
> you don't care which value you get.;
> [info] Project [v#14, sum(i)#11L]
> [info] +- Sort [aggOrder#12 ASC NULLS FIRST], true
> [info] +- !Aggregate [v#14], [v#14, sum(cast(i#7 as bigint)) AS
> sum(i)#11L, v#13 AS aggOrder#12]
> [info] +- SubqueryAlias testcat.t1
> [info]+- Project [if ((length(v#6) <= 3)) v#6 else if
> ((length(rtrim(v#6, None)) > 3)) cast(raise_error(concat(input string of
> length , cast(length(v#6) as string), exceeds varchar type length
> limitation: 3)) as string) else rpad(rtrim(v#6, None), 3, ) AS v#14, i#7]
> [info] +- RelationV2[v#6, i#7, index#15, _partition#16]
> testcat.t1
> [info]
> [info] Project [v#14, sum(i)#11L]
> [info] +- Sort [aggOrder#12 ASC NULLS FIRST], true
> [info] +- !Aggregate [v#14], [v#14, sum(cast(i#7 as bigint)) AS
> sum(i)#11L, v#13 AS aggOrder#12]
> [info] +- SubqueryAlias testcat.t1
> [info]+- Project [if ((length(v#6) <= 3)) v#6 else if
> ((length(rtrim(v#6, None)) > 3)) cast(raise_error(concat(input string of
> length , cast(length(v#6) as string), exceeds varchar type length
> limitation: 3)) as string) else rpad(rtrim(v#6, None), 3, ) AS v#14, i#7]
> [info] +- RelationV2[v#6, i#7, index#15, _partition#16]
> testcat.t1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
