Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16246#discussion_r91848250
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1225,15 +1225,37 @@ class Analyzer(
}
/**
- * Rewrite the subquery in a safe way by preventing that the subquery
and the outer use the same
- * attributes.
+ * Rewrite the subquery in a safe way by preventing that the subquery
and
+ * the outer use the same attributes.
+ *
+ * If this is a scalar subquery, check that GROUP BY columns are a
subset
+ * of the columns used in the correlated predicate(s). Otherwise,
pulling up
+ * correlated predicates could cause incorrect results.
*/
private def rewriteSubQuery(
sub: LogicalPlan,
- outer: Seq[LogicalPlan]): (LogicalPlan, Seq[Expression]) = {
+ outer: Seq[LogicalPlan],
+ scalarSubq: Boolean = false): (LogicalPlan, Seq[Expression]) = {
// Pull out the tagged predicates and rewrite the subquery in the
process.
val (basePlan, baseConditions) = pullOutCorrelatedPredicates(sub)
+ // SPARK-18504/SPARK-18814:
+ // Block cases where GROUP BY columns are not part of the correlated
columns
+ // of a scalar subquery.
+ if (scalarSubq) {
+ sub match {
+ case a @ Aggregate(grouping, _, _) =>
+ val groupByCols = ExpressionSet(grouping.flatMap(_.references))
+ val conditionsCols =
ExpressionSet(baseConditions.flatMap(_.references))
+ val invalidCols = groupByCols.diff(conditionsCols)
+ if (invalidCols.nonEmpty) {
+ failAnalysis("A GROUP BY clause in a scalar correlated
subquery " +
+ "cannot contain non-correlated columns: " +
+ invalidCols.mkString(","))
+ }
+ case _ => None
--- End diff --
This is not needed, if we use `collect` to replace the above `match`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
