Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/11371#discussion_r64521117
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1443,6 +1445,32 @@ class Analyzer(
}
/**
+ * Corrects attribute references in an expression tree of some operators
(e.g., filters and
+ * projects) if these operators have a join as a child and the
references point to columns on the
+ * input relation of the join. This is because some joins change the
nullability of input columns
+ * and this could cause illegal optimization (e.g., NULL propagation)
and wrong answers.
+ * See SPARK-13484 and SPARK-13801 for the concrete queries of this case.
+ */
+ object SolveIllegalReferences extends Rule[LogicalPlan] {
+
+ private def replaceReferences(e: Expression, attrMap:
AttributeMap[Attribute]) = e.transform {
+ case a: AttributeReference => attrMap.get(a).getOrElse(a)
+ }
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case q: LogicalPlan =>
+ q.transform {
+ case f @ Filter(filterCondition,
ExtractJoinOutputAttributes(join, joinOutputMap)) =>
+ f.copy(condition = replaceReferences(filterCondition,
joinOutputMap))
--- End diff --
https://github.com/apache/spark/pull/13290/files
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
