Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/11371#discussion_r64506496 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1443,6 +1445,32 @@ class Analyzer( } /** + * Corrects attribute references in an expression tree of some operators (e.g., filters and + * projects) if these operators have a join as a child and the references point to columns on the + * input relation of the join. This is because some joins change the nullability of input columns + * and this could cause illegal optimization (e.g., NULL propagation) and wrong answers. + * See SPARK-13484 and SPARK-13801 for the concrete queries of this case. + */ + object SolveIllegalReferences extends Rule[LogicalPlan] { + + private def replaceReferences(e: Expression, attrMap: AttributeMap[Attribute]) = e.transform { + case a: AttributeReference => attrMap.get(a).getOrElse(a) + } + + def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators { + case q: LogicalPlan => + q.transform { + case f @ Filter(filterCondition, ExtractJoinOutputAttributes(join, joinOutputMap)) => + f.copy(condition = replaceReferences(filterCondition, joinOutputMap)) --- End diff -- How about we use a `q.transformUp` to fix the nullability in a bottom-up way? For every node, we create an `AttributeMap` using the output of its child. Then, we use `transformExpressions` to fix the nullability if necessary. Let me try it out and ping you when I have a version.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org