Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18968#discussion_r134146523
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
---
@@ -138,46 +138,63 @@ case class Not(child: Expression)
case class In(value: Expression, list: Seq[Expression]) extends Predicate {
require(list != null, "list should not be null")
+
+ lazy val valExprs = value match {
+ case cns: CreateNamedStruct => cns.valExprs
+ case expr => Seq(expr)
+ }
+
+ override lazy val resolved: Boolean = {
+ lazy val checkForInSubquery = list match {
+ case (l @ ListQuery(sub, children, _)) :: Nil =>
+ // SPARK-21759:
+ // TODO: Update this check if we combine the optimizer rules for
subquery rewriting.
+ //
+ // In `CheckAnalysis`, we already check if the size of subquery
plan output match the size
+ // of value expressions. However, we can add extra correlated
predicate references into
+ // the top of subquery plan when pulling up correlated predicates.
Thus, we add extra check
+ // here to make sure we don't mess the query plan.
+
+ // Try to find out if any extra subquery output doesn't in the
subquery condition.
+ val extraOutputAllInCondition =
sub.output.drop(valExprs.length).find { attr =>
+ l.children.forall { c =>
+ !c.references.contains(attr)
+ }
+ }.isEmpty
+
+ if (sub.output.length >= valExprs.length &&
extraOutputAllInCondition) {
+ true
+ } else {
+ false
+ }
--- End diff --
Line 159 - Line 169 can be simplified to
```Scala
val isAllExtraOutputInCondition =
sub.output.drop(valExprs.length).forall { attr =>
children.exists(_.references.contains(attr))
}
sub.output.length >= valExprs.length && isAllExtraOutputInCondition
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
