Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19424 @rdblue we assume the previous batches should push down operators as close to data source relations as possible. One special case is column pruning. The `ColumnPruning` rule tries to generate a new `Project` node under every node that has more input columns than it references, which means it will also generate a `Project` under `Filter`, and this conflicts with `PushDownPredicate` rule. Currently the conflict is resolved by a hacky way so there are cases `Project` under `Filter`.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org