maropu commented on a change in pull request #29094:
URL: https://github.com/apache/spark/pull/29094#discussion_r458575888
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -697,16 +697,21 @@ object ColumnPruning extends Rule[LogicalPlan] {
* `GlobalLimit(LocalLimit)` pattern is also considered.
*/
object CollapseProject extends Rule[LogicalPlan] {
+ // If number of leaf expressions exceed MAX_LEAF_SIZE, do not collapse to
prevent driver oom
+ // due to a single large project.
+ private val MAX_LEAF_SIZE = 1000
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
case p1 @ Project(_, p2: Project) =>
Review comment:
Yea, if we add this logic, I think we need a conf for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
