constzhou commented on a change in pull request #29094:
URL: https://github.com/apache/spark/pull/29094#discussion_r458607522
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CollapseProjectSuite.scala
##########
@@ -121,6 +121,17 @@ class CollapseProjectSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
+ test("do not collapse project if number of leave expressions would be too
big") {
+ var query: LogicalPlan = testRelation
+ for( _ <- 1 to 10) {
+ // after n iterations the number of leaf expressions will be 2^{n+1}
+ // => after 10 iterations we would end up with more than 1000 leaf
expressions
+ query = query.select(('a + 'b).as('a), ('a - 'b).as('b))
Review comment:
Thanks for reminding, my test case is :
var query = spark.range(5).withColumn("new_column", 'id + 5 as
"plus5").toDF("a","b")
for( a <- 1 to 10) {query = query.select(('a + 'b).as('a), ('a - 'b).as('b))}
query.explain(true)
And it works for both Optimized Logical Plan and Physical Plan.
I notice the difference is that my data type is bigint:
org.apache.spark.sql.DataFrame = [a: bigint, b: bigint], it seems the project
will not collapse
I test the case above and the problem exist for Physical Plan, so we also
add a check for that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
