[ https://issues.apache.org/jira/browse/SPARK-21807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li reassigned SPARK-21807: ------------------------------- Assignee: eaton > The getAliasedConstraints function in LogicalPlan will take a long time when > number of expressions is greater than 100 > ------------------------------------------------------------------------------------------------------------------------ > > Key: SPARK-21807 > URL: https://issues.apache.org/jira/browse/SPARK-21807 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.2.0 > Reporter: eaton > Assignee: eaton > > The getAliasedConstraints fuction in LogicalPlan.scala will clone the > expression set when an element added, > and it will take a long time. > Before modified, the cost of getAliasedConstraints is: > 100 expressions: 41 seconds > 150 expressions: 466 seconds > The test is like this: > test("getAliasedConstraints") { > val expressionNum = 150 > val aggExpression = (1 to expressionNum).map(i => Alias(Count(Literal(1)), > s"cnt$i")()) > val aggPlan = Aggregate(Nil, aggExpression, LocalRelation()) > val beginTime = System.currentTimeMillis() > val expressions = aggPlan.validConstraints > println(s"validConstraints cost: ${System.currentTimeMillis() - beginTime}ms") > // The size of Aliased expression is n * (n - 1) / 2 + n > assert( expressions.size === expressionNum * (expressionNum - 1) / 2 + > expressionNum) > } -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org