[
https://issues.apache.org/jira/browse/SPARK-37989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480963#comment-17480963
]
Yuming Wang commented on SPARK-37989:
-------------------------------------
Benchmark 5
{code:java}
import org.apache.spark.benchmark.Benchmark
import org.apache.spark.sql.catalyst.optimizer.LimitPushDown
val numRows = 1024 * 1024 * 1000
spark.sql(s"CREATE TABLE t1 using parquet AS SELECT id % (${numRows} / 3) AS a,
id % (${numRows} / 3) AS b, id % (${numRows} / 3) AS c FROM range(1,
${numRows}L, 1, 5)")
val benchmark = new Benchmark("Push down limit through Aggregate if it is group
only", numRows, minNumIters = 1)
Seq(LimitPushDown.ruleName, "").foreach { execludedRules =>
benchmark.addCase(s"Push down ${if (execludedRules.length > 0) "disabled"
else "enabled" }") { _ =>
withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key -> execludedRules) {
spark.sql("SELECT distinct * FROM t1 LIMIT
1000").write.format("noop").mode("Overwrite").save()
}
}
}
benchmark.run()
{code}
{noformat}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Push down limit through Aggregate if it is group only: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------------
Push down disabled 752585
752585 0 1.4 717.7 1.0X
Push down enabled 266496
266496 0 3.9 254.1 2.8X
{noformat}
> Push down limit through Aggregate if it is group only
> -----------------------------------------------------
>
> Key: SPARK-37989
> URL: https://issues.apache.org/jira/browse/SPARK-37989
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Yuming Wang
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
