[GitHub] spark pull request #22500: [SPARK-25488][TEST] Refactor MiscBenchmark to use...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22500#discussion_r222138844
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/MiscBenchmark.scala
---
@@ -21,247 +21,125 @@ import org.apache.spark.benchmark.Benchmark
/**
* Benchmark to measure whole stage codegen performance.
- * To run this:
- * build/sbt "sql/test-only *benchmark.MiscBenchmark"
- *
- * Benchmarks in this file are skipped in normal builds.
+ * To run this benchmark:
+ * 1. without sbt: bin/spark-submit --class
+ * 2. build/sbt "sql/test:runMain "
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt
"sql/test:runMain "
+ *Results will be written to "benchmarks/MiscBenchmark-results.txt".
*/
-class MiscBenchmark extends BenchmarkWithCodegen {
-
- ignore("filter & aggregate without group") {
--- End diff --
This refactoring introduces a long function body at `runBenchmarkSuite`. In
general, it's not a better direction.
Could you map each `ignore` function into an independent function and make
`runBenchmarkSuite()` invoke a series of those functions?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22500: [SPARK-25488][TEST] Refactor MiscBenchmark to use...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22500#discussion_r220414353 --- Diff: sql/core/benchmarks/MiscBenchmark-results.txt --- @@ -0,0 +1,132 @@ + +filter & aggregate without group + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +range/filter/sum:Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +range/filter/sum wholestage off 36618 / 41080 57.3 17.5 1.0X +range/filter/sum wholestage on2495 / 2609840.4 1.2 14.7X + + + +range/limit/sum + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +range/limit/sum: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +range/limit/sum wholestage off 117 / 121 4477.9 0.2 1.0X +range/limit/sum wholestage on 178 / 187 2938.1 0.3 0.7X + + + +sample + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +sample with replacement: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +sample with replacement wholestage off9142 / 9182 14.3 69.8 1.0X +sample with replacement wholestage on 5926 / 6107 22.1 45.2 1.5X + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +sample without replacement: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +sample without replacement wholestage off 1834 / 1837 71.5 14.0 1.0X +sample without replacement wholestage on 784 / 803167.2 6.0 2.3X + + + +collect + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +collect: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +collect 1 million 186 / 215 5.6 177.5 1.0X +collect 2 millions 361 / 393 2.9 344.2 0.5X +collect 4 millions 884 / 1053 1.2 843.4 0.2X + + + +collect limit + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +collect limit: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +collect limit 1 million206 / 225 5.1 196.6 1.0X +collect limit 2 millions 407 / 419 2.6 387.8 0.5X + + + +generate explode
[GitHub] spark pull request #22500: [SPARK-25488][TEST] Refactor MiscBenchmark to use...
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22500#discussion_r219404392 --- Diff: sql/core/benchmarks/MiscBenchmark-results.txt --- @@ -0,0 +1,132 @@ + +filter & aggregate without group + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +range/filter/sum:Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +range/filter/sum wholestage off 36618 / 41080 57.3 17.5 1.0X +range/filter/sum wholestage on2495 / 2609840.4 1.2 14.7X + + + +range/limit/sum + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +range/limit/sum: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +range/limit/sum wholestage off 117 / 121 4477.9 0.2 1.0X +range/limit/sum wholestage on 178 / 187 2938.1 0.3 0.7X + + + +sample + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +sample with replacement: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +sample with replacement wholestage off9142 / 9182 14.3 69.8 1.0X +sample with replacement wholestage on 5926 / 6107 22.1 45.2 1.5X + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +sample without replacement: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +sample without replacement wholestage off 1834 / 1837 71.5 14.0 1.0X +sample without replacement wholestage on 784 / 803167.2 6.0 2.3X + + + +collect + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +collect: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +collect 1 million 186 / 215 5.6 177.5 1.0X +collect 2 millions 361 / 393 2.9 344.2 0.5X +collect 4 millions 884 / 1053 1.2 843.4 0.2X + + + +collect limit + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +collect limit: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +collect limit 1 million206 / 225 5.1 196.6 1.0X +collect limit 2 millions 407 / 419 2.6 387.8 0.5X + + + +generate explode +
[GitHub] spark pull request #22500: [SPARK-25488][TEST] Refactor MiscBenchmark to use...
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22500#discussion_r219403342 --- Diff: sql/core/benchmarks/MiscBenchmark-results.txt --- @@ -0,0 +1,132 @@ + +filter & aggregate without group + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +range/filter/sum:Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +range/filter/sum wholestage off 36618 / 41080 57.3 17.5 1.0X +range/filter/sum wholestage on2495 / 2609840.4 1.2 14.7X + + + +range/limit/sum + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +range/limit/sum: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +range/limit/sum wholestage off 117 / 121 4477.9 0.2 1.0X +range/limit/sum wholestage on 178 / 187 2938.1 0.3 0.7X + + + +sample + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +sample with replacement: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +sample with replacement wholestage off9142 / 9182 14.3 69.8 1.0X +sample with replacement wholestage on 5926 / 6107 22.1 45.2 1.5X + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +sample without replacement: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +sample without replacement wholestage off 1834 / 1837 71.5 14.0 1.0X +sample without replacement wholestage on 784 / 803167.2 6.0 2.3X + + + +collect + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +collect: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +collect 1 million 186 / 215 5.6 177.5 1.0X +collect 2 millions 361 / 393 2.9 344.2 0.5X +collect 4 millions 884 / 1053 1.2 843.4 0.2X + + + +collect limit + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +collect limit: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +collect limit 1 million206 / 225 5.1 196.6 1.0X +collect limit 2 millions 407 / 419 2.6 387.8 0.5X + + + +generate explode +
[GitHub] spark pull request #22500: [SPARK-25488][TEST] Refactor MiscBenchmark to use...
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22500#discussion_r219219972 --- Diff: sql/core/benchmarks/MiscBenchmark-results.txt --- @@ -0,0 +1,132 @@ + +filter & aggregate without group + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +range/filter/sum:Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +range/filter/sum wholestage off 36618 / 41080 57.3 17.5 1.0X +range/filter/sum wholestage on2495 / 2609840.4 1.2 14.7X + + + +range/limit/sum + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +range/limit/sum: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +range/limit/sum wholestage off 117 / 121 4477.9 0.2 1.0X +range/limit/sum wholestage on 178 / 187 2938.1 0.3 0.7X + + + +sample + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +sample with replacement: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +sample with replacement wholestage off9142 / 9182 14.3 69.8 1.0X +sample with replacement wholestage on 5926 / 6107 22.1 45.2 1.5X + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +sample without replacement: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +sample without replacement wholestage off 1834 / 1837 71.5 14.0 1.0X +sample without replacement wholestage on 784 / 803167.2 6.0 2.3X + + + +collect + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +collect: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +collect 1 million 186 / 215 5.6 177.5 1.0X +collect 2 millions 361 / 393 2.9 344.2 0.5X +collect 4 millions 884 / 1053 1.2 843.4 0.2X + + + +collect limit + + +Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz + +collect limit: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative + +collect limit 1 million206 / 225 5.1 196.6 1.0X +collect limit 2 millions 407 / 419 2.6 387.8 0.5X + + + +generate explode +
[GitHub] spark pull request #22500: [SPARK-25488][TEST] Refactor MiscBenchmark to use...
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/22500#discussion_r219218036
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/MiscBenchmark.scala
---
@@ -17,251 +17,154 @@
package org.apache.spark.sql.execution.benchmark
-import org.apache.spark.util.Benchmark
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.util.{Benchmark, BenchmarkBase =>
FileBenchmarkBase}
/**
* Benchmark to measure whole stage codegen performance.
- * To run this:
- * build/sbt "sql/test-only *benchmark.MiscBenchmark"
- *
- * Benchmarks in this file are skipped in normal builds.
+ * To run this benchmark:
+ * 1. without sbt: bin/spark-submit --class
+ * 2. build/sbt "sql/test:runMain "
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt
"sql/test:runMain "
+ *Results will be written to "benchmarks/MiscBenchmark-results.txt".
*/
-class MiscBenchmark extends BenchmarkBase {
-
- ignore("filter & aggregate without group") {
-val N = 500L << 22
-runBenchmark("range/filter/sum", N) {
- sparkSession.range(N).filter("(id & 1) =
1").groupBy().sum().collect()
+object MiscBenchmark extends FileBenchmarkBase {
+
+ lazy val sparkSession = SparkSession.builder
+.master("local[1]")
+.appName("microbenchmark")
+.config("spark.sql.shuffle.partitions", 1)
+.config("spark.sql.autoBroadcastJoinThreshold", 1)
+.getOrCreate()
+
+ /** Runs function `f` with whole stage codegen on and off. */
+ def runMiscBenchmark(name: String, cardinality: Long)(f: => Unit): Unit
= {
+val benchmark = new Benchmark(name, cardinality, output = output)
+
+benchmark.addCase(s"$name wholestage off", numIters = 2) { iter =>
+ sparkSession.conf.set("spark.sql.codegen.wholeStage", value = false)
+ f
}
-/*
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11
-Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
-
-range/filter/sum:Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-
-range/filter/sum codegen=false 30663 / 31216 68.4
14.6 1.0X
-range/filter/sum codegen=true 2399 / 2409874.1
1.1 12.8X
-*/
- }
- ignore("range/limit/sum") {
-val N = 500L << 20
-runBenchmark("range/limit/sum", N) {
- sparkSession.range(N).limit(100).groupBy().sum().collect()
+benchmark.addCase(s"$name wholestage on", numIters = 5) { iter =>
+ sparkSession.conf.set("spark.sql.codegen.wholeStage", value = true)
+ f
}
-/*
-Westmere E56xx/L56xx/X56xx (Nehalem-C)
-range/limit/sum:Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-
---
-range/limit/sum codegen=false 609 / 672861.6
1.2 1.0X
-range/limit/sum codegen=true 561 / 621935.3
1.1 1.1X
-*/
- }
- ignore("sample") {
-val N = 500 << 18
-runBenchmark("sample with replacement", N) {
- sparkSession.range(N).sample(withReplacement = true,
0.01).groupBy().sum().collect()
-}
-/*
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11
-Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
-
-sample with replacement: Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-
-sample with replacement codegen=false 7073 / 7227 18.5
54.0 1.0X
-sample with replacement codegen=true 5199 / 5203 25.2
39.7 1.4X
-*/
-
-runBenchmark("sample without replacement", N) {
- sparkSession.range(N).sample(withReplacement = false,
0.01).groupBy().sum().collect()
-}
-/*
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11
-Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
-
-sample without replacement: Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-
-sample without replacement codegen=false 1508 / 1529 86.9
11.5 1.0X
-sample without replacement codegen=true644 / 662203.5
[GitHub] spark pull request #22500: [SPARK-25488][TEST] Refactor MiscBenchmark to use...
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22500 [SPARK-25488][TEST] Refactor MiscBenchmark to use main method ## What changes were proposed in this pull request? Refactor `MiscBenchmark ` to use main method. Generate benchmark result: ```sh SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.MiscBenchmark" ``` ## How was this patch tested? manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangyum/spark SPARK-25488 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22500.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22500 commit 6252440c1a079bbb12d41e2ae513f988fcdf5651 Author: Yuming Wang Date: 2018-09-20T15:41:03Z Refactor MiscBenchmark --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
