[GitHub] spark pull request #22488: [SPARK-25479][TEST] Refactor DatasetBenchmark to ...
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/22488 [SPARK-25479][TEST] Refactor DatasetBenchmark to use main method ## What changes were proposed in this pull request? Refactor `DatasetBenchmark` to use main method. Generate benchmark result: ```sh SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.DatasetBenchmark" ``` ## How was this patch tested? manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangyum/spark SPARK-25479 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22488.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22488 commit 21b623aad6a84cca2ab5f89f1c29d3b3b1b82d80 Author: Yuming Wang Date: 2018-09-20T09:46:19Z Refactor DatasetBenchmark --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22488: [SPARK-25479][TEST] Refactor DatasetBenchmark to ...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22488#discussion_r219729655 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala --- @@ -242,75 +248,20 @@ object DatasetBenchmark { benchmark } - def main(args: Array[String]): Unit = { -val spark = SparkSession.builder - .master("local[*]") - .appName("Dataset benchmark") - .getOrCreate() + val spark = SparkSession.builder +.master("local[*]") +.appName("Dataset benchmark") +.getOrCreate() --- End diff -- Can we move this SparkSession building part into `benchmark()` function and before `runBenchmark("Dataset Benchmark")`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22488: [SPARK-25479][TEST] Refactor DatasetBenchmark to ...
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/22488#discussion_r221780844 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala --- @@ -242,75 +249,22 @@ object DatasetBenchmark { benchmark } - def main(args: Array[String]): Unit = { -val spark = SparkSession.builder + override def getSparkSession: SparkSession = { --- End diff -- Need override default `SparkSession` as default `SparkSession` is: ```scala SparkSession.builder() .master("local[1]") .appName(this.getClass.getCanonicalName) .config(SQLConf.SHUFFLE_PARTITIONS.key, 1) .config(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key, 1) .getOrCreate() ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22488: [SPARK-25479][TEST] Refactor DatasetBenchmark to ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22488 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org