Yuming Wang created PARQUET-1355: ------------------------------------ Summary: Improvement parquet Binary write performance Key: PARQUET-1355 URL: https://issues.apache.org/jira/browse/PARQUET-1355 Project: Parquet Issue Type: Improvement Components: parquet-mr Affects Versions: 1.10.0 Reporter: Yuming Wang
*Benchmark code*: {code:java} test("Parquet write benchmark") { val count = 100 * 1024 * 1024 val numIters = 5 withTempPath { path => val benchmark = new Benchmark(s"Parquet write benchmark ${spark.sparkContext.version}", 5) Seq("long", "string", "decimal(18, 0)", "decimal(38, 18)", "timestamp").foreach { dt => benchmark.addCase(s"$dt type", numIters = numIters) { iter => spark.range(count).selectExpr(s"cast(id as $dt) as id") .write.mode("overwrite").parquet(path.getAbsolutePath) } } benchmark.run() } } {code} *Result*: {noformat} -- Spark 2.3.3-SNAPSHOT with Parquet 1.8.3 Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz Parquet write benchmark 2.3.3-SNAPSHOT: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ long type 10963 / 11344 0.0 2192675973.8 1.0X string type 28423 / 29437 0.0 5684553922.2 0.4X decimal(18, 0) type 11558 / 11696 0.0 2311587203.6 0.9X decimal(38, 18) type 43858 / 44432 0.0 8771537663.4 0.2X -- Spark 2.4.0-SNAPSHOT with Parquet 1.10.0 Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz Parquet write benchmark 2.4.0-SNAPSHOT: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ long type 11633 / 12070 0.0 2326572295.8 1.0X string type 31374 / 32178 0.0 6274760187.4 0.4X decimal(18, 0) type 13019 / 13294 0.0 2603841925.4 0.9X decimal(38, 18) type 50719 / 50983 0.0 10143775007.6 0.2X {noformat} The mainly is [toByteBuffer|https://github.com/apache/parquet-mr/blob/d61d221c9e752ce2cc0da65ede8b55653b3ae21f/parquet-column/src/main/java/org/apache/parquet/io/api/Binary.java#L83] affects performance. If do not use the {{toByteBuffer}} when compare binary, the result is: {noformat} -- Spark 2.4.0-SNAPSHOT with Parquet 1.10.0 Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz Parquet write benchmark 2.4.0-SNAPSHOT: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ long type 11171 / 11508 0.0 2234189382.0 1.0X string type 30072 / 30290 0.0 6014346455.4 0.4X decimal(18, 0) type 12150 / 12239 0.0 2430052708.8 0.9X decimal(38, 18) type 44974 / 45423 0.0 8994773738.8 0.2X {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)