LuciferYang edited a comment on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-709061277
Address 724eee6 add a simple microbenchmark EDIT: Address 9edf8ad refactor microbenchmark to test more dynamic partitions number: **With this pr** the result is : ``` Running benchmark: dynamic insert table benchmark, totalRows = 200000 Running case: one partition column, 100 partitions Stopped after 2 iterations, 10421 ms Running case: two partition columns, 500 partitions Stopped after 2 iterations, 49308 ms Running case: three partition columns, 2000 partitions Stopped after 2 iterations, 173533 ms Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz dynamic insert table benchmark, totalRows = 200000: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ---------------------------------------------------------------------------------------------------------------------------------- one partition column, 100 partitions 4946 5211 374 0.0 24731.0 1.0X two partition columns, 500 partitions 22929 24654 2440 0.0 114645.4 0.2X three partition columns, 2000 partitions 82092 86767 2609 0.0 410461.3 0.1X ``` **Without this pr** the result is : ``` Running benchmark: dynamic insert table benchmark, totalRows = 200000 Running case: one partition column, 100 partitions Stopped after 2 iterations, 10252 ms Running case: two partition columns, 500 partitions Stopped after 2 iterations, 45089 ms Running case: three partition columns, 2000 partitions Stopped after 2 iterations, 198925 ms Java HotSpot(TM) 64-Bit Server VM 1.8.0_192-b12 on Mac OS X 10.15.7 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz dynamic insert table benchmark, totalRows = 200000: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ---------------------------------------------------------------------------------------------------------------------------------- one partition column, 100 partitions 4840 5126 404 0.0 24201.4 1.0X two partition columns, 500 partitions 20978 22545 2215 0.0 104892.0 0.2X three partition columns, 2000 partitions 86858 99463 2043 0.0 434288.8 0.1X ``` cc @cloud-fan seems no essential difference ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org