[
https://issues.apache.org/jira/browse/SPARK-42789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-42789:
--------------------------------
Description:
Benchmark result:
{noformat}
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 2
Stopped after 2 iterations, 77193 ms
Running case: Rewrite: 2
Stopped after 2 iterations, 51699 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 2 37914 38597
966 0.2 5244.0 1.0X
Rewrite: 2 24887 25850
1361 0.3 3442.2 1.5X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 3
Stopped after 2 iterations, 110890 ms
Running case: Rewrite: 3
Stopped after 2 iterations, 56102 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 3 52862 55445
NaN 0.1 7311.6 1.0X
Rewrite: 3 26752 28051
1837 0.3 3700.2 2.0X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 4
Stopped after 2 iterations, 150828 ms
Running case: Rewrite: 4
Stopped after 2 iterations, 57110 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 4 71680 75414
NaN 0.1 9914.4 1.0X
Rewrite: 4 28452 28555
145 0.3 3935.4 2.5X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 5
Stopped after 2 iterations, 223367 ms
Running case: Rewrite: 5
Stopped after 2 iterations, 78193 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 5 108479 111684
1447 0.1 15004.2 1.0X
Rewrite: 5 36830 39097
NaN 0.2 5094.0 2.9X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 10
Stopped after 2 iterations, 311453 ms
Running case: Rewrite: 10
Stopped after 2 iterations, 65873 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 10 153952 155727
2510 0.0 21293.7 1.0X
Rewrite: 10 32436 32937
708 0.2 4486.3 4.7X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 15
Stopped after 2 iterations, 451911 ms
Running case: Rewrite: 15
Stopped after 2 iterations, 69790 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 15 224950 225956
1423 0.0 31113.6 1.0X
Rewrite: 15 34806 34895
126 0.2 4814.2 6.5X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 20
Stopped after 2 iterations, 587378 ms
Running case: Rewrite: 20
Stopped after 2 iterations, 76667 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 20 293155 293689
756 0.0 40547.3 1.0X
Rewrite: 20 38148 38334
262 0.2 5276.4 7.7X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 25
Stopped after 2 iterations, 732228 ms
Running case: Rewrite: 25
00:38:55.284 WARN org.apache.spark.sql.catalyst.util.package: Truncated the
string representation of a plan since it was too large. This behavior can be
adjusted by setting 'spark.sql.debug.maxToStringFields'.
Stopped after 2 iterations, 93089 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 25 366037 366114
110 0.0 50627.9 1.0X
Rewrite: 25 45725 46545
1159 0.2 6324.4 8.0X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 30
Stopped after 2 iterations, 881176 ms
Running case: Rewrite: 30
Stopped after 2 iterations, 107241 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 30 439567 440588
1444 0.0 60798.2 1.0X
Rewrite: 30 52949 53621
950 0.1 7323.5 8.3X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 36
Stopped after 2 iterations, 1055559 ms
Running case: Rewrite: 36
Stopped after 2 iterations, 124081 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 35 526481 527780
1837 0.0 72819.5 1.0X
Rewrite: 35 60586 62041
2058 0.1 8379.8 8.7X
{noformat}
was:
Benchmark result:
{noformat}
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 2
Stopped after 2 iterations, 80787 ms
Running case: Rewrite: 2
Stopped after 2 iterations, 48900 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 2 39026 40394
1935 0.2 5397.8 1.0X
Rewrite: 2 24354 24450
137 0.3 3368.4 1.6X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 3
Stopped after 2 iterations, 115055 ms
Running case: Rewrite: 3
Stopped after 2 iterations, 62297 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 3 54652 57528
NaN 0.1 7559.1 1.0X
Rewrite: 3 30702 31149
631 0.2 4246.6 1.8X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 4
Stopped after 2 iterations, 155392 ms
Running case: Rewrite: 4
Stopped after 2 iterations, 54776 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 4 75503 77696
NaN 0.1 10443.1 1.0X
Rewrite: 4 26962 27388
602 0.3 3729.3 2.8X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 5
Stopped after 2 iterations, 192836 ms
Running case: Rewrite: 5
Stopped after 2 iterations, 51967 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 5 94923 96418
2115 0.1 13129.1 1.0X
Rewrite: 5 25362 25984
880 0.3 3507.8 3.7X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 10
Stopped after 2 iterations, 317246 ms
Running case: Rewrite: 10
Stopped after 2 iterations, 56734 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 10 157458 158623
1648 0.0 21778.6 1.0X
Rewrite: 10 28296 28367
100 0.3 3913.8 5.6X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 20
Stopped after 2 iterations, 618089 ms
Running case: Rewrite: 20
Stopped after 2 iterations, 63576 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 20 285338 309045
NaN 0.0 39466.2 1.0X
Rewrite: 20 31682 31788
151 0.2 4382.0 9.0X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 30
07:25:58.851 WARN org.apache.spark.sql.catalyst.util.package: Truncated the
string representation of a plan since it was too large. This behavior can be
adjusted by setting 'spark.sql.debug.maxToStringFields'.
Stopped after 2 iterations, 1113910 ms
Running case: Rewrite: 30
Stopped after 2 iterations, 101468 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 30 481691 556955
1722 0.0 66624.5 1.0X
Rewrite: 30 50497 50734
335 0.1 6984.5 9.5X
Running benchmark: Benchmark rewrite GetJsonObjects
Running case: Default: 36
Stopped after 2 iterations, 1272619 ms
Running case: Rewrite: 36
Stopped after 2 iterations, 81609 ms
Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Default: 36 576500 636310
NaN 0.0 79737.8 1.0X
Rewrite: 36 40461 40805
486 0.2 5596.4 14.2X
{noformat}
> Rewrite multiple GetJsonObjects to a JsonTuple if their json expression is
> the same
> -----------------------------------------------------------------------------------
>
> Key: SPARK-42789
> URL: https://issues.apache.org/jira/browse/SPARK-42789
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.0
> Reporter: Yuming Wang
> Priority: Major
>
> Benchmark result:
> {noformat}
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 2
> Stopped after 2 iterations, 77193 ms
> Running case: Rewrite: 2
> Stopped after 2 iterations, 51699 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 2 37914 38597
> 966 0.2 5244.0 1.0X
> Rewrite: 2 24887 25850
> 1361 0.3 3442.2 1.5X
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 3
> Stopped after 2 iterations, 110890 ms
> Running case: Rewrite: 3
> Stopped after 2 iterations, 56102 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 3 52862 55445
> NaN 0.1 7311.6 1.0X
> Rewrite: 3 26752 28051
> 1837 0.3 3700.2 2.0X
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 4
> Stopped after 2 iterations, 150828 ms
> Running case: Rewrite: 4
> Stopped after 2 iterations, 57110 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 4 71680 75414
> NaN 0.1 9914.4 1.0X
> Rewrite: 4 28452 28555
> 145 0.3 3935.4 2.5X
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 5
> Stopped after 2 iterations, 223367 ms
> Running case: Rewrite: 5
> Stopped after 2 iterations, 78193 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 5 108479 111684
> 1447 0.1 15004.2 1.0X
> Rewrite: 5 36830 39097
> NaN 0.2 5094.0 2.9X
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 10
> Stopped after 2 iterations, 311453 ms
> Running case: Rewrite: 10
> Stopped after 2 iterations, 65873 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 10 153952 155727
> 2510 0.0 21293.7 1.0X
> Rewrite: 10 32436 32937
> 708 0.2 4486.3 4.7X
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 15
> Stopped after 2 iterations, 451911 ms
> Running case: Rewrite: 15
> Stopped after 2 iterations, 69790 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 15 224950 225956
> 1423 0.0 31113.6 1.0X
> Rewrite: 15 34806 34895
> 126 0.2 4814.2 6.5X
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 20
> Stopped after 2 iterations, 587378 ms
> Running case: Rewrite: 20
> Stopped after 2 iterations, 76667 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 20 293155 293689
> 756 0.0 40547.3 1.0X
> Rewrite: 20 38148 38334
> 262 0.2 5276.4 7.7X
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 25
> Stopped after 2 iterations, 732228 ms
> Running case: Rewrite: 25
> 00:38:55.284 WARN org.apache.spark.sql.catalyst.util.package: Truncated the
> string representation of a plan since it was too large. This behavior can be
> adjusted by setting 'spark.sql.debug.maxToStringFields'.
> Stopped after 2 iterations, 93089 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 25 366037 366114
> 110 0.0 50627.9 1.0X
> Rewrite: 25 45725 46545
> 1159 0.2 6324.4 8.0X
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 30
> Stopped after 2 iterations, 881176 ms
> Running case: Rewrite: 30
> Stopped after 2 iterations, 107241 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 30 439567 440588
> 1444 0.0 60798.2 1.0X
> Rewrite: 30 52949 53621
> 950 0.1 7323.5 8.3X
> Running benchmark: Benchmark rewrite GetJsonObjects
> Running case: Default: 36
> Stopped after 2 iterations, 1055559 ms
> Running case: Rewrite: 36
> Stopped after 2 iterations, 124081 ms
> Java HotSpot(TM) 64-Bit Server VM 17.0.4.1+1-LTS-2 on Mac OS X 13.2.1
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark rewrite GetJsonObjects: Best Time(ms) Avg Time(ms)
> Stdev(ms) Rate(M/s) Per Row(ns) Relative
> ------------------------------------------------------------------------------------------------------------------------
> Default: 35 526481 527780
> 1837 0.0 72819.5 1.0X
> Rewrite: 35 60586 62041
> 2058 0.1 8379.8 8.7X
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
