[jira] [Comment Edited] (SPARK-40303) The performance will be worse after codegen
[
https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603883#comment-17603883
]
Yang Jie edited comment on SPARK-40303 at 9/14/22 5:24 AM:
---
I did a simple experiment to compare the following scenarios：
# A method with 127 input parameters
# Encapsulate the input parameters of the above method as a specific type,
including 127 fields, and create a new parameter object before each call
# Encapsulate the input parameters of the above method as a specific type,
including 127 fields, reuse one parameter object and reset + refill parameter
data before each call
I confirmed that the JIT compilation failure mentioned above will occur in the
1&2 test scenarios， the test result as follows:
Java 8
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
Use multiple parameters method 35772 36161
550 0.3 3577.2 1.0X
Use TestParameters create new 38701 38783
115 0.3 3870.1 0.9X
Use TestParameters reuse 17986 18125
196 0.6 1798.6 2.0X {code}
Java 11
{code:java}
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test sum: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
Use multiple parameters method 12253 12286
46 0.8 1225.3 1.0X
Use TestParameters create new 13644 13665
30 0.7 1364.4 0.9X
Use TestParameters reuse 13188 13219
44 0.8 1318.8 0.9X {code}
Java 17
{code:java}
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Test sum: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
Use multiple parameters method 14044 14128
119 0.7 1404.4 1.0X
Use TestParameters create new 16174 16289
162 0.6 1617.4 0.9X
Use TestParameters reuse 15633 15638
8 0.6 1563.3 0.9X {code}
>From the test results, encapsulating and reusing specific parameter types will
>only alleviate the problem when using Java 8, but running programs using Java
>11 or Java 17 seems to be a simpler and more effective way.
So for the current issue, I suggest upgrading the Java runtime environment to
solve the problem. [~yumwang]
Uploaded the test program to the attachment
was (Author: luciferyang):
I did a simple experiment to compare the following scenarios：
# A method with 127 input parameters
# Encapsulate the input parameters of the above method as a specific type,
including 127 fields, and create a new parameter object before each call
# Encapsulate the input parameters of the above method as a specific type,
including 127 fields, reuse one parameter object and reset + refill parameter
data before each call
I confirmed that the JIT compilation failure mentioned above will occur in the
1&2 test scenarios， the test result as follows:
Java 8
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test sum: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
Use multiple parameters method 35772 36161
550 0.3 3577.2 1.0X
Use TestParameters create new 38701 38783
115 0.3 3870.1 0.9X
Use TestParameters reuse 17986 18125
196 0.6 1798.6 2.0X {code}
Java 11
{code:java}
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test sum:
[jira] [Comment Edited] (SPARK-40303) The performance will be worse after codegen
[
https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599263#comment-17599263
]
Yang Jie edited comment on SPARK-40303 at 9/2/22 5:48 AM:
--
If run with Java 17, the performance gap will be smaller. The compilation logs
of `hashAgg_doConsume_0` and `hashAgg_doAggregateWithKeys_0` as follows:
{code:java}
102158 22568 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
102180 22606 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
102228 22606 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (retry
at different tier)
102228 22619 1
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
102240 22568 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) made not entrant
218296 24067 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2::hashAgg_doConsume_0$
(2052 bytes)
218463 22568 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) made zombie {code}
{code:java}
105832 22708 % 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes)
105955 22709 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
(2712 bytes)
108247 22741 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes)
108484 22741 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (retry at
different tier)
108727 22708 % 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) made not entrant
108727 22743 % 1
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes)
218463 22708 % 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) made zombie {code}
was (Author: luciferyang):
If you run with Java 17, the performance gap will be smaller. The compilation
logs of `hashAgg_doConsume_0` and `hashAgg_doAggregateWithKeys_0` as follows:
{code:java}
102158 22568 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
102180 22606 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
102228 22606 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (retry
at different tier)
102228 22619 1
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
102240 22568 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) made not entrant
218296 24067 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2::hashAgg_doConsume_0$
(2052 bytes)
218463 22568 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) made zombie {code}
{code:java}
105832 22708 % 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes)
105955 22709 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
(2712 bytes)
108247 22741 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes)
108484 22741 % 4
[jira] [Comment Edited] (SPARK-40303) The performance will be worse after codegen
[
https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599262#comment-17599262
]
Yang Jie edited comment on SPARK-40303 at 9/2/22 5:15 AM:
--
{code:java}
64265 21820 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
64308 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable)
1735350 24101 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2::hashAgg_doConsume_0$
(2052 bytes){code}
{code:java}
103941 22065 % 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes)
104602 22067 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
(2712 bytes)
135979 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
The compilation relevant logs are as above when Java 8 is used
was (Author: luciferyang):
{code:java}
64265 21820 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
64308 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes)
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable) {code}
{code:java}
103941 22065 % 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes)
104602 22067 3
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
(2712 bytes)
135979 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
The compilation relevant logs are as above when Java 8 is used
> The performance will be worse after codegen
> ---
>
> Key: SPARK-40303
> URL: https://issues.apache.org/jira/browse/SPARK-40303
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.0
>Reporter: Yuming Wang
>Priority: Major
>
> {code:scala}
> import org.apache.spark.benchmark.Benchmark
> val dir = "/tmp/spark/benchmark"
> val N = 200
> val columns = Range(0, 100).map(i => s"id % $i AS id$i")
> spark.range(N).selectExpr(columns: _*).write.mode("Overwrite").parquet(dir)
> // Seq(1, 2, 5, 10, 15, 25, 40, 60, 100)
> Seq(60).foreach{ cnt =>
> val selectExps = columns.take(cnt).map(_.split(" ").last).map(c =>
> s"count(distinct $c)")
> val benchmark = new Benchmark("Benchmark count distinct", N, minNumIters =
> 1)
> benchmark.addCase(s"$cnt count distinct with codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "true",
> "spark.sql.codegen.factoryMode" -> "FALLBACK") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.addCase(s"$cnt count distinct without codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "false",
> "spark.sql.codegen.factoryMode" -> "NO_CODEGEN") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.run()
> }
> {code}
> {noformat}
> Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark count distinct: Best Time(ms) Avg Time(ms)
>
[jira] [Comment Edited] (SPARK-40303) The performance will be worse after codegen
[
https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599254#comment-17599254
]
Yang Jie edited comment on SPARK-40303 at 9/2/22 4:59 AM:
--
Run use Java 8 with `-XX:+PrintCompilation`, I found the following logs:
{code:java}
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
[~rednaxelafx] In this case, will compilation optimization degenerate to Level
3 or give up when use Java 8?
was (Author: luciferyang):
Run use Java 8 with `-XX:+PrintCompilation`, I found the following logs:
{code:java}
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
[~rednaxelafx] In this case, will compilation optimization degenerate to Level
2 or give up when use Java 8?
> The performance will be worse after codegen
> ---
>
> Key: SPARK-40303
> URL: https://issues.apache.org/jira/browse/SPARK-40303
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.0
>Reporter: Yuming Wang
>Priority: Major
>
> {code:scala}
> import org.apache.spark.benchmark.Benchmark
> val dir = "/tmp/spark/benchmark"
> val N = 200
> val columns = Range(0, 100).map(i => s"id % $i AS id$i")
> spark.range(N).selectExpr(columns: _*).write.mode("Overwrite").parquet(dir)
> // Seq(1, 2, 5, 10, 15, 25, 40, 60, 100)
> Seq(60).foreach{ cnt =>
> val selectExps = columns.take(cnt).map(_.split(" ").last).map(c =>
> s"count(distinct $c)")
> val benchmark = new Benchmark("Benchmark count distinct", N, minNumIters =
> 1)
> benchmark.addCase(s"$cnt count distinct with codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "true",
> "spark.sql.codegen.factoryMode" -> "FALLBACK") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.addCase(s"$cnt count distinct without codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "false",
> "spark.sql.codegen.factoryMode" -> "NO_CODEGEN") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.run()
> }
> {code}
> {noformat}
> Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark count distinct: Best Time(ms) Avg Time(ms)
> Stdev(ms)Rate(M/s) Per Row(ns) Relative
>
> 60 count distinct with codegen 628146 628146
>0 0.0 314072.8 1.0X
> 60 count distinct without codegen147635 147635
>0 0.0 73817.5 4.3X
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-40303) The performance will be worse after codegen
[
https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599254#comment-17599254
]
Yang Jie edited comment on SPARK-40303 at 9/2/22 4:58 AM:
--
Run use Java 8 with `-XX:+PrintCompilation`, I found the following logs:
{code:java}
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
[~rednaxelafx] In this case, will compilation optimization degenerate to Level
2 or give up when use Java 8?
was (Author: luciferyang):
Run use Java 8 with `-XX:+PrintCompilation`, I found the following logs:
{code:java}
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
[~rednaxelafx] In this case, will compilation optimization degenerate to Level
2 or give up?
> The performance will be worse after codegen
> ---
>
> Key: SPARK-40303
> URL: https://issues.apache.org/jira/browse/SPARK-40303
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.0
>Reporter: Yuming Wang
>Priority: Major
>
> {code:scala}
> import org.apache.spark.benchmark.Benchmark
> val dir = "/tmp/spark/benchmark"
> val N = 200
> val columns = Range(0, 100).map(i => s"id % $i AS id$i")
> spark.range(N).selectExpr(columns: _*).write.mode("Overwrite").parquet(dir)
> // Seq(1, 2, 5, 10, 15, 25, 40, 60, 100)
> Seq(60).foreach{ cnt =>
> val selectExps = columns.take(cnt).map(_.split(" ").last).map(c =>
> s"count(distinct $c)")
> val benchmark = new Benchmark("Benchmark count distinct", N, minNumIters =
> 1)
> benchmark.addCase(s"$cnt count distinct with codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "true",
> "spark.sql.codegen.factoryMode" -> "FALLBACK") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.addCase(s"$cnt count distinct without codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "false",
> "spark.sql.codegen.factoryMode" -> "NO_CODEGEN") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.run()
> }
> {code}
> {noformat}
> Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark count distinct: Best Time(ms) Avg Time(ms)
> Stdev(ms)Rate(M/s) Per Row(ns) Relative
>
> 60 count distinct with codegen 628146 628146
>0 0.0 314072.8 1.0X
> 60 count distinct without codegen147635 147635
>0 0.0 73817.5 4.3X
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-40303) The performance will be worse after codegen
[
https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599254#comment-17599254
]
Yang Jie edited comment on SPARK-40303 at 9/2/22 4:56 AM:
--
Run use Java 8 with `-XX:+PrintCompilation`, I found the following logs:
{code:java}
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
[~rednaxelafx] In this case, will compilation optimization degenerate to Level
2 or give up?
was (Author: luciferyang):
Run use Java 8 with `-XX:+PrintCompilation`, I found the following logs:
{code:java}
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
[~rednaxelafx] In this case, will compilation optimization degenerate to Level
2?
> The performance will be worse after codegen
> ---
>
> Key: SPARK-40303
> URL: https://issues.apache.org/jira/browse/SPARK-40303
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.0
>Reporter: Yuming Wang
>Priority: Major
>
> {code:scala}
> import org.apache.spark.benchmark.Benchmark
> val dir = "/tmp/spark/benchmark"
> val N = 200
> val columns = Range(0, 100).map(i => s"id % $i AS id$i")
> spark.range(N).selectExpr(columns: _*).write.mode("Overwrite").parquet(dir)
> // Seq(1, 2, 5, 10, 15, 25, 40, 60, 100)
> Seq(60).foreach{ cnt =>
> val selectExps = columns.take(cnt).map(_.split(" ").last).map(c =>
> s"count(distinct $c)")
> val benchmark = new Benchmark("Benchmark count distinct", N, minNumIters =
> 1)
> benchmark.addCase(s"$cnt count distinct with codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "true",
> "spark.sql.codegen.factoryMode" -> "FALLBACK") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.addCase(s"$cnt count distinct without codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "false",
> "spark.sql.codegen.factoryMode" -> "NO_CODEGEN") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.run()
> }
> {code}
> {noformat}
> Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark count distinct: Best Time(ms) Avg Time(ms)
> Stdev(ms)Rate(M/s) Per Row(ns) Relative
>
> 60 count distinct with codegen 628146 628146
>0 0.0 314072.8 1.0X
> 60 count distinct without codegen147635 147635
>0 0.0 73817.5 4.3X
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-40303) The performance will be worse after codegen
[
https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599254#comment-17599254
]
Yang Jie edited comment on SPARK-40303 at 9/2/22 4:49 AM:
--
Run use Java 8 with `-XX:+PrintCompilation`, I found the following logs:
{code:java}
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
[~rednaxelafx] In this case, will compilation optimization degenerate to Level
2?
was (Author: luciferyang):
Run use Java 8 with `-XX:+PrintCompilation`, I found the following logs:
{code:java}
64350 21862 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doConsume_0$
(2051 bytes) COMPILE SKIPPED: unsupported incoming calling sequence (not
retryable)
137245 22141 % 4
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1::hashAgg_doAggregateWithKeys_0$
@ 38 (2712 bytes) COMPILE SKIPPED: unsupported calling sequence (not
retryable) {code}
> The performance will be worse after codegen
> ---
>
> Key: SPARK-40303
> URL: https://issues.apache.org/jira/browse/SPARK-40303
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.4.0
>Reporter: Yuming Wang
>Priority: Major
>
> {code:scala}
> import org.apache.spark.benchmark.Benchmark
> val dir = "/tmp/spark/benchmark"
> val N = 200
> val columns = Range(0, 100).map(i => s"id % $i AS id$i")
> spark.range(N).selectExpr(columns: _*).write.mode("Overwrite").parquet(dir)
> // Seq(1, 2, 5, 10, 15, 25, 40, 60, 100)
> Seq(60).foreach{ cnt =>
> val selectExps = columns.take(cnt).map(_.split(" ").last).map(c =>
> s"count(distinct $c)")
> val benchmark = new Benchmark("Benchmark count distinct", N, minNumIters =
> 1)
> benchmark.addCase(s"$cnt count distinct with codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "true",
> "spark.sql.codegen.factoryMode" -> "FALLBACK") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.addCase(s"$cnt count distinct without codegen") { _ =>
> withSQLConf(
> "spark.sql.codegen.wholeStage" -> "false",
> "spark.sql.codegen.factoryMode" -> "NO_CODEGEN") {
> spark.read.parquet(dir).selectExpr(selectExps:
> _*).write.format("noop").mode("Overwrite").save()
> }
> }
> benchmark.run()
> }
> {code}
> {noformat}
> Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
> Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
> Benchmark count distinct: Best Time(ms) Avg Time(ms)
> Stdev(ms)Rate(M/s) Per Row(ns) Relative
>
> 60 count distinct with codegen 628146 628146
>0 0.0 314072.8 1.0X
> 60 count distinct without codegen147635 147635
>0 0.0 73817.5 4.3X
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
