[
https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701533#comment-16701533
]
Ke Jia commented on SPARK-26155:
--------------------------------
[~viirya] [~adrian-wang]
Here is the result in spark2.1, spark2.3 and spark2.4 same with above cluster
info except the slave node from 7 to 4.
|spark2.1|spark2.3 remove L486&487|spark2.3 addL486&487|spark2.4|
|49s|47s|307s|270s|
> Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS
> in 3TB scale
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-26155
> URL: https://issues.apache.org/jira/browse/SPARK-26155
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0, 2.3.1, 2.3.2, 2.4.0
> Reporter: Ke Jia
> Priority: Major
> Attachments: Q19 analysis in Spark2.3 with L486&487.pdf, Q19 analysis
> in Spark2.3 without L486&487.pdf, q19.sql
>
>
> In our test environment, we found a serious performance degradation issue in
> Spark2.3 when running TPC-DS on SKX 8180. Several queries have serious
> performance degradation. For example, TPC-DS Q19 needs 126 seconds with Spark
> 2.3 while it needs only 29 seconds with Spark2.1 on 3TB data. We investigated
> this problem and figured out the root cause is in community patch SPARK-21052
> which add metrics to hash join process. And the impact code is
> [L486|https://github.com/apache/spark/blob/1d3dd58d21400b5652b75af7e7e53aad85a31528/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L486]
> and
> [L487|https://github.com/apache/spark/blob/1d3dd58d21400b5652b75af7e7e53aad85a31528/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L487]
> . Q19 costs about 30 seconds without these two lines code and 126 seconds
> with these code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
