[GitHub] spark issue #23204: Revert "[SPARK-21052][SQL] Add hash map metrics to join"

[JkSelf]

Github user JkSelf commented on the issue:

    https://github.com/apache/spark/pull/23204
  
    **Cluster info:**
    
      | Master Node | Worker Nodes
    -- | -- | --
    Node | 1x | 4x
    Processor | Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz | Intel(R) Xeon(R) 
Platinum 8180 CPU @ 2.50GHz
    Memory | 192 GB | 384 GB
    Storage Main | 8 x 960G SSD | 8 x 960G SSD
    Network | 10Gbe
    Role | CM Management NameNodeSecondary NameNodeResource ManagerHive 
Metastore Server | DataNodeNodeManager
    OS Version | CentOS 7.2 | CentOS 7.2
    Hadoop | Apache Hadoop 2.7.5 | Apache Hadoop 2.7.5
    Hive | Apache Hive 2.2.0 |  
    Spark | Apache Spark 2.1.0  & Apache Spark2.3.0 |  
    JDK  version | 1.8.0_112 | 1.8.0_112
    
    **Related parameters setting:**
    
    Component | Parameter | Value
    -- | -- | --
    Yarn Resource Manager | yarn.scheduler.maximum-allocation-mb | 40GB
      | yarn.scheduler.minimum-allocation-mb | 1GB
      | yarn.scheduler.maximum-allocation-vcores | 121
      | Yarn.resourcemanager.scheduler.class | Fair Scheduler
    Yarn Node Manager | yarn.nodemanager.resource.memory-mb | 40GB
      | yarn.nodemanager.resource.cpu-vcores | 121
    Spark | spark.executor.memory | 34GB
      | spark.executor.cores | 40
    
    In above test environment, we found a serious performance degradation issue 
in Spark2.3 when running TPC-DS on SKX 8180. We investigated this problem and 
figured out the root cause is in community patch SPARK-21052 which add metrics 
to hash join process. And the impact code is 
[L486](https://github.com/apache/spark/blob/1d3dd58d21400b5652b75af7e7e53aad85a31528/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L486)
 and 
[L487](https://github.com/apache/spark/blob/1d3dd58d21400b5652b75af7e7e53aad85a31528/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L487)
  .
    
    Following is the result of TPC-DS Q19 in spark2.1, spark2.3 remove 
L486&487, spark2.3 add L486&487 and spark2.4.
    
    spark2.1 | spark2.3 remove L486&487 | spark2.3 addL486&487 | spark2.4
    -- | -- | -- | --
    49s | 47s | 307s | 270s
    
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org