Lifeng Wang created SPARK-18738:
-----------------------------------

             Summary: Some Spark SQL queries has poor performance on HDFS 
Erasure Coding feature when enabling dynamic allocation.
                 Key: SPARK-18738
                 URL: https://issues.apache.org/jira/browse/SPARK-18738
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.2
            Reporter: Lifeng Wang
             Fix For: 2.2.0


We run TPCx-BB with Spark SQL engine on local cluster using Spark 2.0.3 trunk 
and Hadoop 3.0 alpha 2 trunk. We run Spark SQL queries with same data size on 
both Erasure Coding and 3-replication.  The test results show that some queries 
has much worse performance on EC compared to 3-replication. After initial 
investigations, we found spark starts one third executors to execute queries on 
EC compared to 3-replication. 

We use query 30 as example, our cluster can totally launch 108 executors. When 
we run the query from 3-replication database, spark will start all 108 
executors to execute the query.  When we run the query from Erasure Coding 
database, spark will launch 108 executors and kill 72 executors due to they’re 
idle, at last there are only 36 executors to execute the query which leads to 
poor performance.

This issues only happens when we enable dynamic allocations mechanism. When we 
disable the dynamic allocations, Spark SQL query on EC has the similar 
performance with on 3-replication.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to