Sabyasachi Nayak created SPARK-17457:
----------------------------------------
Summary: Spark SQL shows poor performance for group by on multiple
columns
Key: SPARK-17457
URL: https://issues.apache.org/jira/browse/SPARK-17457
Project: Spark
Issue Type: Improvement
Affects Versions: 1.4.0
Reporter: Sabyasachi Nayak
In one of the use case when we are running one hive query with Tez it is taking
45 mnts.But the same query when I am running in Spark SQL using hivecontext it
is taking more than 2 hours.This query has no joins only group by and sort by
on multiple columns.
spark-submit --class DataLoadingSpark --master yarn --deploy-mode client
--num-executors 60 --executor-memory 16G --driver-memory 4G --executor-cores 5
--conf spark.yarn.executor.memoryOverhead=2048 --conf
spark.shuffle.consolidateFiles=true --conf spark.shuffle.memoryFraction=0.5
--conf spark.storage.memoryFraction=.1 --conf spark.io.compression.codec=lzf
--conf spark.driver.extraJavaOptions="-XX:MaxPermSize=1024m -XX:PermSize=512m
-Dhdp.version=2.3.2.0-2950" --conf spark.shuffle.blockTransferService=nio
DataLoadingSpark.jar --inputFile basket_txn.
Spark version - 1.4.0
HDP 2.3.2.0-2950
50 node cluster 1100 Vcores
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
