Hi，friends:
I use spark(spark 1.1) sql operate data in hive-0.12, and the job fails when
data is large. So how to tune it ?
spark-defaults.conf:
spark.shuffle.consolidateFiles true
spark.shuffle.manager SORT
spark.akka.threads 4
spark.sql.inMemoryColumnarStorage.compressed
Try to increase the driver memory.
2014-10-28 17:33 GMT+08:00 Zhanfeng Huo huozhanf...@gmail.com:
Hi，friends:
I use spark(spark 1.1) sql operate data in hive-0.12, and the job fails
when data is large. So how to tune it ?
spark-defaults.conf:
spark.shuffle.consolidateFiles true
