Hi，friends:
I use spark(spark 1.1) sql operate data in hive-0.12, and the job fails when
data is large. So how to tune it ?
spark-defaults.conf:
spark.shuffle.consolidateFiles true
spark.shuffle.manager SORT
spark.akka.threads 4
spark.sql.inMemoryColumnarStorage.compressed true
Try to increase the driver memory.
2014-10-28 17:33 GMT+08:00 Zhanfeng Huo :
> Hi，friends:
>
> I use spark(spark 1.1) sql operate data in hive-0.12, and the job fails
> when data is large. So how to tune it ?
>
> spark-defaults.conf:
>
> spark.shuffle.consolidateFiles true
> spark.shuffle
It works， thanks very much
Zhanfeng Huo
From: Yanbo Liang
Date: 2014-10-28 18:50
To: Zhanfeng Huo
CC: user
Subject: Re: SparkSql OutOfMemoryError
Try to increase the driver memory.
2014-10-28 17:33 GMT+08:00 Zhanfeng Huo :
Hi，friends:
I use spark(spark 1.1) sql operate data in hive-0.12
