you are using YARN?
>
> -Pat
>
> From: KhajaAsmath Mohammed <mdkhajaasm...@gmail.com>
> Date: Friday, August 18, 2017 at 5:30 AM
> To: Pralabh Kumar <pralabhku...@gmail.com>
> Cc: "user @spark" <user@spark.apache.org>
> Subject: Re: GC overhead ex
: KhajaAsmath Mohammed <mdkhajaasm...@gmail.com>
Date: Friday, August 18, 2017 at 5:30 AM
To: Pralabh Kumar <pralabhku...@gmail.com>
Cc: "user @spark" <user@spark.apache.org>
Subject: Re: GC overhead exceeded
It is just a sql from hive table with transformation if adding
It is just a sql from hive table with transformation if adding 10 more columns
calculated for currency. Input size for this query is 2 months which has around
450gb data.
I added persist but it didn't help. Also the executor memory is 8g . Any
suggestions please ?
Sent from my iPhone
> On
what's is your exector memory , please share the code also
On Fri, Aug 18, 2017 at 10:06 AM, KhajaAsmath Mohammed <
mdkhajaasm...@gmail.com> wrote:
>
> HI,
>
> I am getting below error when running spark sql jobs. This error is thrown
> after running 80% of tasks. any solution?
>
>
HI,
I am getting below error when running spark sql jobs. This error is thrown
after running 80% of tasks. any solution?
spark.storage.memoryFraction=0.4
spark.sql.shuffle.partitions=2000
spark.default.parallelism=100
#spark.eventLog.enabled=false
#spark.scheduler.revive.interval=1s
