What is the size of your data, size of the cluster, are you using
spark-submit or an IDE, what spark version are you using?
Try spark-submit and increase the memory of the driver or the executors.
a.
On 22/3/19 17:19, KhajaAsmath Mohammed wrote:
Hi,
I am getting the below exception when usi
y what you want, to consolidate all the
> productionName and product catagory together, without even consider removing
> duplication.
>
> But both query still should push similar records count per partition, but
> with much of different volume size of data.
>
> Yong
>
> S
ving
> duplication.
>
> But both query still should push similar records count per partition, but
> with much of different volume size of data.
>
> Yong
>
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 18:56:51 +0300
&
r, without even consider removing duplication.
But both query still should push similar records count per partition, but with
much of different volume size of data.
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 18:56:51 +0300
CC: jingyu.zh...@news.com.au; u
e is huge and the others are very small.
>
>
> ——
> So how can i balance this shuffle read size between partitions?
>
>
> On 24 Sep 2015, at 03:35, Zhang, Jingyu <mailto:jingyu.zh...@news.com.au>> wrote:
>
> Is you sql works if do not runs a regex on
simply depends on the Hash partitioner of "userid".
Can you show us the query after you add "regex" and "concatenation"?
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 15:34:48 +0300
CC: user@spark.apache.org
To: jingyu.zh...@
the stuff without String operations?
>
> On 24 September 2015 at 10:11, java8964 <mailto:java8...@hotmail.com>> wrote:
> Try to increase partitions count, that will make each partition has less data.
>
> Yong
>
> Subject: Re: Java Heap Space Error
> From: yu...@useinsi
t;
> Yong
>
> ------
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 00:32:47 +0300
> CC: user@spark.apache.org
> To: java8...@hotmail.com
>
>
> Yes, it’s possible. I use S3 as data source. My external tables has
> partition
Try to increase partitions count, that will make each partition has less data.
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 00:32:47 +0300
CC: user@spark.apache.org
To: java8...@hotmail.com
Yes, it’s possible. I use S3 as data source. My external
Yes, it’s possible. I use S3 as data source. My external tables has
partitioned. Belowed task is 193/200. Job has 2 stages and its 193. task of 200
in 2.stage because of sql.shuffle.partitions.
How can i avoid this situation, this is my query:
select userid,concat_ws(' ',collect_list(concat_ws
ecutor, it
>> will lower the memory requirement, with running in a slower speed.
>>
>> Yong
>>
>> --
>> Date: Wed, 8 Apr 2015 04:57:22 +0800
>> Subject: Re: 'Java heap space' error occured when query 4G data file from
oncurrency of your executor, it
> will lower the memory requirement, with running in a slower speed.
>
> Yong
>
> ------
> Date: Wed, 8 Apr 2015 04:57:22 +0800
> Subject: Re: 'Java heap space' error occured when query 4G data file from
&
ower the
cores for executor by set "-Dspark.deploy.defaultCores=". When you have not
enough memory, reduce the concurrency of your executor, it will lower the
memory requirement, with running in a slower speed.
Yong
Date: Wed, 8 Apr 2015 04:57:22 +0800
Subject: Re: 'Java heap space' error
Any help?please.
Help me do a right configure.
李铖 于2015年4月7日星期二写道：
> In my dev-test env .I have 3 virtual machines ,every machine have 12G
> memory,8 cpu core.
>
> Here is spark-defaults.conf,and spark-env.sh.Maybe some config is not
> right.
>
> I run this command :*spark-submit --master yarn-
14 matches
Mail list logo
