What is the size of your data, size of the cluster, are you using
spark-submit or an IDE, what spark version are you using?
Try spark-submit and increase the memory of the driver or the executors.
a.
On 22/3/19 17:19, KhajaAsmath Mohammed wrote:
Hi,
I am getting the below exception when
Hi,
I am getting the below exception when using Spark Kmeans. Any solutions
from the experts. Would be really helpful.
val kMeans = new KMeans().setK(reductionCount).setMaxIter(30)
val kMeansModel = kMeans.fit(df)
Error is occured when calling kmeans.fit
Exception in thread "main"
Hello,
I am trying to debug a PySpark program and quite frankly, I am stumped.
I see the following error in the logs. I verified the input parameters - all
appear to be in order. Driver and executors appear to be proper - about 3MB of
7GB being used on each node.
I do see that the DAG plan that
te all the
> productionName and product catagory together, without even consider removing
> duplication.
>
> But both query still should push similar records count per partition, but
> with much of different volume size of data.
>
> Yong
>
> Subject: Re: Java Heap
the stuff without String operations?
>
> On 24 September 2015 at 10:11, java8964 <java8...@hotmail.com
> <mailto:java8...@hotmail.com>> wrote:
> Try to increase partitions count, that will make each partition has less data.
>
> Yong
>
> Subject: Re: Java Heap Space
rtitioner of "userid".
Can you show us the query after you add "regex" and "concatenation"?
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 15:34:48 +0300
CC: user@spark.apache.org
To: jingyu.zh...@news.com.au; java8...@hotma
ns simply depends on the Hash partitioner of "userid".
>
> Can you show us the query after you add "regex" and "concatenation"?
>
> Yong
>
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 15:34:48 +0300
&
ion.
>
> But both query still should push similar records count per partition, but
> with much of different volume size of data.
>
> Yong
>
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 18:56:51 +0300
> CC: jingyu.
.
But both query still should push similar records count per partition, but with
much of different volume size of data.
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 18:56:51 +0300
CC: jingyu.zh...@news.com.au; user@spark.apache.org
To: java8...@hotmail.com
Yes
ans that one
> partition having way more data than the rest of partitions.
>
> Yong
>
> From: yu...@useinsider.com
> Subject: Java Heap Space Error
> Date: Wed, 23 Sep 2015 23:07:17 +0300
> To: user@spark.apache.org
>
> What can cause this issue in the attached pic
Try to increase partitions count, that will make each partition has less data.
Yong
Subject: Re: Java Heap Space Error
From: yu...@useinsider.com
Date: Thu, 24 Sep 2015 00:32:47 +0300
CC: user@spark.apache.org
To: java8...@hotmail.com
Yes, it’s possible. I use S3 as data source. My external
ta.
>
> Yong
>
> ------
> Subject: Re: Java Heap Space Error
> From: yu...@useinsider.com
> Date: Thu, 24 Sep 2015 00:32:47 +0300
> CC: user@spark.apache.org
> To: java8...@hotmail.com
>
>
> Yes, it’s possible. I use S3 as data source. My extern
In my dev-test env .I have 3 virtual machines ,every machine have 12G
memory,8 cpu core.
Here is spark-defaults.conf,and spark-env.sh.Maybe some config is not right.
I run this command :*spark-submit --master yarn-client --driver-memory 7g
--executor-memory 6g /home/hadoop/spark/main.py*
Any help?please.
Help me do a right configure.
李铖 lidali...@gmail.com于2015年4月7日星期二写道：
In my dev-test env .I have 3 virtual machines ,every machine have 12G
memory,8 cpu core.
Here is spark-defaults.conf,and spark-env.sh.Maybe some config is not
right.
I run this command :*spark-submit
-Dspark.deploy.defaultCores=. When you have not
enough memory, reduce the concurrency of your executor, it will lower the
memory requirement, with running in a slower speed.
Yong
Date: Wed, 8 Apr 2015 04:57:22 +0800
Subject: Re: 'Java heap space' error occured when query 4G data file from HDFS
From: lidali
in a slower speed.
Yong
--
Date: Wed, 8 Apr 2015 04:57:22 +0800
Subject: Re: 'Java heap space' error occured when query 4G data file from
HDFS
From: lidali...@gmail.com
To: user@spark.apache.org
Any help?please.
Help me do a right configure.
李铖 lidali
I am facing same issue, posted a new thread. Please respond.
On Wed, Jul 9, 2014 at 1:56 AM, Rahul Bhojwani rahulbhojwani2...@gmail.com
wrote:
Hi,
My code was running properly but then it suddenly gave this error. Can you
just put some light on it.
###
0 KB, free: 38.7
I'm trying to process a large dataset, mapping/filtering works ok, but
as long as I try to reduceByKey, I get out of memory errors:
http://pastebin.com/70M5d0Bn
Any ideas how I can fix that?
Thanks.
-
To unsubscribe, e-mail:
Hi Kane-
http://spark.apache.org/docs/latest/tuning.html has excellent information that
may be helpful. In particular increasing the number of tasks may help, as well
as confirming that you don’t have more data than you're expecting landing on a
key.
Also, if you are using spark 1.2.0,
What operation are you performing before doing the saveAsTextFile? If you
are doing a groupBy/sortBy/mapPartition/reduceByKey operations then you can
specify the number of partitions. We were facing these kind of problems and
specifying the correct partition solved the issue.
Thanks
Best Regards
Hello,
My job keeps failing on saveAsTextFile stage (frustrating after a 3 hour
run) with an OOM exception. The log is below. I'm running the job on an
input of ~8Tb gzipped JSON files, executing on 15 m3.xlarge instances.
Executor is given 13Gb memory, and I'm setting two custom preferences in
21 matches
Mail list logo
