@spark.apache.org
Subject: RE: --driver-memory parameter doesn't work for spark-submmit on yarn?
Shuai,
What did ps aux | grep spark-submit reveal?
When you compare using _JAVA_OPTIONS and without using it, where do you see the
difference?
Thanks
Christopher
-Original Message-
From
Once you submit the job do a ps aux | grep spark-submit and see how much is
the heap space allocated to the process (the -Xmx params), if you are
seeing a lower value you could try increasing it yourself with:
export _JAVA_OPTIONS=-Xmx5g
Thanks
Best Regards
On Wed, Apr 1, 2015 at 1:57 AM, Shuai
[mailto:ak...@sigmoidanalytics.com]
Sent: Wednesday, April 01, 2015 2:40 AM
To: Shuai Zheng
Cc: user@spark.apache.org
Subject: Re: --driver-memory parameter doesn't work for spark-submmit on yarn?
Once you submit the job do a ps aux | grep spark-submit and see how much is the
heap space
parameter doesn't work for spark-submmit on
yarn?
Once you submit the job do a ps aux | grep spark-submit and see how much is
the heap space allocated to the process (the -Xmx params), if you are seeing
a lower value you could try increasing it yourself with:
export _JAVA_OPTIONS=-Xmx5g
,
Shuai
-Original Message-
From: Sean Owen [mailto:so...@cloudera.com]
Sent: Wednesday, April 01, 2015 10:51 AM
To: Shuai Zheng
Cc: Akhil Das; user@spark.apache.org
Subject: Re: --driver-memory parameter doesn't work for spark-submmit on yarn?
I feel like I recognize that problem, and it's
Owen'
Cc: 'Akhil Das'; user@spark.apache.org
Subject: RE: --driver-memory parameter doesn't work for spark-submmit on yarn?
Nice.
But when my case shows that even I use Yarn-Client, I have same issue. I do
verify it several times.
And I am running 1.3.0 on EMR (use the version dispatch
Hi All,
Below is the my shell script:
/home/hadoop/spark/bin/spark-submit --driver-memory=5G --executor-memory=40G
--master yarn-client --class com.***.FinancialEngineExecutor
/home/hadoop/lib/my.jar s3://bucket/vriscBatchConf.properties
My driver will load some resources and then
