)
Or you can try the -- spark.executor.memory 2g while submitting the
jar.
Regards
Jishnu Prathap
*From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
*Sent:* Monday, March 16, 2015 2:06 PM
*To:* Xi Shen
*Cc:* user@spark.apache.org
*Subject:* Re: How to set Spark executor memory
:* Monday, March 16, 2015 2:06 PM
*To:* Xi Shen
*Cc:* user@spark.apache.org
*Subject:* Re: How to set Spark executor memory?
By default spark.executor.memory is set to 512m, I'm assuming since you
are submiting the job using spark-submit and it is not able to override the
value since you
the jar.
Regards
Jishnu Prathap
From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: Monday, March 16, 2015 2:06 PM
To: Xi Shen
Cc: user@spark.apache.org
Subject: Re: How to set Spark executor memory?
By default spark.executor.memory is set to 512m, I'm assuming since you
: How to set Spark executor memory?
By default spark.executor.memory is set to 512m, I'm assuming since you
are submiting the job using spark-submit and it is not able to override the
value since you are running in local mode. Can you try it without using
spark-submit as a standalone project
*Subject:* Re: How to set Spark executor memory?
By default spark.executor.memory is set to 512m, I'm assuming since you
are submiting the job using spark-submit and it is not able to override the
value since you are running in local mode. Can you try it without using
spark-submit as a standalone
the jar.
Regards
Jishnu Prathap
*From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
*Sent:* Monday, March 16, 2015 2:06 PM
*To:* Xi Shen
*Cc:* user@spark.apache.org
*Subject:* Re: How to set Spark executor memory?
By default spark.executor.memory is set to 512m, I'm assuming since
, 2015 2:06 PM
To: Xi Shen
Cc: user@spark.apache.org
Subject: Re: How to set Spark executor memory?
By default spark.executor.memory is set to 512m, I'm assuming since you are
submiting the job using spark-submit and it is not able to override the value
since you are running in local mode. Can you
There are a number of small misunderstandings here.
In the first instance, the executor memory is not actually being set
to 2g and the default of 512m is being used. If you are writing code
to launch an app, then you are trying to duplicate what spark-submit
does, and you don't use spark-submit.
How are you setting it? and how are you submitting the job?
Thanks
Best Regards
On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen davidshe...@gmail.com wrote:
Hi,
I have set spark.executor.memory to 2048m, and in the UI Environment
page, I can see this value has been set correctly. But in the
I set it in code, not by configuration. I submit my jar file to local. I am
working in my developer environment.
On Mon, 16 Mar 2015 18:28 Akhil Das ak...@sigmoidanalytics.com wrote:
How are you setting it? and how are you submitting the job?
Thanks
Best Regards
On Mon, Mar 16, 2015 at
By default spark.executor.memory is set to 512m, I'm assuming since you are
submiting the job using spark-submit and it is not able to override the
value since you are running in local mode. Can you try it without using
spark-submit as a standalone project?
Thanks
Best Regards
On Mon, Mar 16,
Hi Akhil,
Yes, you are right. If I ran the program from IDE as a normal java program,
the executor's memory is increased...but not to 2048m, it is set to
6.7GB...Looks like there's some formula to calculate this value.
Thanks,
David
On Mon, Mar 16, 2015 at 7:36 PM Akhil Das
How much memory are you having on your machine? I think default value is
0.6 of the spark.executor.memory as you can see from here
http://spark.apache.org/docs/1.2.1/configuration.html#execution-behavior.
Thanks
Best Regards
On Mon, Mar 16, 2015 at 2:26 PM, Xi Shen davidshe...@gmail.com wrote:
I set spark.executor.memory to 2048m. If the executor storage memory is
0.6 of executor memory, it should be 2g * 0.6 = 1.2g.
My machine has 56GB memory, and 0.6 of that should be 33.6G...I hate math xD
On Mon, Mar 16, 2015 at 7:59 PM Akhil Das ak...@sigmoidanalytics.com
wrote:
How much
Strange, even i'm having it while running in local mode.
[image: Inline image 1]
I set it as .set(spark.executor.memory, 1g)
Thanks
Best Regards
On Mon, Mar 16, 2015 at 2:43 PM, Xi Shen davidshe...@gmail.com wrote:
I set spark.executor.memory to 2048m. If the executor storage memory
is 0.6
15 matches
Mail list logo