OK, I actually got the answer days ago from StackOverflow, but I did not
check it :(
When running in "local" mode, to set the executor memory
- when using spark-submit, use "--driver-memory"
- when running as a Java application, like executing from IDE, set the
"-Xmx" vm option
Thanks,
David
On Sun, Mar 22, 2015 at 2:10 PM Ted Yu <yuzhih...@gmail.com> wrote:
> bq. the BLAS native cannot be loaded
>
> Have you tried specifying --driver-library-path option ?
>
> Cheers
>
> On Sat, Mar 21, 2015 at 4:42 PM, Xi Shen <davidshe...@gmail.com> wrote:
>
>> Yeah, I think it is harder to troubleshot the properties issues in a IDE.
>> But the reason I stick to IDE is because if I use spark-submit, the BLAS
>> native cannot be loaded. May be I should open another thread to discuss
>> that.
>>
>> Thanks,
>> David
>>
>> On Sun, 22 Mar 2015 10:38 Xi Shen <davidshe...@gmail.com> wrote:
>>
>>> In the log, I saw
>>>
>>> MemoryStorage: MemoryStore started with capacity 6.7GB
>>>
>>> But I still can not find where to set this storage capacity.
>>>
>>> On Sat, 21 Mar 2015 20:30 Xi Shen <davidshe...@gmail.com> wrote:
>>>
>>>> Hi Sean,
>>>>
>>>> It's getting strange now. If I ran from IDE, my executor memory is
>>>> always set to 6.7G, no matter what value I set in code. I have check my
>>>> environment variable, and there's no value of 6.7, or 12.5
>>>>
>>>> Any idea?
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>> On Tue, 17 Mar 2015 00:35 null <jishnu.prat...@wipro.com> wrote:
>>>>
>>>>> Hi Xi Shen,
>>>>>
>>>>> You could set the spark.executor.memory in the code itself . new
>>>>> SparkConf()..set("spark.executor.memory", "2g")
>>>>>
>>>>> Or you can try the -- spark.executor.memory 2g while submitting the
>>>>> jar.
>>>>>
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>> Jishnu Prathap
>>>>>
>>>>>
>>>>>
>>>>> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
>>>>> *Sent:* Monday, March 16, 2015 2:06 PM
>>>>> *To:* Xi Shen
>>>>> *Cc:* user@spark.apache.org
>>>>> *Subject:* Re: How to set Spark executor memory?
>>>>>
>>>>>
>>>>>
>>>>> By default spark.executor.memory is set to 512m, I'm assuming since
>>>>> you are submiting the job using spark-submit and it is not able to
>>>>> override
>>>>> the value since you are running in local mode. Can you try it without
>>>>> using
>>>>> spark-submit as a standalone project?
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Best Regards
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 16, 2015 at 1:52 PM, Xi Shen <davidshe...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> I set it in code, not by configuration. I submit my jar file to local.
>>>>> I am working in my developer environment.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, 16 Mar 2015 18:28 Akhil Das <ak...@sigmoidanalytics.com>
>>>>> wrote:
>>>>>
>>>>> How are you setting it? and how are you submitting the job?
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Best Regards
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen <davidshe...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> I have set spark.executor.memory to 2048m, and in the UI "Environment"
>>>>> page, I can see this value has been set correctly. But in the "Executors"
>>>>> page, I saw there's only 1 executor and its memory is 265.4MB. Very
>>>>> strange
>>>>> value. why not 256MB, or just as what I set?
>>>>>
>>>>>
>>>>>
>>>>> What am I missing here?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The information contained in this electronic message and any
>>>>> attachments to this message are intended for the exclusive use of the
>>>>> addressee(s) and may contain proprietary, confidential or privileged
>>>>> information. If you are not the intended recipient, you should not
>>>>> disseminate, distribute or copy this e-mail. Please notify the sender
>>>>> immediately and destroy all copies of this message and any attachments.
>>>>> WARNING: Computer viruses can be transmitted via email. The recipient
>>>>> should check this email and any attachments for the presence of viruses.
>>>>> The company accepts no liability for any damage caused by any virus
>>>>> transmitted by this email. www.wipro.com
>>>>>
>>>>
>
