bq. the BLAS native cannot be loaded
Have you tried specifying --driver-library-path option ?
Cheers
On Sat, Mar 21, 2015 at 4:42 PM, Xi Shen <davidshe...@gmail.com> wrote:
> Yeah, I think it is harder to troubleshot the properties issues in a IDE.
> But the reason I stick to IDE is because if I use spark-submit, the BLAS
> native cannot be loaded. May be I should open another thread to discuss
> that.
>
> Thanks,
> David
>
> On Sun, 22 Mar 2015 10:38 Xi Shen <davidshe...@gmail.com> wrote:
>
>> In the log, I saw
>>
>> MemoryStorage: MemoryStore started with capacity 6.7GB
>>
>> But I still can not find where to set this storage capacity.
>>
>> On Sat, 21 Mar 2015 20:30 Xi Shen <davidshe...@gmail.com> wrote:
>>
>>> Hi Sean,
>>>
>>> It's getting strange now. If I ran from IDE, my executor memory is
>>> always set to 6.7G, no matter what value I set in code. I have check my
>>> environment variable, and there's no value of 6.7, or 12.5
>>>
>>> Any idea?
>>>
>>> Thanks,
>>> David
>>>
>>> On Tue, 17 Mar 2015 00:35 null <jishnu.prat...@wipro.com> wrote:
>>>
>>>> Hi Xi Shen,
>>>>
>>>> You could set the spark.executor.memory in the code itself . new
>>>> SparkConf()..set("spark.executor.memory", "2g")
>>>>
>>>> Or you can try the -- spark.executor.memory 2g while submitting the jar.
>>>>
>>>>
>>>>
>>>> Regards
>>>>
>>>> Jishnu Prathap
>>>>
>>>>
>>>>
>>>> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com]
>>>> *Sent:* Monday, March 16, 2015 2:06 PM
>>>> *To:* Xi Shen
>>>> *Cc:* user@spark.apache.org
>>>> *Subject:* Re: How to set Spark executor memory?
>>>>
>>>>
>>>>
>>>> By default spark.executor.memory is set to 512m, I'm assuming since you
>>>> are submiting the job using spark-submit and it is not able to override the
>>>> value since you are running in local mode. Can you try it without using
>>>> spark-submit as a standalone project?
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Best Regards
>>>>
>>>>
>>>>
>>>> On Mon, Mar 16, 2015 at 1:52 PM, Xi Shen <davidshe...@gmail.com> wrote:
>>>>
>>>> I set it in code, not by configuration. I submit my jar file to local.
>>>> I am working in my developer environment.
>>>>
>>>>
>>>>
>>>> On Mon, 16 Mar 2015 18:28 Akhil Das <ak...@sigmoidanalytics.com> wrote:
>>>>
>>>> How are you setting it? and how are you submitting the job?
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Best Regards
>>>>
>>>>
>>>>
>>>> On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen <davidshe...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> I have set spark.executor.memory to 2048m, and in the UI "Environment"
>>>> page, I can see this value has been set correctly. But in the "Executors"
>>>> page, I saw there's only 1 executor and its memory is 265.4MB. Very strange
>>>> value. why not 256MB, or just as what I set?
>>>>
>>>>
>>>>
>>>> What am I missing here?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The information contained in this electronic message and any
>>>> attachments to this message are intended for the exclusive use of the
>>>> addressee(s) and may contain proprietary, confidential or privileged
>>>> information. If you are not the intended recipient, you should not
>>>> disseminate, distribute or copy this e-mail. Please notify the sender
>>>> immediately and destroy all copies of this message and any attachments.
>>>> WARNING: Computer viruses can be transmitted via email. The recipient
>>>> should check this email and any attachments for the presence of viruses.
>>>> The company accepts no liability for any damage caused by any virus
>>>> transmitted by this email. www.wipro.com
>>>>
>>>
