bq. the BLAS native cannot be loaded Have you tried specifying --driver-library-path option ?
Cheers On Sat, Mar 21, 2015 at 4:42 PM, Xi Shen <davidshe...@gmail.com> wrote: > Yeah, I think it is harder to troubleshot the properties issues in a IDE. > But the reason I stick to IDE is because if I use spark-submit, the BLAS > native cannot be loaded. May be I should open another thread to discuss > that. > > Thanks, > David > > On Sun, 22 Mar 2015 10:38 Xi Shen <davidshe...@gmail.com> wrote: > >> In the log, I saw >> >> MemoryStorage: MemoryStore started with capacity 6.7GB >> >> But I still can not find where to set this storage capacity. >> >> On Sat, 21 Mar 2015 20:30 Xi Shen <davidshe...@gmail.com> wrote: >> >>> Hi Sean, >>> >>> It's getting strange now. If I ran from IDE, my executor memory is >>> always set to 6.7G, no matter what value I set in code. I have check my >>> environment variable, and there's no value of 6.7, or 12.5 >>> >>> Any idea? >>> >>> Thanks, >>> David >>> >>> On Tue, 17 Mar 2015 00:35 null <jishnu.prat...@wipro.com> wrote: >>> >>>> Hi Xi Shen, >>>> >>>> You could set the spark.executor.memory in the code itself . new >>>> SparkConf()..set("spark.executor.memory", "2g") >>>> >>>> Or you can try the -- spark.executor.memory 2g while submitting the jar. >>>> >>>> >>>> >>>> Regards >>>> >>>> Jishnu Prathap >>>> >>>> >>>> >>>> *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com] >>>> *Sent:* Monday, March 16, 2015 2:06 PM >>>> *To:* Xi Shen >>>> *Cc:* user@spark.apache.org >>>> *Subject:* Re: How to set Spark executor memory? >>>> >>>> >>>> >>>> By default spark.executor.memory is set to 512m, I'm assuming since you >>>> are submiting the job using spark-submit and it is not able to override the >>>> value since you are running in local mode. Can you try it without using >>>> spark-submit as a standalone project? >>>> >>>> >>>> Thanks >>>> >>>> Best Regards >>>> >>>> >>>> >>>> On Mon, Mar 16, 2015 at 1:52 PM, Xi Shen <davidshe...@gmail.com> wrote: >>>> >>>> I set it in code, not by configuration. I submit my jar file to local. >>>> I am working in my developer environment. >>>> >>>> >>>> >>>> On Mon, 16 Mar 2015 18:28 Akhil Das <ak...@sigmoidanalytics.com> wrote: >>>> >>>> How are you setting it? and how are you submitting the job? >>>> >>>> >>>> Thanks >>>> >>>> Best Regards >>>> >>>> >>>> >>>> On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen <davidshe...@gmail.com> >>>> wrote: >>>> >>>> Hi, >>>> >>>> >>>> >>>> I have set spark.executor.memory to 2048m, and in the UI "Environment" >>>> page, I can see this value has been set correctly. But in the "Executors" >>>> page, I saw there's only 1 executor and its memory is 265.4MB. Very strange >>>> value. why not 256MB, or just as what I set? >>>> >>>> >>>> >>>> What am I missing here? >>>> >>>> >>>> >>>> >>>> >>>> Thanks, >>>> >>>> David >>>> >>>> >>>> >>>> >>>> >>>> >>>> The information contained in this electronic message and any >>>> attachments to this message are intended for the exclusive use of the >>>> addressee(s) and may contain proprietary, confidential or privileged >>>> information. If you are not the intended recipient, you should not >>>> disseminate, distribute or copy this e-mail. Please notify the sender >>>> immediately and destroy all copies of this message and any attachments. >>>> WARNING: Computer viruses can be transmitted via email. The recipient >>>> should check this email and any attachments for the presence of viruses. >>>> The company accepts no liability for any damage caused by any virus >>>> transmitted by this email. www.wipro.com >>>> >>>