Re: Using YARN w/o HDFS
chang your fs.defaultFS to point to local file system and have a try On Wed, Jun 21, 2017 at 4:50 PM, Alaa Zubaidi (PDF) wrote: > Hi, > > Can we run Spark on YARN with out installing HDFS? > If yes, where would HADOOP_CONF_DIR point to? > > Regards, > > *This message may contain confidential and privileged information. If it > has been sent to you in error, please reply to advise the sender of the > error and then immediately permanently delete it and all attachments to it > from your systems. If you are not the intended recipient, do not read, > copy, disclose or otherwise use this message or any attachments to it. The > sender disclaims any liability for such unauthorized use. PLEASE NOTE that > all incoming e-mails sent to PDF e-mail accounts will be archived and may > be scanned by us and/or by external service providers to detect and prevent > threats to our systems, investigate illegal or inappropriate behavior, > and/or eliminate unsolicited promotional e-mails (“spam”). If you have any > concerns about this process, please contact us at * > *legal.departm...@pdf.com* *.*
Re: Spark on Raspberry Pi?
Here is the POC of cubieboards hadoop cluster performance and power consumption. I suggest cubietrunk would be a choice since cubieboard2 only has 100Mbps ethernet. http://www.slideshare.net/airbots/hadoop-mapreduce-performance-study-on-arm-cluster On Thu, Sep 11, 2014 at 12:50 PM, Chanwit Kaewkasi wrote: > We've found that Raspberry Pi is not enough for Hadoop/Spark mainly > because the memory consumption. What we've built is a cluster form > with 22 Cubieboards, each contains 1 GB RAM. > > Best regards, > > -chanwit > > -- > Chanwit Kaewkasi > linkedin.com/in/chanwit > > > On Thu, Sep 11, 2014 at 8:04 PM, Sandeep Singh > wrote: > > Has anyone tried using Raspberry Pi for Spark? How efficient is it to use > > around 10 Pi's for local testing env ? > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Raspberry-Pi-tp13965.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > - > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Fwd: Spark on Raspberry Pi?
Pi's bus speed, memory size and access speed, and processing ability are limited. The only benefit could be the power consumption. On Thu, Sep 11, 2014 at 8:04 AM, Sandeep Singh wrote: > Has anyone tried using Raspberry Pi for Spark? How efficient is it to use > around 10 Pi's for local testing env ? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Raspberry-Pi-tp13965.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: CUDA in spark, especially in MLlib?
JCUDA can let you do that in Java http://www.jcuda.org On Wed, Aug 27, 2014 at 1:48 AM, Antonio Jesus Navarro < ajnava...@stratio.com> wrote: > Maybe this would interest you: > > CPU and GPU-accelerated Machine Learning Library: > > https://github.com/BIDData/BIDMach > > > 2014-08-27 4:08 GMT+02:00 Matei Zaharia : > > You should try to find a Java-based library, then you can call it from >> Scala. >> >> Matei >> >> On August 26, 2014 at 6:58:11 PM, Wei Tan (w...@us.ibm.com) wrote: >> >> Hi I am trying to find a CUDA library in Scala, to see if some matrix >> manipulation in MLlib can be sped up. >> >> I googled a few but found no active projects on Scala+CUDA. Python is >> supported by CUDA though. Any suggestion on whether this idea makes any >> sense? >> >> Best regards, >> Wei >> >> >
