Hey Ken, Yes, you need to create the relationship between the 2 entities to they know about each other.
Looking at the list of hooks for the charm <https://github.com/Archethought/spark-charm/tree/master/hooks> you can see there are 2 hooks named namenode-relation-changed <https://github.com/Archethought/spark-charm/blob/master/hooks/namenode-relation-changed> and resourcemanager-relation-changed <https://github.com/Archethought/spark-charm/blob/master/hooks/resourcemanager-relation-changed> which are related to YARN/Hadoop. Looking deeper in the code, you'll notice they reference a function found in bdutils.py called "setHadoopEnvVar()", which based on its name should set the HADOOP_CONF_DIR. There are 2 relations, so add both of them. Note that I didn't test this myself, but I expect this should fix the problem. If it doesn't please come back to us... Thanks! Sam Best, Samuel -- Samuel Cozannet Cloud, Big Data and IoT Strategy Team Business Development - Cloud and ISV Ecosystem Changing the Future of Cloud Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> / Juju <https://jujucharms.com> samuel.cozan...@canonical.com mob: +33 616 702 389 skype: samnco Twitter: @SaMnCo_23 On Fri, Jan 30, 2015 at 11:51 AM, Ken Williams <ke...@theasi.co> wrote: > > Thanks, Kapil - this works :-) > > I can now run the SparkPi example successfully. > root@ip-172-31-60-53:~# spark-submit --class > org.apache.spark.examples.SparkPi /tmp/spark-examples-1.2.0-hadoop2.4.0.jar > Spark assembly has been built with Hive, including Datanucleus jars on > classpath > 15/01/30 10:29:33 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Pi is roughly 3.14318 > > root@ip-172-31-60-53:~# > > I'm now trying to run the same example with the spark-submit '--master' > option set to either 'yarn-cluster' or 'yarn-client' > but I keep getting the same error : > > root@ip-172-31-60-53:~# spark-submit --class > org.apache.spark.examples.SparkPi --master yarn-client > --num-executors 3 --driver-memory 1g --executor-memory 1g > --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10 > Spark assembly has been built with Hive, including Datanucleus jars on > classpath > Exception in thread "main" java.lang.Exception: When running with master > 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the > environment. > > But on my spark-master/0 machine there is no /etc/hadoop/conf directory. > So what should the HADOOP_CONF_DIR or YARN_CONF_DIR value be ? > Do I need to add a juju relation between spark-master and ... > yarn-hdfs-master to make them aware of each other ? > > Thanks for any help, > > Ken > > > > > > On 28 January 2015 at 19:32, Kapil Thangavelu < > kapil.thangav...@canonical.com> wrote: > >> >> >> On Wed, Jan 28, 2015 at 1:54 PM, Ken Williams <ke...@theasi.co> wrote: >> >>> >>> Hi Sam/Amir, >>> >>> I've been able to 'juju ssh spark-master/0' and I successfully ran >>> the two >>> simple examples for pyspark and spark-shell, >>> >>> ./bin/pyspark >>> >>> sc.parallelize(range(1000)).count() >>> 1000 >>> >>> ./bin/spark-shell >>> scala> sc.parallelize(1 to 1000).count() >>> 1000 >>> >>> >>> Now I want to run some of the spark examples in the spark-exampes*.jar >>> file, which I have on my local machine. How do I copy the jar file from >>> my local machine to the AWS machine ? >>> >>> I have tried 'scp' and 'juju scp' from the local command-line but both >>> fail (below), >>> >>> root@adminuser:~# scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar >>> ubuntu@ip-172-31-59:/tmp >>> ssh: Could not resolve hostname ip-172-31-59: Name or service not known >>> lost connection >>> root@adminuser:~# juju scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar >>> ubuntu@ip-172-31-59:/tmp >>> ERROR exit status 1 (nc: getaddrinfo: Name or service not known) >>> >>> Any ideas ? >>> >> >> juju scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar spark-master/0:/tmp >> >>> >>> Ken >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On 28 January 2015 at 17:29, Samuel Cozannet < >>> samuel.cozan...@canonical.com> wrote: >>> >>>> Glad it worked! >>>> >>>> I'll make a merge request to the upstream so that it works natively >>>> from the store asap. >>>> >>>> Thanks for catching that! >>>> Samuel >>>> >>>> Best, >>>> Samuel >>>> >>>> -- >>>> Samuel Cozannet >>>> Cloud, Big Data and IoT Strategy Team >>>> Business Development - Cloud and ISV Ecosystem >>>> Changing the Future of Cloud >>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> / >>>> Juju <https://jujucharms.com> >>>> samuel.cozan...@canonical.com >>>> mob: +33 616 702 389 >>>> skype: samnco >>>> Twitter: @SaMnCo_23 >>>> >>>> On Wed, Jan 28, 2015 at 6:15 PM, Ken Williams <ke...@theasi.co> wrote: >>>> >>>>> >>>>> Hi Sam (and Maarten), >>>>> >>>>> Cloning Spark 1.2.0 from github seems to have worked! >>>>> I can install the Spark examples afterwards. >>>>> >>>>> Thanks for all your help! >>>>> >>>>> Yes - Andrew and Angie both say 'hi' :-) >>>>> >>>>> Best Regards, >>>>> >>>>> Ken >>>>> >>>>> >>>>> On 28 January 2015 at 16:43, Samuel Cozannet < >>>>> samuel.cozan...@canonical.com> wrote: >>>>> >>>>>> Hey Ken, >>>>>> >>>>>> So I had a closer look to your Spark problem and found out what went >>>>>> wrong. >>>>>> >>>>>> The charm available on the charmstore is trying to download Spark >>>>>> 1.0.2, and the versions available on the Apache website are 1.1.0, 1.1.1 >>>>>> and 1.2.0. >>>>>> >>>>>> There is another version of the charm available on GitHub that >>>>>> actually will deploy 1.2.0 >>>>>> >>>>>> 1. On your computer, the below folders & get there: >>>>>> >>>>>> cd ~ >>>>>> mkdir charms >>>>>> mkdir charms/trusty >>>>>> cd charms/trusty >>>>>> >>>>>> 2. Branch the Spark charm. >>>>>> >>>>>> git clone https://github.com/Archethought/spark-charm spark >>>>>> >>>>>> 3. Deploy Spark from local repository >>>>>> >>>>>> juju deploy --repository=~/charms local:trusty/spark spark-master >>>>>> juju deploy --repository=~/charms local:trusty/spark spark-slave >>>>>> juju add-relation spark-master:master spark-slave:slave >>>>>> >>>>>> Worked on AWS for me just minutes ago. Let me know how it goes for >>>>>> you. Note that this version of the charm does NOT install the Spark >>>>>> examples. The files are present though, so you'll find them in >>>>>> /var/lib/juju/agents/unit-spark-master-0/charm/files/archive >>>>>> >>>>>> Hope that helps... >>>>>> Let me know if it works for you! >>>>>> >>>>>> Best, >>>>>> Sam >>>>>> >>>>>> >>>>>> Best, >>>>>> Samuel >>>>>> >>>>>> -- >>>>>> Samuel Cozannet >>>>>> Cloud, Big Data and IoT Strategy Team >>>>>> Business Development - Cloud and ISV Ecosystem >>>>>> Changing the Future of Cloud >>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> / >>>>>> Juju <https://jujucharms.com> >>>>>> samuel.cozan...@canonical.com >>>>>> mob: +33 616 702 389 >>>>>> skype: samnco >>>>>> Twitter: @SaMnCo_23 >>>>>> >>>>>> On Wed, Jan 28, 2015 at 4:44 PM, Ken Williams <ke...@theasi.co> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> Hi folks, >>>>>>> >>>>>>> I'm completely new to juju so any help is appreciated. >>>>>>> >>>>>>> I'm trying to create a hadoop/analytics-type platform. >>>>>>> >>>>>>> I've managed to install the 'data-analytics-with-sql-like' bundle >>>>>>> (using this command) >>>>>>> >>>>>>> juju quickstart >>>>>>> bundle:data-analytics-with-sql-like/data-analytics-with-sql-like >>>>>>> >>>>>>> This is very impressive, and gives me virtually everything that I >>>>>>> want >>>>>>> (hadoop, hive, etc) - but I also need Spark. >>>>>>> >>>>>>> The Spark charm (http://manage.jujucharms.com/~asanjar/trusty/spark >>>>>>> ) >>>>>>> and bundle ( >>>>>>> http://manage.jujucharms.com/bundle/~asanjar/spark/spark-cluster) >>>>>>> however do not seem stable or available and I can't figure out how >>>>>>> to install them. >>>>>>> >>>>>>> Should I just download and install the Spark tar-ball on the nodes >>>>>>> in my AWS cluster, or is there a better way to do this ? >>>>>>> >>>>>>> Thanks in advance, >>>>>>> >>>>>>> Ken >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Juju mailing list >>>>>>> Juju@lists.ubuntu.com >>>>>>> Modify settings or unsubscribe at: >>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> -- >>> Juju mailing list >>> Juju@lists.ubuntu.com >>> Modify settings or unsubscribe at: >>> https://lists.ubuntu.com/mailman/listinfo/juju >>> >>> >> >
-- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
