Hi Sam/Amir,
I've been able to 'juju ssh spark-master/0' and I successfully ran the
two
simple examples for pyspark and spark-shell,
./bin/pyspark
>>> sc.parallelize(range(1000)).count()
1000
./bin/spark-shell
scala> sc.parallelize(1 to 1000).count()
1000
Now I want to run some of the spark examples in the spark-exampes*.jar
file, which I have on my local machine. How do I copy the jar file from
my local machine to the AWS machine ?
I have tried 'scp' and 'juju scp' from the local command-line but both fail
(below),
root@adminuser:~# scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
ubuntu@ip-172-31-59:/tmp
ssh: Could not resolve hostname ip-172-31-59: Name or service not known
lost connection
root@adminuser:~# juju scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
ubuntu@ip-172-31-59:/tmp
ERROR exit status 1 (nc: getaddrinfo: Name or service not known)
Any ideas ?
Ken
On 28 January 2015 at 17:29, Samuel Cozannet <samuel.cozan...@canonical.com>
wrote:
> Glad it worked!
>
> I'll make a merge request to the upstream so that it works natively from
> the store asap.
>
> Thanks for catching that!
> Samuel
>
> Best,
> Samuel
>
> --
> Samuel Cozannet
> Cloud, Big Data and IoT Strategy Team
> Business Development - Cloud and ISV Ecosystem
> Changing the Future of Cloud
> Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> /
> Juju <https://jujucharms.com>
> samuel.cozan...@canonical.com
> mob: +33 616 702 389
> skype: samnco
> Twitter: @SaMnCo_23
>
> On Wed, Jan 28, 2015 at 6:15 PM, Ken Williams <ke...@theasi.co> wrote:
>
>>
>> Hi Sam (and Maarten),
>>
>> Cloning Spark 1.2.0 from github seems to have worked!
>> I can install the Spark examples afterwards.
>>
>> Thanks for all your help!
>>
>> Yes - Andrew and Angie both say 'hi' :-)
>>
>> Best Regards,
>>
>> Ken
>>
>>
>> On 28 January 2015 at 16:43, Samuel Cozannet <
>> samuel.cozan...@canonical.com> wrote:
>>
>>> Hey Ken,
>>>
>>> So I had a closer look to your Spark problem and found out what went
>>> wrong.
>>>
>>> The charm available on the charmstore is trying to download Spark 1.0.2,
>>> and the versions available on the Apache website are 1.1.0, 1.1.1 and
>>> 1.2.0.
>>>
>>> There is another version of the charm available on GitHub that actually
>>> will deploy 1.2.0
>>>
>>> 1. On your computer, the below folders & get there:
>>>
>>> cd ~
>>> mkdir charms
>>> mkdir charms/trusty
>>> cd charms/trusty
>>>
>>> 2. Branch the Spark charm.
>>>
>>> git clone https://github.com/Archethought/spark-charm spark
>>>
>>> 3. Deploy Spark from local repository
>>>
>>> juju deploy --repository=~/charms local:trusty/spark spark-master
>>> juju deploy --repository=~/charms local:trusty/spark spark-slave
>>> juju add-relation spark-master:master spark-slave:slave
>>>
>>> Worked on AWS for me just minutes ago. Let me know how it goes for you.
>>> Note that this version of the charm does NOT install the Spark examples.
>>> The files are present though, so you'll find them in
>>> /var/lib/juju/agents/unit-spark-master-0/charm/files/archive
>>>
>>> Hope that helps...
>>> Let me know if it works for you!
>>>
>>> Best,
>>> Sam
>>>
>>>
>>> Best,
>>> Samuel
>>>
>>> --
>>> Samuel Cozannet
>>> Cloud, Big Data and IoT Strategy Team
>>> Business Development - Cloud and ISV Ecosystem
>>> Changing the Future of Cloud
>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> /
>>> Juju <https://jujucharms.com>
>>> samuel.cozan...@canonical.com
>>> mob: +33 616 702 389
>>> skype: samnco
>>> Twitter: @SaMnCo_23
>>>
>>> On Wed, Jan 28, 2015 at 4:44 PM, Ken Williams <ke...@theasi.co> wrote:
>>>
>>>>
>>>> Hi folks,
>>>>
>>>> I'm completely new to juju so any help is appreciated.
>>>>
>>>> I'm trying to create a hadoop/analytics-type platform.
>>>>
>>>> I've managed to install the 'data-analytics-with-sql-like' bundle
>>>> (using this command)
>>>>
>>>> juju quickstart
>>>> bundle:data-analytics-with-sql-like/data-analytics-with-sql-like
>>>>
>>>> This is very impressive, and gives me virtually everything that I want
>>>> (hadoop, hive, etc) - but I also need Spark.
>>>>
>>>> The Spark charm (http://manage.jujucharms.com/~asanjar/trusty/spark)
>>>> and bundle (
>>>> http://manage.jujucharms.com/bundle/~asanjar/spark/spark-cluster)
>>>> however do not seem stable or available and I can't figure out how to
>>>> install them.
>>>>
>>>> Should I just download and install the Spark tar-ball on the nodes
>>>> in my AWS cluster, or is there a better way to do this ?
>>>>
>>>> Thanks in advance,
>>>>
>>>> Ken
>>>>
>>>>
>>>> --
>>>> Juju mailing list
>>>> Juju@lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>
>>>>
>>>
>>
>
--
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
