Hmm.. kinda working...
I'm getting a broken apache/ganglion at the last step, although spark-shell
does run.
Starting GANGLIA gmetad: [ OK ]
Stopping httpd: [FAILED]
Starting httpd: httpd: Syntax error on line 153 of
/etc/httpd/conf/httpd.conf: Cannot load modules/mod_authn_alias.so into
server: /etc/httpd/modules/mod_authn_alias.so: cannot open shared object
file: No such file or directory
[FAILED]
I've seen this before, though apparently it's not as fatal as I thought it
was. But I will miss ganglia. I think it's exclusive to the hvm ami. I
don't know.
And I definitely have to create /mnt/spark and copy-dir it before the
examples (bin/run-example SparkPi) will run, or this:
14/06/18 10:00:20 ERROR scheduler.TaskSetManager: Task 0.0:1 failed 1
times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 0.0:1 failed 1 times, most recent failure: Exception
failure in TID 1 on host localhost: java.io.IOException: No such file or
directory
java.io.UnixFileSystem.createFileExclusively(Native Method)
java.io.File.createNewFile(File.java:1006)
java.io.File.createTempFile(File.java:1989)
org.apache.spark.util.Utils$.fetchFile(Utils.scala:326)
I still feel like I'm missing something obvious, and making it harder than
it should be. (I wanted to try out Cassandra, but getting that on the
cluster as well looks like too much pain for now.) I'd be willing to step
up to YARN/Ambari, but is it worth it? They don't say much about Spark, and
the doco is pretty thin. I could just be trading up to more complicated
problems.
On Wed, Jun 18, 2014 at 4:05 PM, Jeremy Lee <unorthodox.engine...@gmail.com>
wrote:
> Ah, right. So only the launch script has changed. Everything else is still
> essentially binary compatible?
>
> Well, that makes it too easy! Thanks!
>
>
> On Wed, Jun 18, 2014 at 2:35 PM, Patrick Wendell <pwend...@gmail.com>
> wrote:
>
>> Actually you'll just want to clone the 1.0 branch then use the
>> spark-ec2 script in there to launch your cluster. The --spark-git-repo
>> flag is if you want to launch with a different version of Spark on the
>> cluster. In your case you just need a different version of the launch
>> script itself, which will be present in the 1.0 branch of Spark.
>>
>> - Patrick
>>
>> On Tue, Jun 17, 2014 at 9:29 PM, Jeremy Lee
>> <unorthodox.engine...@gmail.com> wrote:
>> > I am about to spin up some new clusters, so I may give that a go... any
>> > special instructions for making them work? I assume I use the "
>> > --spark-git-repo=" option on the spark-ec2 command. Is it as easy as
>> > concatenating your string as the value?
>> >
>> > On cluster management GUIs... I've been looking around at Amabari,
>> Datastax,
>> > Cloudera, OpsCenter etc. Not totally convinced by any of them yet.
>> Anyone
>> > using a good one I should know about? I'm really beginning to lean in
>> the
>> > direction of Cassandra as the distributed data store...
>> >
>> >
>> > On Wed, Jun 18, 2014 at 1:46 PM, Patrick Wendell <pwend...@gmail.com>
>> wrote:
>> >>
>> >> By the way, in case it's not clear, I mean our maintenance branches:
>> >>
>> >> https://github.com/apache/spark/tree/branch-1.0
>> >>
>> >> On Tue, Jun 17, 2014 at 8:35 PM, Patrick Wendell <pwend...@gmail.com>
>> >> wrote:
>> >> > Hey Jeremy,
>> >> >
>> >> > This is patched in the 1.0 and 0.9 branches of Spark. We're likely to
>> >> > make a 1.0.1 release soon (this patch being one of the main reasons),
>> >> > but if you are itching for this sooner, you can just checkout the
>> head
>> >> > of branch-1.0 and you will be able to use r3.XXX instances.
>> >> >
>> >> > - Patrick
>> >> >
>> >> > On Tue, Jun 17, 2014 at 4:17 PM, Jeremy Lee
>> >> > <unorthodox.engine...@gmail.com> wrote:
>> >> >> Some people (me included) might have wondered why all our m1.large
>> spot
>> >> >> instances (in us-west-1) shut down a few hours ago...
>> >> >>
>> >> >> Simple reason: The EC2 spot price for Spark's default "m1.large"
>> >> >> instances
>> >> >> just jumped from 0.016 per hour, to about 0.750. Yes, Fifty times.
>> >> >> Probably
>> >> >> something to do with world cup.
>> >> >>
>> >> >> So far this is just us-west-1, but prices have a tendency to
>> equalize
>> >> >> across
>> >> >> centers as the days pass. Time to make backups and plans.
>> >> >>
>> >> >> "m3" spot prices are still down at $0.02 (and being new, will be
>> >> >> bypassed by
>> >> >> older systems), so it would be REAAALLYY nice if there had been some
>> >> >> progress on that issue. Let me know if I can help with testing and
>> >> >> whatnot.
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Jeremy Lee BCompSci(Hons)
>> >> >> The Unorthodox Engineers
>> >
>> >
>> >
>> >
>> > --
>> > Jeremy Lee BCompSci(Hons)
>> > The Unorthodox Engineers
>>
>
>
>
> --
> Jeremy Lee BCompSci(Hons)
> The Unorthodox Engineers
>
--
Jeremy Lee BCompSci(Hons)
The Unorthodox Engineers
