Ryan, did you come right with this? I've just ran into the same problem on a new 1.0.0 cluster I spun up. The issue was that my app was not running against the Spark master, but in local mode (a default setting in my app that was a throwback from 0.9.1 and was overriding the spark defaults on the cluster). So it was trying to create temp files in the spark.local.dir, as set in the Spark config - spark/conf/spark-env.sh has export SPARK_LOCAL_DIRS="/mnt/spark,/mnt2/spark"
But these directories do not get created on the master by the spark-ec2 launch scripts, only on the slaves. If you want to be able to run local mode jobs on the master, you can do "mkdir /mnt/spark" and "mkdir /mnt2/spark" and try again. Interestingly, if you try running the app without the --master local[4] flag, it should run it against the cluster workers and should actually work fine. Let me know if that is the case. Not sure if this should be considered a bug in ec2 scripts or not - I guess we'd probably want to be able to run local mode jobs on the master. Another issue is that, straight out the box if anyone spins up an EC2 cluster and just runs an example using run-example, they will hit this error since the examples run in local[] by default. So it was pretty confusing for me for a while since spark-shell worked fine and launched on the cluster, but run-example did not and hit this error... N On Fri, Aug 1, 2014 at 3:32 PM, Dean Wampler <[email protected]> wrote: > It looked like you were running in standalone mode (master set to > local[4]). That's how I ran it. > > Dean Wampler, Ph.D. > Author: Programming Scala, 2nd Edition > <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) > Typesafe <http://typesafe.com> > @deanwampler <http://twitter.com/deanwampler> > http://polyglotprogramming.com > > > On Thu, Jul 31, 2014 at 8:37 PM, ratabora <[email protected]> wrote: > >> Hey Dean! Thanks! >> >> Did you try running this on a local environment or one generated by the >> spark-ec2 script? >> >> The environment I am running on is a 4 data node 1 master spark cluster >> generated by the spark-ec2 script. I haven't modified anything in the >> environment except for adding data to the ephemeral hdfs. >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Issue-with-Spark-on-EC2-using-spark-ec2-script-tp11088p11117.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >
