Hi Apologies if this is a noob question. I have setup Spark 1.0.1 on EMR using a slightly modified version of script @ s3://elasticmapreduce/samples/spark/1.0.0/install-spark-shark-yarn.rb. It seems to be running fine with master logs stating:
14/08/05 14:36:56 INFO Master: I have been elected leader! New state: ALIVE 14/08/05 14:37:21 INFO Master: Registering worker ip-10-0-2-80.ec2.internal:52029 with 2 cores, 6.3 GB RAM The script has also created spark-env.sh under conf which has the following content: export SPARK_MASTER_IP=x.x.x.x export SCALA_HOME=/home/hadoop/.versions/scala-2.10.3 export SPARK_LOCAL_DIRS=/mnt/spark/ export SPARK_CLASSPATH="/usr/share/aws/emr/emr-fs/lib/*:/usr/share/aws/emr/lib/*:/home/hadoop/share/hadoop/common/lib/*:/home/hadoop/.versions/2.4.0/share/hadoop/common/lib/hadoop-lzo.jar" export SPARK_DAEMON_JAVA_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" export SPARK_ASSEMBLY_JAR=/home/hadoop/spark/lib/spark-assembly-1.0.1-hadoop2.4.0.jar However, when I run the spark-shell, sc.isLocal returns true. Also, no matter how many RDDs I cache, the used memory in the master UI (x.x.x.x:7077) shows 0B used. This leads me to believe that the spark-shell isn't connecting to Spark master and has started a local instance of spark. Is there something I am missing in my setup that allows for spark-shell to connect to master? Thanks, Aniket
