When you specify master as local[2], it starts the spark components in a
single jvm. You need to specify the master correctly.
I have a default AWS EMR cluster (1 master, 1 slave) with Spark. When I run
a Spark process, it works fine -- but only on the master, as if it were
standalone.
The web-UI and logging code shows only 1 executor, the localhost.
How can I diagnose this?
(I create *SparkConf, *in Python, with *setMaster('local[2]'). )*
(Strangely, though I don't think that this causes the problem, there is
almost nothing spark-related on the slave machine:* /usr/lib/spark *has a
few jars, but that's it: *datanucleus-api-jdo.jar datanucleus-core.jar
datanucleus-rdbms.jar spark-yarn-shuffle.jar. *But this is an AWS EMR
cluster as created by* create-cluster*, so I would assume that the slave
and master are configured OK out-of the box.)
Joshua
