The slaves should connect to the master using the scripts in sbin... You can read about it here: http://spark.apache.org/docs/latest/spark-standalone.html
On Thu, Oct 6, 2016 at 6:46 PM, Mendelson, Assaf <assaf.mendel...@rsa.com> wrote: > Hi, > > I have a spark standalone cluster. On it, I am using 3 workers per node. > > So I added SPARK_WORKER_INSTANCES set to 3 in spark-env.sh > > The problem is, that when I run spark-shell I get the following warning: > > WARN SparkConf: > > SPARK_WORKER_INSTANCES was detected (set to '3'). > > This is deprecated in Spark 1.0+. > > > > Please instead use: > > - ./spark-submit with --num-executors to specify the number of executors > > - Or set SPARK_EXECUTOR_INSTANCES > > - spark.executor.instances to configure the number of instances in the > spark config. > > > > So how would I start a cluster of 3? SPARK_WORKER_INSTANCES is the only > way I see to start the standalone cluster and the only way I see to define > it is in spark-env.sh. The spark submit option, SPARK_EXECUTOR_INSTANCES > and spark.executor.instances are all related to submitting the job. > > > > Any ideas? > > Thanks > > Assaf > -- Regards, Ofer Eliassaf
