It seems that the conf/slaves file is only for consumption by the following scripts:
sbin/start-slaves.sh sbin/stop-slaves.sh sbin/start-all.sh sbin/stop-all.sh I.e., conf/slaves file doesn't affect a running cluster. Is this true? On Mon, Mar 28, 2016 at 9:31 PM, Sung Hwan Chung <coded...@cs.stanford.edu> wrote: > No I didn't add it to the conf/slaves file. > > What I want to do is leverage auto-scale from AWS, without needing to stop > all the slaves (e.g. if a lot of slaves are idle, terminate those). > > Also, the book-keeping is easier if I don't have to deal with some > centralized list of slave list that needs to be modified every time a node > is added/removed. > > > On Mon, Mar 28, 2016 at 9:20 PM, Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> Have you added the slave host name to $SPARK_HOME/conf? >> >> Then you can use start-slaves.sh or stop-slaves.sh for all instances >> >> The assumption is that slave boxes have $SPARK_HOME installed in the same >> directory as $SPARK_HOME is installed in the master. >> >> HTH >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> On 28 March 2016 at 22:06, Sung Hwan Chung <coded...@cs.stanford.edu> >> wrote: >> >>> Hello, >>> >>> I found that I could dynamically add/remove new workers to a running >>> standalone Spark cluster by simply triggering: >>> >>> start-slave.sh (SPARK_MASTER_ADDR) >>> >>> and >>> >>> stop-slave.sh >>> >>> E.g., I could instantiate a new AWS instance and just add it to a >>> running cluster without needing to add it to slaves file and restarting the >>> whole cluster. >>> It seems that there's no need for me to stop a running cluster. >>> >>> Is this a valid way of dynamically resizing a spark cluster (as of now, >>> I'm not concerned about HDFS)? Or will there be certain unforeseen problems >>> if nodes are added/removed this way? >>> >> >> >
