Nick, Did you open a JIRA ticket for this feature to be implemented in spark-ec2? If so can you please point me to the ticket?
I would also like to do the autoscaling of spark nodes(add/remove slave nodes). So curious to know how did you acheive this. Sirisha Nick Chammas wrote > Sweet, thanks for the instructions. This will do for resizing a dev > cluster > that you can bring down at will. > > I will open a JIRA issue about adding the functionality I described to > spark-ec2. > > > On Fri, Apr 4, 2014 at 3:43 PM, Matei Zaharia < > matei.zaharia@ > >wrote: > >> This can't be done through the script right now, but you can do it >> manually as long as the cluster is stopped. If the cluster is stopped, >> just >> go into the AWS Console, right click a slave and choose "launch more of >> these" to add more. Or select multiple slaves and delete them. When you >> run >> spark-ec2 start the next time to start your cluster, it will set it up on >> all the machines it finds in the mycluster-slaves security group. >> >> This is pretty hacky so it would definitely be good to add this feature; >> feel free to open a JIRA about it. >> >> Matei >> >> On Apr 4, 2014, at 12:16 PM, Nicholas Chammas < > nicholas.chammas@ > > >> wrote: >> >> I would like to be able to use spark-ec2 to launch new slaves and add >> them >> to an existing, running cluster. Similarly, I would also like to remove >> slaves from an existing cluster. >> >> Use cases include: >> >> 1. Oh snap, I sized my cluster incorrectly. Let me add/remove some >> slaves. >> 2. During scheduled batch processing, I want to add some new slaves, >> perhaps on spot instances. When that processing is done, I want to >> kill >> them. (Cruel, I know.) >> >> I gather this is not possible at the moment. spark-ec2 appears to be able >> to launch new slaves for an existing cluster only if the master is >> stopped. >> I also do not see any ability to remove slaves from a cluster. >> >> Is that correct? Are there plans to add such functionality to spark-ec2 >> in >> the future? >> >> Nick >> >> >> ------------------------------ >> View this message in context: Having spark-ec2 join new slaves to >> existing >> cluster<http://apache-spark-user-list.1001560.n3.nabble.com/Having-spark-ec2-join-new-slaves-to-existing-cluster-tp3783.html> >> Sent from the Apache Spark User List mailing list >> archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at >> Nabble.com. >> >> >> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Having-spark-ec2-join-new-slaves-to-existing-cluster-tp3783p6795.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
