Hi Dillian and Nicholas If you figure out how to do this please post your recipe. It would be very useful
andy From: Nicholas Chammas <nicholas.cham...@gmail.com> Date: Wednesday, November 25, 2015 at 11:36 AM To: Dillian Murphey <crackshotm...@gmail.com>, "user @spark" <user@spark.apache.org> Subject: Re: Adding more slaves to a running cluster > spark-ec2 does not directly support adding instances to an existing cluster, > apart from the special case of adding slaves to a cluster with a master but no > slaves. There is an open issue to track adding this support, SPARK-2008 > <https://issues.apache.org/jira/browse/SPARK-2008> , but it doesn't have any > momentum at the moment. > > Your best bet currently is to do what you did and hack your way through using > spark-ec2's various scripts. > > You probably already know this, but to be clear, note that Spark itself > supports adding slaves to a running cluster. It's just that spark-ec2 hasn't > implemented a feature to do this work for you. > > Nick > > On Wed, Nov 25, 2015 at 2:27 PM Dillian Murphey <crackshotm...@gmail.com> > wrote: >> It appears start-slave.sh works on a running cluster. I'm surprised I can't >> find more info on this. Maybe I'm not looking hard enough? >> >> Using AWS and spot instances is incredibly more efficient, which begs for the >> need of dynamically adding more nodes while the cluster is up, yet everything >> I've found so far seems to indicate it isn't supported yet. >> >> But yet here I am with 1.5 and it at least appears to be working. Am I >> missing something? >> >> On Tue, Nov 24, 2015 at 4:40 PM, Dillian Murphey <crackshotm...@gmail.com> >> wrote: >>> What's the current status on adding slaves to a running cluster? I want to >>> leverage spark-ec2 and autoscaling groups. I want to launch slaves as spot >>> instances when I need to do some heavy lifting, but I don't want to bring >>> down my cluster in order to add nodes. >>> >>> Can this be done by just running start-slave.sh?? >>> >>> What about using Mesos? >>> >>> I just want to create an AMI for a slave and on some trigger launch it and >>> have it automatically add itself to the cluster. >>> >>> thanks >>
