I agree with this - there is also the issue of different sized masters and slaves, and numbers of executors for hefty machines (e.g. r3.8xlarges), tagging of instances and volumes (we use this for cost attribution at my workplace), and running in VPCs.
I think think it might be useful to take a layered approach: the first step could be getting a good reliable image produced - Nick's ticket - then doing some work on the launch script. Regarding the EMR like service - I think I heard that AWS is planning to add spark support to EMR, but as usual there's nothing firm until it's released. On Tue, Oct 7, 2014 at 7:48 AM, Daniil Osipov <daniil.osi...@shazam.com> wrote: > I've also been looking at this. Basically, the Spark EC2 script is > excellent for small development clusters of several nodes, but isn't > suitable for production. It handles instance setup in a single threaded > manner, while it can easily be parallelized. It also doesn't handle failure > well, ex when an instance fails to start or is taking too long to respond. > > Our desire was to have an equivalent to Amazon EMR[1] API that would > trigger Spark jobs, including specified cluster setup. I've done some work > towards that end, and it would benefit from an updated AMI greatly. > > Dan > > [1] > > http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-cli-commands.html > > On Sat, Oct 4, 2014 at 7:28 AM, Nicholas Chammas < > nicholas.cham...@gmail.com > > wrote: > > > Thanks for posting that script, Patrick. It looks like a good place to > > start. > > > > Regarding Docker vs. Packer, as I understand it you can use Packer to > > create Docker containers at the same time as AMIs and other image types. > > > > Nick > > > > > > On Sat, Oct 4, 2014 at 2:49 AM, Patrick Wendell <pwend...@gmail.com> > > wrote: > > > > > Hey All, > > > > > > Just a couple notes. I recently posted a shell script for creating the > > > AMI's from a clean Amazon Linux AMI. > > > > > > https://github.com/mesos/spark-ec2/blob/v3/create_image.sh > > > > > > I think I will update the AMI's soon to get the most recent security > > > updates. For spark-ec2's purpose this is probably sufficient (we'll > > > only need to re-create them every few months). > > > > > > However, it would be cool if someone wanted to tackle providing a more > > > general mechanism for defining Spark-friendly "images" that can be > > > used more generally. I had thought that docker might be a good way to > > > go for something like this - but maybe this packer thing is good too. > > > > > > For one thing, if we had a standard image we could use it to create > > > containers for running Spark's unit test, which would be really cool. > > > This would help a lot with random issues around port and filesystem > > > contention we have for unit tests. > > > > > > I'm not sure if the long term place for this would be inside the spark > > > codebase or a community library or what. But it would definitely be > > > very valuable to have if someone wanted to take it on. > > > > > > - Patrick > > > > > > On Fri, Oct 3, 2014 at 5:20 PM, Nicholas Chammas > > > <nicholas.cham...@gmail.com> wrote: > > > > FYI: There is an existing issue -- SPARK-3314 > > > > <https://issues.apache.org/jira/browse/SPARK-3314> -- about > scripting > > > the > > > > creation of Spark AMIs. > > > > > > > > With Packer, it looks like we may be able to script the creation of > > > > multiple image types (VMWare, GCE, AMI, Docker, etc...) at once from > a > > > > single Packer template. That's very cool. > > > > > > > > I'll be looking into this. > > > > > > > > Nick > > > > > > > > > > > > On Thu, Oct 2, 2014 at 8:23 PM, Nicholas Chammas < > > > nicholas.cham...@gmail.com > > > >> wrote: > > > > > > > >> Thanks for the update, Nate. I'm looking forward to seeing how these > > > >> projects turn out. > > > >> > > > >> David, Packer looks very, very interesting. I'm gonna look into it > > more > > > >> next week. > > > >> > > > >> Nick > > > >> > > > >> > > > >> On Thu, Oct 2, 2014 at 8:00 PM, Nate D'Amico <n...@reactor8.com> > > wrote: > > > >> > > > >>> Bit of progress on our end, bit of lagging as well. Our guy > leading > > > >>> effort got little bogged down on client project to update hive/sql > > > testbed > > > >>> to latest spark/sparkSQL, also launching public service so we have > > > been bit > > > >>> scattered recently. > > > >>> > > > >>> Will have some more updates probably after next week. We are > > planning > > > on > > > >>> taking our client work around hive/spark, plus taking over the > bigtop > > > >>> automation work to modernize and get that fit for human consumption > > > outside > > > >>> or org. All our work and puppet modules will be open sourced, > > > documented, > > > >>> hopefully start to rally some other folks around effort that find > it > > > useful > > > >>> > > > >>> Side note, another effort we are looking into is gradle > > tests/support. > > > >>> We have been leveraging serverspec for some basic infrastructure > > > tests, but > > > >>> with bigtop switching over to gradle builds/testing setup in 0.8 we > > > want to > > > >>> include support for that in our own efforts, probably some stuff > that > > > can > > > >>> be learned and leveraged in spark world for repeatable/tested > > > infrastructure > > > >>> > > > >>> If anyone has any specific automation questions to your environment > > you > > > >>> can drop me a line directly.., will try to help out best I can. > Else > > > will > > > >>> post update to dev list once we get on top of our own product > release > > > and > > > >>> the bigtop work > > > >>> > > > >>> Nate > > > >>> > > > >>> > > > >>> -----Original Message----- > > > >>> From: David Rowe [mailto:davidr...@gmail.com] > > > >>> Sent: Thursday, October 02, 2014 4:44 PM > > > >>> To: Nicholas Chammas > > > >>> Cc: dev; Shivaram Venkataraman > > > >>> Subject: Re: EC2 clusters ready in launch time + 30 seconds > > > >>> > > > >>> I think this is exactly what packer is for. See e.g. > > > >>> http://www.packer.io/intro/getting-started/build-image.html > > > >>> > > > >>> On a related note, the current AMI for hvm systems (e.g. m3.*, > r3.*) > > > has > > > >>> a bad package for httpd, whcih causes ganglia not to start. For > some > > > reason > > > >>> I can't get access to the raw AMI to fix it. > > > >>> > > > >>> On Fri, Oct 3, 2014 at 9:30 AM, Nicholas Chammas < > > > >>> nicholas.cham...@gmail.com > > > >>> > wrote: > > > >>> > > > >>> > Is there perhaps a way to define an AMI programmatically? Like, a > > > >>> > collection of base AMI id + list of required stuff to be > installed > > + > > > >>> > list of required configuration changes. I'm guessing that's what > > > >>> > people use things like Puppet, Ansible, or maybe also AWS > > > >>> CloudFormation for, right? > > > >>> > > > > >>> > If we could do something like that, then with every new release > of > > > >>> > Spark we could quickly and easily create new AMIs that have > > > everything > > > >>> we need. > > > >>> > spark-ec2 would only have to bring up the instances and do a > > minimal > > > >>> > amount of configuration, and the only thing we'd need to track in > > the > > > >>> > Spark repo is the code that defines what goes on the AMI, as well > > as > > > a > > > >>> > list of the AMI ids specific to each release. > > > >>> > > > > >>> > I'm just thinking out loud here. Does this make sense? > > > >>> > > > > >>> > Nate, > > > >>> > > > > >>> > Any progress on your end with this work? > > > >>> > > > > >>> > Nick > > > >>> > > > > >>> > > > > >>> > On Sun, Jul 13, 2014 at 8:53 PM, Shivaram Venkataraman < > > > >>> > shiva...@eecs.berkeley.edu> wrote: > > > >>> > > > > >>> > > It should be possible to improve cluster launch time if we are > > > >>> > > careful about what commands we run during setup. One way to do > > this > > > >>> > > would be to walk down the list of things we do for cluster > > > >>> > > initialization and see if there is anything we can do make > things > > > >>> > > faster. Unfortunately this might > > > >>> > be > > > >>> > > pretty time consuming, but I don't know of a better strategy. > The > > > >>> > > place > > > >>> > to > > > >>> > > start would be the setup.sh file at > > > >>> > > https://github.com/mesos/spark-ec2/blob/v3/setup.sh > > > >>> > > > > > >>> > > Here are some things that take a lot of time and could be > > improved: > > > >>> > > 1. Creating swap partitions on all machines. We could check if > > > there > > > >>> > > is a way to get EC2 to always mount a swap partition 2. > Copying / > > > >>> > > syncing things across slaves. The copy-dir script is called too > > > many > > > >>> > > times right now and each time it pauses for a few milliseconds > > > >>> > > between slaves [1]. This could be improved by removing > > unnecessary > > > >>> > > copies 3. We could make less frequently used modules like > > Tachyon, > > > >>> > > persistent > > > >>> > hdfs > > > >>> > > not a part of the default setup. > > > >>> > > > > > >>> > > [1] https://github.com/mesos/spark-ec2/blob/v3/copy-dir.sh#L42 > > > >>> > > > > > >>> > > Thanks > > > >>> > > Shivaram > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > On Sat, Jul 12, 2014 at 7:02 PM, Nicholas Chammas < > > > >>> > > nicholas.cham...@gmail.com> wrote: > > > >>> > > > > > >>> > > > On Thu, Jul 10, 2014 at 8:10 PM, Nate D'Amico < > > n...@reactor8.com > > > > > > > >>> > wrote: > > > >>> > > > > > > >>> > > > > Starting to work through some automation/config stuff for > > spark > > > >>> > > > > stack > > > >>> > > on > > > >>> > > > > EC2 with a project, will be focusing the work through the > > > apache > > > >>> > bigtop > > > >>> > > > > effort to start, can then share with spark community > directly > > > as > > > >>> > things > > > >>> > > > > progress if people are interested > > > >>> > > > > > > >>> > > > > > > >>> > > > Let us know how that goes. I'm definitely interested in > hearing > > > >>> more. > > > >>> > > > > > > >>> > > > Nick > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > >> > > > > > >
