[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183390#comment-14183390 ]
Nicholas Chammas commented on SPARK-3821: ----------------------------------------- Going for something like EMR's CLI is potentially very useful, though perhaps a bit outside the scope of the original {{spark-ec2}} (and there's nothing wrong with that!). What I'm doing will keep {{spark-ec2}} mostly as-is on the surface, but tackle the launch times and parallelism as you described. I'm currently only generating AMIs with Hadoop 2 and Spark 1.1.0, or a base AMI with everything except Hadoop and Spark. I haven't yet figured out the details of how to handle the full version matrix. Right now I'm leaning towards having a "base" AMI that any version of Spark can be installed on relatively quickly and AMIs for specific versions of Spark starting from 1.1.0. > Develop an automated way of creating Spark images (AMI, Docker, and others) > --------------------------------------------------------------------------- > > Key: SPARK-3821 > URL: https://issues.apache.org/jira/browse/SPARK-3821 > Project: Spark > Issue Type: Improvement > Components: Build, EC2 > Reporter: Nicholas Chammas > Assignee: Nicholas Chammas > > Right now the creation of Spark AMIs or Docker containers is done manually. > With tools like [Packer|http://www.packer.io/], we should be able to automate > this work, and do so in such a way that multiple types of machine images can > be created from a single template. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org