[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256683#comment-14256683 ]
Nicholas Chammas commented on SPARK-3821: ----------------------------------------- Per the discussion earlier, I've [updated|https://github.com/nchammas/spark-ec2/tree/packer/packer] the Packer build configuration to drop the release-specific builds. I've also added GNU parallel to the list of installed tools and will use it in place of the {{while ... rsync ... & wait}} pattern used throughout the various setup scripts. I'll test out these changes on small (< 5 nodes) and large (>= 100 nodes) cluster launches and post updated benchmarks as well as an updated README and proposal. > Develop an automated way of creating Spark images (AMI, Docker, and others) > --------------------------------------------------------------------------- > > Key: SPARK-3821 > URL: https://issues.apache.org/jira/browse/SPARK-3821 > Project: Spark > Issue Type: Improvement > Components: Build, EC2 > Reporter: Nicholas Chammas > Assignee: Nicholas Chammas > Attachments: packer-proposal.html > > > Right now the creation of Spark AMIs or Docker containers is done manually. > With tools like [Packer|http://www.packer.io/], we should be able to automate > this work, and do so in such a way that multiple types of machine images can > be created from a single template. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org