Module Name: othersrc
Committed By: riz
Date: Fri Apr 20 21:01:03 UTC 2012
Added Files:
othersrc/share/examples/ec2: README TODO build_ec2_img.sh create-new.sh
create_ec2_ami.sh set_ec2_env.sh
othersrc/share/examples/ec2/files: bootstrap.sh ec2_bootmail
ec2_firstboot ec2_init motd spec.in
Log Message:
Example scripts and supporting files for creating Amazon EC2 AMIs
(Amazon Machine Images) running NetBSD. Requires an Amazon EC2 account.
Please test and extend these scripts!
The original scripts were written by Jean-Yves Migeon, heavily modified
>From the README:
Notes on using these scripts for creating EC2 AMIs
--------------------------------------------------
Here are some assumptions made:
- You have the EC2 API tools already set up. If you don't already, the
easiest way to do so is to install the misc/ec2-api-tools package
from pkgsrc. An easy way to check if they're set up properly is
if you can run "ec2-describe-instances" without an error.
- You are running this from a NetBSD system. It's certainly not impossible
to update this for running on another platform, but it'll be necessary
to adapt to using a NetBSD tools build for (at least) makefs.
- Your default EC2 profile allows ssh to your running instances from
the location you're running this script!
- The most-used (and tested!) script (create_ec2_ami.sh) assumes that there
are NetBSD 6.0 AMIs available on EC2 already, for bootstrapping. The
public images in account 101367081206 (an account owned by me, Jeff Rizzo,
a NetBSD developer) are available for use, and are searched in addition
to the user's own AMIs. You can also specify a particular AMI to use
for the bootstrapping. There is another script (create-new.sh) which
is less well tested, and which may currently be broken, which allows
you to use a Linux AMI to create a NetBSD one. You will need to
create a 'bootstrap' image with an ext2 file system (see build_ec2_img.sh,
specifically the "-b" flag).
CREATING AN AMAZON MACHINE IMAGE (AMI)
--------------------------------------
Once you've verified the Amazon EC2 command-line tools work (see above),
the basic workflow is as follows:
- Edit the set_ec2_env.sh script to have the location of your certs/keys.
This file is read by several of the scripts.
- Either build sets with build.sh, or identify where the release you want to
base your AMI on lives. You can access built sets via http, ftp, or on
the local fs.
- Build an image for upload, using the build_ec2_img.sh script. This
image will contain everything *except* the kernel (the kernel will be
on a separate ext2fs partition, due to EC2 boot requirements), and will
be resized to fit the volume created in the AMI creation script.
An example of doing this:
./build_ec2_img.sh -d
http://ftp.netbsd.org/pub/NetBSD/NetBSD-6.0_BETA/amd64/binary/sets
...this will create an image file in the current directory.
- Create the AMI from the above image plus a kernel. You have to
specify which region to create the AMI in:
./create_ec2_ami.sh -i ./NetBSD-AMI.img.xz -a x86_64 -d "AMI description goes
here" -k
http://ftp.netbsd.org/pub/NetBSD/NetBSD-6.0_BETA/amd64/binary/kernel/netbsd-XEN3_DOMU.gz
-v "6.0_BETA-or-whatever-version-string-you-want" us-east-1
The process can take anywhere from 10-60 minutes, depending on which region
you choose and how fast your connection is for uploading. After a successful
run, be sure to test your AMI!
To generate a diff of this commit:
cvs rdiff -u -r0 -r1.1 othersrc/share/examples/ec2/README \
othersrc/share/examples/ec2/TODO \
othersrc/share/examples/ec2/build_ec2_img.sh \
othersrc/share/examples/ec2/create-new.sh \
othersrc/share/examples/ec2/create_ec2_ami.sh \
othersrc/share/examples/ec2/set_ec2_env.sh
cvs rdiff -u -r0 -r1.1 othersrc/share/examples/ec2/files/bootstrap.sh \
othersrc/share/examples/ec2/files/ec2_bootmail \
othersrc/share/examples/ec2/files/ec2_firstboot \
othersrc/share/examples/ec2/files/ec2_init \
othersrc/share/examples/ec2/files/motd \
othersrc/share/examples/ec2/files/spec.in
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
Added files:
Index: othersrc/share/examples/ec2/README
diff -u /dev/null othersrc/share/examples/ec2/README:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/README Fri Apr 20 21:01:02 2012
@@ -0,0 +1,61 @@
+Notes on using these scripts for creating EC2 AMIs
+--------------------------------------------------
+
+Here are some assumptions made:
+
+- You have the EC2 API tools already set up. If you don't already, the
+ easiest way to do so is to install the misc/ec2-api-tools package
+ from pkgsrc. An easy way to check if they're set up properly is
+ if you can run "ec2-describe-instances" without an error.
+
+- You are running this from a NetBSD system. It's certainly not impossible
+ to update this for running on another platform, but it'll be necessary
+ to adapt to using a NetBSD tools build for (at least) makefs.
+
+- Your default EC2 profile allows ssh to your running instances from
+ the location you're running this script!
+
+- The most-used (and tested!) script (create_ec2_ami.sh) assumes that there
+ are NetBSD 6.0 AMIs available on EC2 already, for bootstrapping. The
+ public images in account 101367081206 (an account owned by me, Jeff Rizzo,
+ a NetBSD developer) are available for use, and are searched in addition
+ to the user's own AMIs. You can also specify a particular AMI to use
+ for the bootstrapping. There is another script (create-new.sh) which
+ is less well tested, and which may currently be broken, which allows
+ you to use a Linux AMI to create a NetBSD one. You will need to
+ create a 'bootstrap' image with an ext2 file system (see build_ec2_img.sh,
+ specifically the "-b" flag).
+
+CREATING AN AMAZON MACHINE IMAGE (AMI)
+--------------------------------------
+
+Once you've verified the Amazon EC2 command-line tools work (see above),
+the basic workflow is as follows:
+
+- Edit the set_ec2_env.sh script to have the location of your certs/keys.
+ This file is read by several of the scripts.
+
+- Either build sets with build.sh, or identify where the release you want to
+ base your AMI on lives. You can access built sets via http, ftp, or on
+ the local fs.
+
+- Build an image for upload, using the build_ec2_img.sh script. This
+ image will contain everything *except* the kernel (the kernel will be
+ on a separate ext2fs partition, due to EC2 boot requirements), and will
+ be resized to fit the volume created in the AMI creation script.
+
+ An example of doing this:
+
+ ./build_ec2_img.sh -d http://ftp.netbsd.org/pub/NetBSD/NetBSD-6.0_BETA/amd64/binary/sets
+
+ ...this will create an image file in the current directory.
+
+- Create the AMI from the above image plus a kernel. You have to
+ specify which region to create the AMI in:
+
+ ./create_ec2_ami.sh -i ./NetBSD-AMI.img.xz -a x86_64 -d "AMI description goes here" -k http://ftp.netbsd.org/pub/NetBSD/NetBSD-6.0_BETA/amd64/binary/kernel/netbsd-XEN3_DOMU.gz -v "6.0_BETA-or-whatever-version-string-you-want" us-east-1
+
+
+The process can take anywhere from 10-60 minutes, depending on which region
+you choose and how fast your connection is for uploading. After a successful
+run, be sure to test your AMI!
Index: othersrc/share/examples/ec2/TODO
diff -u /dev/null othersrc/share/examples/ec2/TODO:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/TODO Fri Apr 20 21:01:02 2012
@@ -0,0 +1,11 @@
+TO DO (in no particular order):
+
+- Allow selecting different compression schemes. I chose xz because it
+ makes the smallest images and therefore costs the least, but it's S-L-O-W.
+- Better documentation
+- Improve the automatic selection of the bootstrap AMI.
+- Improve tracking of launched instances to decrease the likelihood of
+ leaving anything running accidentally.
+- Allow the sets installed to be changed some way besides editing the
+ build_ec2_img.sh script.
+- Allow additional files to be installed more smoothly.
Index: othersrc/share/examples/ec2/build_ec2_img.sh
diff -u /dev/null othersrc/share/examples/ec2/build_ec2_img.sh:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/build_ec2_img.sh Fri Apr 20 21:01:02 2012
@@ -0,0 +1,205 @@
+#!/bin/sh
+
+# This script is used to generate a correct NetBSD AMI image, suitable
+# for uploading.
+
+# mkfs.ext2 is required to build a boot image - install e2fsprogs from pkgsrc.
+# also requires rump_ext2fs to be able to mount a file image, so root
+# privileges are required.
+
+# Assume the base directory of this script is the right place for some
+# stuff.
+if [ "$(dirname $0 | cut -c 1)" != "/" ]; then
+ BASEDIR="$(pwd)/$(dirname $0)"
+else
+ BASEDIR=$(dirname $0)
+fi
+
+EC2DIR="$BASEDIR/work"
+MOUNTDIR="/mnt"
+KERNEL=/tmp/netbsd
+BOOTIMGFILE="$BASEDIR/NetBSD-boot.img"
+IMGFILE="$BASEDIR/NetBSD-AMI.img"
+
+# list of sets to install
+SETS="base.tgz comp.tgz etc.tgz games.tgz man.tgz misc.tgz tests.tgz text.tgz"
+
+# SETSDIR is where to fetch the above sets from, and is set with -d
+SETSDIR="http://ftp.NetBSD.org/pub/NetBSD/NetBSD-6.0_BETA/amd64/binary/sets";
+
+FILES="$BASEDIR/files"
+SPECFILE="/tmp/spec.$$"
+BOOTMAIL=NO
+
+set -e
+
+usage() {
+ echo "usage: $0 <options>"
+ echo "options:"
+ echo " -b <bootimage> Create an ext2 boot image (with kernel) in a file."
+ echo " -d <dir> sets directory"
+ echo " -i <imagefile> name of disk image file"
+ echo " -k <kernel> location of kernel to use in bootimage"
+ echo " -m <email> Enable first-boot email, send to <email>"
+ echo " -w <dir> work directory"
+ exit 1
+}
+
+# Handle commandline args
+while getopts d:i:k:m:w: opt
+do
+ case $opt in
+ b) CREATE_BOOTIMG=YES; BOOTIMGFILE=$OPTARG;;
+ d) SETSDIR=$OPTARG;;
+ i) IMGFILE=$OPTARG;;
+ k) KERNEL=$OPTARG;;
+ m) BOOTMAIL=YES; EMAIL=$OPTARG;;
+ w) EC2DIR=$OPTARG;;
+ \?) usage;;
+ esac
+done
+shift $(expr $OPTIND - 1)
+
+unpack_sets() {
+ # make EC2DIR just in case
+ mkdir -p $EC2DIR
+ # clear out EC2DIR
+ rm -rf $EC2DIR/*
+ for set in $SETS; do
+ case "${SETSDIR}" in
+ ftp:*|http:*)
+ ftp -o - ${SETSDIR}/${set} | progress -z tar -xpf - -C $EC2DIR
+ ;;
+ *)
+ progress -zf ${SETSDIR}/${set} tar -xpf - -C $EC2DIR
+ ;;
+ esac
+ done
+
+}
+
+makedev() {
+ local wd=$(pwd)
+ cd $EC2DIR/dev
+ sh ./MAKEDEV all
+ cd $wd
+}
+
+# this is tricky because EC2 requires an ext2fs partition for boot, but
+# our ext2 handling is not as sophisticated as other stuff. We actually
+# need to mount an ext2fs partition to put files on it.
+create_bootimg() {
+ # make the image 256m to speed xfer and writing. EC2 minimum
+ # disk size is 1G.
+ dd if=/dev/zero of=${BOOTIMGFILE} bs=1m count=256
+ mkfs.ext2 -F ${BOOTIMGFILE}
+
+ # use rump_ext2fs to mount from a file
+ rump_ext2fs ${BOOTIMGFILE} ${MOUNTDIR}
+
+ mkdir -p ${MOUNTDIR}/boot/grub
+ cat > ${MOUNTDIR}/boot/grub/menu.lst << EOF
+default=0
+timeout=0
+hiddenmenu
+
+title NetBSD AMI
+root (hd0)
+kernel /boot/netbsd root=xbd1
+EOF
+
+ cp ${KERNEL} ${MOUNTDIR}/boot/netbsd
+ umount ${MOUNTDIR}
+ sync
+ echo "Compressing $BOOTIMGFILE with xz..."
+ xz -f -6 ${BOOTIMGFILE}
+}
+
+patch_install() {
+ local wd=$(pwd)
+ cd $EC2DIR
+
+ TMPFILE=$(mktemp /tmp/netbsd-ec2.XXXXXX)
+ trap "rm -f $TMPFILE; exit 1" INT EXIT QUIT
+
+ sed 's/rc_configured=NO/rc_configured=YES/g' \
+ etc/rc.conf > $TMPFILE
+ cp "$TMPFILE" etc/rc.conf
+
+ # required to allow connection through EC2 SSH keys
+ cat >> etc/rc.conf << EOF
+
+ec2_init=YES # for setup of SSH key pair(s)
+ec2_bootmail=${BOOTMAIL} # send initial email
+ec2_bootmail_addr="${EMAIL}" # email address to send initial email
+sshd=YES # for remote shell access to instance
+dhcpcd_flags="-t 0" # Wait for the DHCP server forever
+noswap=YES # not configured for swap, don't warn
+EOF
+
+ sed 's/^#PermitRootLogin.*/PermitRootLogin without-password/g' \
+ etc/ssh/sshd_config > $TMPFILE
+ cp -f "$TMPFILE" etc/ssh/sshd_config
+
+ cp -f $FILES/ec2_init etc/rc.d
+ cp -f $FILES/ec2_bootmail etc/rc.d
+ cp -f $FILES/ec2_firstboot etc/rc.d
+ touch root/firstboot
+ chmod 555 etc/rc.d/ec2_*
+
+ # Some required files and directories
+ # Add proc and kern directories
+ mkdir -p grub kern proc
+ # EC2 network configuration, via DHCP
+ echo "dhcp" > etc/ifconfig.xennet0
+ # Basic fstab entries
+ cat > etc/fstab << 'EOF'
+/dev/xbd1a / ffs rw,log 1 1
+/dev/xbd0a /grub ext2fs rw 2 2
+kernfs /kern kernfs rw
+ptyfs /dev/pts ptyfs rw
+procfs /proc procfs rw
+EOF
+
+ cp -f ${FILES}/motd etc/motd
+
+ # bootstrap script user is prompted to run
+ cp -f ${FILES}/bootstrap.sh usr/bootstrap.sh
+ chmod 755 usr/bootstrap.sh
+ cd $wd
+}
+
+generate_img() {
+ echo "Preparing spec files for makefs..."
+ rm -f ${SPECFILE}
+ cat ${EC2DIR}/etc/mtree/* |
+ sed -e 's/ size=[0-9]*//' > ${SPECFILE}
+ sh ${EC2DIR}/dev/MAKEDEV -s all |
+ sed -e '/^\. type=dir/d' -e 's,^\.,./dev,' >> ${SPECFILE}
+ cat ${FILES}/spec.in >> ${SPECFILE}
+ chmod +r ${EC2DIR}/var/spool/ftp/hidden
+ makefs -t ffs -B le -f 300000 -b 25% -N ${EC2DIR}/etc/ \
+ -F ${SPECFILE} $IMGFILE $EC2DIR
+ echo "Compressing $IMGFILE with xz..."
+ xz -f -6 $IMGFILE
+ echo "Cleaning up..."
+ rm -f ${SPECFILE}
+ rm -rf ${EC2DIR}/*
+}
+
+summary() {
+ echo "Creation of NetBSD Amazon EC2 image successful."
+if [ "$CREATE_BOOTIMG" = "YES" ]; then
+ echo "Boot image (compressed): $BOOTIMGFILE.xz"
+fi
+ echo "Root image (compressed): $IMGFILE.xz"
+}
+
+unpack_sets
+makedev
+patch_install
+generate_img
+if [ "$CREATE_BOOTIMG" = "YES" ]; then
+ create_bootimg
+fi
+summary
Index: othersrc/share/examples/ec2/create-new.sh
diff -u /dev/null othersrc/share/examples/ec2/create-new.sh:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/create-new.sh Fri Apr 20 21:01:02 2012
@@ -0,0 +1,245 @@
+#!/bin/sh
+
+# Assume the base directory of this script is the right place for some
+# stuff.
+if [ "$(dirname $0 | cut -c 1)" != "/" ]; then
+ BASEDIR="$(pwd)/$(dirname $0)"
+else
+ BASEDIR=$(dirname $0)
+fi
+
+ARCH="i386"
+VERSION="6.0_BETA"
+DESCRIPTION="AMI built: $(date)"
+IMGFILE="/space/NetBSD-AMI.img.xz"
+BOOTIMGFILE="/space/NetBSD-boot.img.xz"
+
+# Load credentials
+. ${BASEDIR}/set_ec2_env.sh
+
+usage() {
+ echo "Usage: ${0##*/} region"
+ exit 1
+}
+
+if [ $# -eq 1 ]; then
+ REGION=$1
+else
+ usage
+fi
+
+get_ami_to_run() {
+ # grab the first amazon linux ami. Should work fine.
+ local ami=$(ec2-describe-images -o amazon --region $REGION \
+ -F name=amzn-ami* -F architecture=$ARCH | \
+ awk 'NR == 1 {print $2}')
+
+ echo $ami
+}
+
+# find a kernel image of the correct arch. This may need to be tweaked
+# as Amazon updates their images.
+get_aki() {
+ local aki=$(ec2-describe-images -o amazon -F image-type=kernel \
+ -F manifest-location=*pv-grub-hd0_* -F architecture=$ARCH \
+ --region $REGION | awk 'NR == 1 {print $2}')
+
+ echo $aki
+}
+
+run_instance() {
+ local ami=$1
+
+ instance=$(ec2-run-instances $ami -k $EC2_SSH_KEYNAME \
+ -t t1.micro --region $REGION)
+
+ instanceid=$(echo $instance | awk '{print $6}')
+ zoneid=$(echo $instance | awk '{print $13}')
+
+ echo "Launched instance $instanceid from $ami"
+
+ while true; do
+ address=$(ec2-describe-instances $instanceid --region $REGION |\
+ awk '/running/ {print $4}')
+ if [ "$address" = "" ]; then
+ echo "Waiting for instance $instanceid..."
+ sleep 30
+ else
+ echo "$instanceid running, address: $address"
+ break
+ fi
+ done
+}
+
+get_ssh_fingerprints()
+{
+ local TMPFILE1=$(mktemp /tmp/ec2-ssh-host-keys.XXXXXX)
+ local TMPFILE2=$(mktemp /tmp/ec2-ssh-host-keys2.XXXXXX)
+ local LOGFILE="/tmp/system.log.$INSTANCEID"
+
+ trap "rm -f $TMPFILE1 $TMPFILE2; exit 1" INT EXIT QUIT
+
+ local count=0
+
+ while true; do
+ ec2-get-console-output $instanceid --region $REGION > $TMPFILE1
+
+ if grep -q -- "-----END SSH HOST KEY FINGERPRINTS-----" \
+ $TMPFILE1; then
+ echo "EC2 instance report:"
+ awk '/-----BEGIN SSH HOST KEY FINGERPRINTS-----/ {flag=1;next} /-----END SSH HOST KEY FINGERPRINTS-----/ {exit} flag { print }' $TMPFILE1 |sed -n "s/^ec2: //gp" | sort > $TMPFILE2
+ mv $TMPFILE2 $TMPFILE1
+ cat $TMPFILE1
+ break
+ fi
+
+ count=$((count + 1))
+ if [ $count -gt 5 ]; then
+ echo "Failed to get ssh fingerprints from console"
+ echo "after $count tries. The log is available"
+ echo "under $LOGFILE"
+ echo "Connect to AWS Console to clean up volumes and"
+ echo "terminate instances."
+ ec2-get-console-output $instanceid --region $REGION \
+ >$LOGFILE
+ exit 1
+ else
+ echo "Waiting for console output from instance..."
+ sleep 90
+ fi
+ done
+
+ local fp1=$(awk '/RSA/ {print $2}' $TMPFILE1)
+ ssh-keyscan -t rsa $address 2>/dev/null > $TMPFILE2
+ local fp2=$(ssh-keygen -l -f $TMPFILE2 | awk '{print $2}')
+ echo "Console fingerprint: $fp1"
+ echo "Keyscan fingerprint: $fp2"
+
+ ssh-keygen -R $address
+ cat $TMPFILE2 >> ~/.ssh/known_hosts
+}
+
+terminate_instance() {
+ local i=$1
+
+ ec2-terminate-instances $i --region $REGION
+}
+
+create_volume() {
+ local size=$1
+
+ local volid=$(ec2-create-volume -s $size --region $REGION -z $zoneid \
+ | awk '{print $2}')
+
+ echo $volid
+}
+
+create_snapshot() {
+ local volid=$1
+ local desc="$2"
+
+echo volid = $volid >&2
+echo desc = $desc >&2
+
+ local snapid=$(ec2-create-snapshot $volid --region $REGION -d "$desc" \
+ | awk '{print $2}')
+
+ echo $snapid
+}
+
+
+# use wait_volume to wait for a volume to be available or attached
+wait_volume() {
+ local volid=$1
+ local state=$2
+ local avail
+
+ while true; do
+ avail=$(ec2-describe-volumes $volid --region $REGION | \
+ awk "/$state/ {print}")
+ if [ "$avail" = "" ]; then
+ echo "Waiting for volume $volid"
+ sleep 15
+ else
+ break
+ fi
+ done
+}
+
+wait_snapshot() {
+ local snapid=$1
+ local state=$2
+ local avail
+
+ while true; do
+ avail=$(ec2-describe-snapshots $snapid --region $REGION | \
+ awk "/$state/ {print}")
+ if [ "$avail" = "" ]; then
+ echo "Waiting for snapshot $snapid"
+ sleep 15
+ else
+ break
+ fi
+ done
+}
+
+run_remote() {
+ echo doing remote: $*
+ ssh -i $EC2_SSH_KEY -t -t ec2-user@$address $*
+}
+
+amiid=$(get_ami_to_run)
+akiid=$(get_aki)
+
+run_instance $amiid
+
+get_ssh_fingerprints
+
+volume=$(create_volume 1)
+ec2-attach-volume $volume --region $REGION -i $instanceid -d "/dev/sdf"
+
+volume2=$(create_volume 5)
+ec2-attach-volume $volume2 --region $REGION -i $instanceid -d "/dev/sdg"
+
+wait_volume $volume attached
+wait_volume $volume2 attached
+
+echo "Copying $IMGFILE to $instanceid"
+scp -i $EC2_SSH_KEY $IMGFILE ec2-user@$address:
+
+echo "Copying $BOOTIMGFILE to $instanceid"
+scp -i $EC2_SSH_KEY $BOOTIMGFILE ec2-user@$address:
+
+echo "ssh 1"
+run_remote "xzcat -f $(basename $IMGFILE) | sudo dd of=/dev/xvdg bs=512k"
+echo "ssh 2"
+run_remote "xzcat -f $(basename $BOOTIMGFILE) | sudo dd of=/dev/xvdf bs=512k"
+
+
+echo "Doing Snapshots:"
+
+snap1id=$(create_snapshot $volume "Grub NetBSD boot volume $VERSION $DESCRIPTION")
+snap2id=$(create_snapshot $volume2 "NetBSD root $VERSION $DESCRIPTION")
+
+terminate_instance $instanceid
+
+wait_snapshot $snap1id completed
+wait_snapshot $snap2id completed
+
+echo ec2-register -a $ARCH --kernel $akiid --region $REGION -b "/dev/sda1=$snap1id" -b "/dev/sda2=$snap2id" -n "NetBSD-$ARCH-$VERSION-$(date +%Y%m%d-%H%M)" -d "$DESCRIPTION"
+newami=$(ec2-register -a $ARCH --kernel $akiid --region $REGION -b "/dev/sda1=$snap1id" -b "/dev/sda2=$snap2id" -n "NetBSD-$ARCH-$VERSION-$(date +%Y%m%d-%H%M)" -d "$DESCRIPTION" | awk '{print $2}')
+
+# Summary
+echo
+echo "A new AMI has been created: $newami"
+echo "You are advised to check that it is bootable by running the"
+echo "following command:"
+echo
+echo " ec2-run-instances $newami -t t1.micro --region $REGION"
+echo
+echo "then make the AMI public if it is bootable:"
+echo " ec2-modify-image-attribute $newami --region $REGION -l -a all"
+echo
+echo "You may also wish to remove any older AMIs, and the snapshots"
+echo "associated with them, if appropriate."
+echo
Index: othersrc/share/examples/ec2/create_ec2_ami.sh
diff -u /dev/null othersrc/share/examples/ec2/create_ec2_ami.sh:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/create_ec2_ami.sh Fri Apr 20 21:01:03 2012
@@ -0,0 +1,318 @@
+#!/bin/sh
+
+# This script is used to:
+# - query for current state of NetBSD AMIs (when executed without arguments)
+# - update an image disk and create an AMI remotely
+
+# In update mode, it performs the following steps:
+# - create a volume disk for the / in the proper region
+# - attach a copy of the grub disk to update kernel
+
+# - upload the kernel and update the Grub boot volume
+# - upload the image and create the volume
+# - snapshots the grub and / disks
+# - output identifiers, and exit.
+
+# Assume the base directory of this script is the right place for some
+# stuff.
+if [ "$(dirname $0 | cut -c 1)" != "/" ]; then
+ BASEDIR="$(pwd)/$(dirname $0)"
+else
+ BASEDIR=$(dirname $0)
+fi
+
+
+ARCH="x86_64"
+VERSION="5.1.2"
+DESCRIPTION="built: $(date) using script ${0##*/}"
+KERNEL="/tmp/netbsd"
+# IMGFILE needs to be in a format that gunzip can handle.
+IMGFILE="/tmp/NetBSD-AMI.img.xz"
+IMGSIZE=5 # default size in GiB of final disk image
+
+# Default owner account of public NetBSD AMIs.
+NETBSD_EC2_OWNER=101367081206 # Jeff Rizzo
+
+if [ "$TMPDIR" = "" ]; then
+ export TMPDIR=/tmp
+fi
+
+# load the Amazon EC2 credentials
+. ${BASEDIR}/set_ec2_env.sh
+
+usage() {
+ echo "Usage: ${0##*/} <options> region"
+ echo "options:"
+ echo " -a <arch> set EC2 architecture (i386 or x86_64)"
+ echo " -d <descr> set AMI description"
+ echo " -i <imgfile> location of disk image file"
+ echo " -k <kernel> location of kernel to use"
+ echo " -l List available NetBSD AMIs by region and exit"
+ echo " -m <ami> Use a specific AMI as a base to create new."
+ echo " -s <size> Size of NetBSD volume in GiB"
+ echo " -v <version> version string to add to AMI name"
+ exit 1
+}
+
+listamis() {
+ # Assume we just want information about NetBSD-6 AMIs
+ # The name should contain NetBSD 6.0, at least...
+ # NetBSD 6 is used because resize_ffs and xz are required, and not
+ # available in NetBSD 5.
+ echo "REGION ID NAME ARCH ACCESS"
+ for REGION in $(ec2-describe-regions | awk '{print $2}'); do
+ ec2-describe-images --region $REGION -F name=*NetBSD*6.0* \
+ -o self -o ${NETBSD_EC2_OWNER} | \
+ (
+ while read TYPE ID NAME OWNER AVAIL PUBLIC ARCH VOID; do
+ # Skip all except IMAGEs
+ if [ "$TYPE" != "IMAGE" ]; then continue; fi
+ echo "$REGION $ID $NAME $ARCH $PUBLIC"
+ done
+ )
+ done
+ exit
+}
+
+while getopts a:d:i:k:lm:s:v: opt
+do
+ case $opt in
+ a) ARCH=$OPTARG;;
+ d) DESCRIPTION=$OPTARG;;
+ i) IMGFILE=$OPTARG;;
+ k) KERNEL=$OPTARG;;
+ l) listamis;;
+ m) AMIID=$OPTARG;;
+ s) IMGSIZE=$OPTARG;;
+ v) VERSION=$OPTARG;;
+ \?) usage;;
+ esac
+done
+shift $(expr $OPTIND - 1)
+
+if [ $# -eq 1 ]; then
+ REGION=$1
+else
+ usage
+fi
+
+# Some preliminary checks
+case $KERNEL in
+ ftp:*|http:*)
+ ftp -o ${TMPDIR}/netbsd $KERNEL
+ if [ $? -ne 0 ]; then
+ echo "Could not fetch kernel from $KERNEL"
+ exit 1
+ fi
+ KERNEL=${TMPDIR}/netbsd
+ ;;
+ *)
+ if [ ! -r $KERNEL ]; then
+ echo "Kernel file '$KERNEL' does not exist!"
+ exit 1
+ fi
+esac
+if [ ! -r $IMGFILE ]; then
+ echo "Image files '$IMGFILE' does not exist!"
+ exit 1
+fi
+
+# We are in update mode. Start a micro instance for the region we want.
+# Locate the proper Amazon IMage ID for NetBSD, it's necessarily one the
+# current user "owns" (so we can deregister it later).
+
+# Set AMIID if it was not already set on the command line.
+if [ "$AMIID" = "" ]; then
+ AMIID=$(ec2-describe-images -o self --region $REGION \
+ -o self -o ${NETBSD_EC2_OWNER} \
+ -F name=*NetBSD*6.0* -F architecture=$ARCH |\
+ awk 'NR == 1 {print $2}')
+fi
+
+if [ "$AMIID" = "" ]; then
+ # No NetBSD AMI for this region. Bail out.
+ echo "There is no NetBSD AMI available for '$REGION'."
+ echo "Consult http://wiki.netbsd.org/amazon_ec2/build_your_own_ami/";
+ echo "to see how you can build one from scratch."
+ exit 1
+fi
+
+# Run an instance
+INSTANCE=$(ec2-run-instances $AMIID -k $EC2_SSH_KEYNAME \
+ -t t1.micro --region $REGION)
+INSTANCEID=$(echo $INSTANCE | awk '{print $6}')
+ZONEID=$(echo $INSTANCE | awk '{print $13}')
+
+# We will need the AKI later to generate the new Amazon IMage
+AKIID=$(echo $INSTANCE | awk '{print $14}')
+
+echo "Creating: $ZONEID $AMIID $INSTANCEID $AKIID"
+
+# Create the ffs volume. Will contain the root file-system.
+VOL1ID=$(ec2-create-volume -s $IMGSIZE --region $REGION -z $ZONEID | awk '{print $2}')
+echo "Creating: volume $VOL1ID (${IMGSIZE}GiB)"
+
+# Wait for the instance.
+while true; do
+ ADDRESS=$(ec2-describe-instances $INSTANCEID --region $REGION |\
+ awk '/running/ {print $4}')
+ if [ "$ADDRESS" = "" ]; then
+ echo "Waiting for instance $INSTANCEID..."
+ sleep 30
+ else
+ echo "$INSTANCEID running, address: $ADDRESS"
+ break
+ fi
+done
+
+# Now grab the correct SSH host key. This can take a while, the console
+# output is buffered on EC2's side.
+# By convention, the EC2 initialization script should prefix all its output
+# lines by "ec2: "
+TMPFILE1=$(mktemp /tmp/ec2-ssh-host-keys.XXXXXX)
+TMPFILE2=$(mktemp /tmp/ec2-ssh-host-keys2.XXXXXX)
+TMPFILE3=$(mktemp /tmp/ec2-ssh-host-keys3.XXXXXX)
+LOGFILE="/tmp/system.log.$INSTANCEID"
+trap "rm -f $TMPFILE1 $TMPFILE2 $TMPFILE3; exit 1" INT EXIT QUIT
+COUNT=0
+while true; do
+ ec2-get-console-output $INSTANCEID --region $REGION > $TMPFILE3
+ grep -q 'Setting EC2 hostname' < $TMPFILE3
+ if [ $? -eq 0 ]; then
+ sed -n "s/^ec2: //gp" $TMPFILE3 > $TMPFILE1
+ if grep -q "^-----END SSH HOST KEY FINGERPRINTS-----" \
+ $TMPFILE1; then
+ echo "EC2 instance report:"
+ cat $TMPFILE1
+ break
+ fi
+
+ COUNT=$(($COUNT + 1))
+ if [ $COUNT -gt 5 ]; then
+ echo "Failed contacting instance ($COUNT tries). The system "
+ echo "log is available under:"
+ echo " $LOGFILE"
+ echo "Connect to AWS Console to terminate and clean-up volumes."
+ ec2-get-console-output $INSTANCEID --region $REGION >\
+ $LOGFILE
+ exit 1
+ fi
+ else
+ echo "Waiting for console output of EC2 init script..."
+ sleep 90
+ fi
+done
+
+# Compare the fingerprints between EC2 console output and direct connection.
+FP1=$(awk '/(RSA)/ {print $2}' $TMPFILE1)
+echo "Console fingerprint: $FP1" # XXX DELETE AFTERWARDS
+
+ssh-keyscan $ADDRESS 2>/dev/null > $TMPFILE2
+FP2=$(ssh-keygen -l -f $TMPFILE2 | awk '{print $2}')
+echo "Keyscan fingerprint: $FP2" # XXX DELETE AFTERWARDS
+
+if [ "$FP1" != "$FP2" ]; then
+ echo "Fingerprint mismatch:"
+ echo "Console reported: $FP1"
+ echo "Host keyscan reported: $FP2"
+ exit 1
+fi
+
+# Well, fingerprints matched... Yippeee
+ssh-keygen -R $ADDRESS
+cat $TMPFILE2 >> ~/.ssh/known_hosts
+
+# Figure out what the number of xbd which will attach is.
+# XXX assumes that attachments are in order and none are skipped.
+xnum=$(ssh -i $EC2_SSH_KEY root@$ADDRESS "/sbin/dmesg" | \
+ egrep 'xbd[0-9]+ ' | cut -f 1 -d ' ' | sort -n | \
+ uniq | wc -l | tr -d '[:space:]')
+
+echo "Attaching $VOL1ID to $INSTANCEID"
+ec2-attach-volume $VOL1ID --region $REGION -i $INSTANCEID -d "/dev/sdf" \
+ > /dev/null
+
+# Copy kernel
+echo "Copying $KERNEL to $INSTANCEID"
+scp -i $EC2_SSH_KEY $KERNEL root@$ADDRESS:/grub/boot/netbsd
+
+# Wait for the volume to be attached
+while true; do
+ PRESENT=$(ssh -i $EC2_SSH_KEY root@$ADDRESS \
+ "/sbin/dmesg | /usr/bin/grep ^xbd${xnum}:")
+ if [ "$PRESENT" != "" ]; then
+ break
+ else
+ echo "Waiting for volume $VOL1ID to be attached..."
+ sleep 15
+ fi
+done
+
+# Upload the image file
+echo "Copying $IMGFILE to $INSTANCEID"
+progress -f $IMGFILE ssh -i $EC2_SSH_KEY root@$ADDRESS \
+ "gunzip | dd of=/dev/rxbd${xnum}d bs=1m"
+
+# stretch the fs to the right size
+echo "Resizing volume to full size"
+ssh -i $EC2_SSH_KEY root@$ADDRESS "/sbin/resize_ffs -y /dev/rxbd${xnum}d"
+
+# Everything done. sync and umount.
+echo "Resize done. Unmounting..."
+ssh -i $EC2_SSH_KEY root@$ADDRESS "/bin/sync; /sbin/umount /grub"
+
+# Snapshot the volumes. First, the main one.
+echo "Creating snapshot from vol1id $VOL1ID"
+SNAP2ID=$(ec2-create-snapshot $VOL1ID --region $REGION -d "NetBSD root $VERSION $DESCRIPTION" |\
+ awk '{print $2}')
+
+echo "Snapshot created, snap2id is $SNAP2ID"
+
+# We need the ID of the 1GiB volume.
+VOL1ID=$(ec2-describe-instances $INSTANCEID --region $REGION |\
+ awk '($1 == "BLOCKDEVICE" && $2 == "/dev/sda1") {print $3}')
+
+echo "Creating snapshot from vol1id $VOL1ID"
+SNAP1ID=$(ec2-create-snapshot $VOL1ID --region $REGION \
+ -d "Grub NetBSD boot $VERSION $DESCRIPTION" | awk '{print $2}')
+
+echo "Created snapshots: $SNAP1ID (1GiB -- boot volume), $SNAP2ID (${IMGSIZE}GiB -- root)"
+
+# Destroy the instance
+ec2-terminate-instances $INSTANCEID --region $REGION
+
+# Wait for the snapshots to complete
+while true; do
+ SNAP1READY=$(ec2-describe-snapshots $SNAP1ID --region $REGION |\
+ awk '/completed/ {print $4}')
+ SNAP2READY=$(ec2-describe-snapshots $SNAP2ID --region $REGION |\
+ awk '/completed/ {print $4}')
+ if [ "$SNAP1READY" = "" -o "$SNAP2READY" = "" ]; then
+ echo "Waiting for snapshots $SNAP1ID and $SNAP2ID to complete..."
+ sleep 30
+ else
+ break
+ fi
+done
+
+# Register the new AMI
+NEWAMIID=$(ec2-register -a $ARCH --kernel $AKIID --region $REGION \
+ -b "/dev/sda1=$SNAP1ID" -b "/dev/sda2=$SNAP2ID" \
+ -n "NetBSD-$ARCH-$VERSION-$(date +%Y%m%d-%H%M)" \
+ -d "$DESCRIPTION" | awk '{print $2}')
+
+# Obtain the snapshot IDs used to create the old AMI
+SNAPIDS=$(ec2-describe-images $AMIID --region $REGION |\
+ awk '/BLOCKDEVICEMAPPING/ {print $3}')
+
+# Summary:
+echo
+echo "A new AMI has been created: $NEWAMIID"
+echo "You are advised to check that it is bootable by running the"
+echo "following command:"
+echo
+echo " ec2-run-instances $NEWAMIID -t t1.micro --region $REGION"
+echo
+echo "then make the AMI public (if you wish) if it is bootable:"
+echo " ec2-modify-image-attribute $NEWAMIID --region $REGION -l -a all"
+echo
Index: othersrc/share/examples/ec2/set_ec2_env.sh
diff -u /dev/null othersrc/share/examples/ec2/set_ec2_env.sh:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/set_ec2_env.sh Fri Apr 20 21:01:03 2012
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+# Set up these options to allow ec2 tools to work properly.
+
+export EC2_PRIVATE_KEY=$HOME/.ec2/pk-XXXXXXX.pem
+export EC2_CERT=$HOME/.ec2/cert-XXXXXXX.pem
+export EC2_SSH_KEY=$HOME/.ec2/my-key-name
+export EC2_SSH_KEYNAME=my-key-name
Index: othersrc/share/examples/ec2/files/bootstrap.sh
diff -u /dev/null othersrc/share/examples/ec2/files/bootstrap.sh:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/files/bootstrap.sh Fri Apr 20 21:01:03 2012
@@ -0,0 +1,52 @@
+#!/bin/sh
+
+URLROOT="http://ftp.netbsd.org/pub/";
+PKG_PATH="${URLROOT}/pkgsrc/packages/NetBSD/$(uname -m)/$(uname -r|cut -f 1-2 -d .| cut -f 1 -d _)/All"
+PKGIN_CONF="/usr/pkg/etc/pkgin/repositories.conf"
+
+
+usage() {
+ echo "Usage: ${0##*/} [src|pkgsrc|binpkg]"
+ exit
+}
+
+if [ $# -ne 1 ]; then
+ usage
+fi
+
+cd /tmp
+
+case $1 in
+src)
+ echo "Downloading -current src..."
+ ftp -a "$URLROOT/NetBSD/NetBSD-current/tar_files/src.tar.gz"
+ echo "Extracting -current src under /usr..."
+ tar -xzpf src.tar.gz -C /usr/
+ rm -f src.tar.gz
+ ;;
+pkgsrc)
+ echo "Downloading latest pkgsrc stable release..."
+ ftp -a "$URLROOT/pkgsrc/stable/pkgsrc.tar.bz2"
+ echo "Extracting latest pkgsrc stable release under /usr..."
+ tar -xjpf pkgsrc.tar.bz2 -C /usr/
+ rm -f pkgsrc.tar.bz2
+ ;;
+binpkg)
+ echo "Setting up for binary packages with pkgin..."
+ pkg_add ${PKG_PATH}/pkgin
+ if [ $? -ne 0 ] ; then
+ echo "pkgin setup failed"
+ exit 1
+ fi
+ sed -e 's/^[^#].*$/#&/' ${PKGIN_CONF} > ${PKGIN_CONF}.new
+ echo $PKG_PATH >> ${PKGIN_CONF}.new
+ mv ${PKGIN_CONF}.new ${PKGIN_CONF}
+ echo "Updating pkgin database..."
+ /usr/pkg/bin/pkgin update
+ echo
+ echo "You may now use pkgin to manage binary packages."
+ ;;
+*)
+ usage
+ ;;
+esac
Index: othersrc/share/examples/ec2/files/ec2_bootmail
diff -u /dev/null othersrc/share/examples/ec2/files/ec2_bootmail:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/files/ec2_bootmail Fri Apr 20 21:01:03 2012
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+# PROVIDE: ec2_bootmail
+# REQUIRE: mail
+# BEFORE: ec2_firstboot
+
+# Define ec2_bootmail=YES in /etc/rc.conf and create /root/firstboot
+# to enable sending an instance-launched email when the system next boots.
+#
+
+$_rc_subr_loaded . /etc/rc.subr
+
+name="ec2_bootmail"
+rcvar=${name}
+start_cmd="ec2_bootmail_run"
+stop_cmd=":"
+
+ec2_bootmail_run()
+{
+
+ # If this is the first boot, send an email.
+ if [ -f /root/firstboot ]; then
+ (
+ echo "To: ${ec2_bootmail_addr}"
+ echo "From: EC2 Instance <root@localhost>"
+ echo "Subject: Instance launch"
+ echo
+ uname -a
+ echo
+ dmesg
+ ) | sendmail -t
+ fi
+}
+
+load_rc_config $name
+run_rc_command "$1"
Index: othersrc/share/examples/ec2/files/ec2_firstboot
diff -u /dev/null othersrc/share/examples/ec2/files/ec2_firstboot:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/files/ec2_firstboot Fri Apr 20 21:01:03 2012
@@ -0,0 +1,26 @@
+#!/bin/sh
+
+# PROVIDE: ec2_firstboot
+# REQUIRE: mountcritlocal
+
+# This script removes the /root/firstboot marker used by ec2_boot and
+# ec2_ephemeralswap; as such, it doesn't make sense to have it not run.
+: ${ec2_firstboot=YES}
+
+$_rc_subr_loaded . /etc/rc.subr
+
+name="ec2_firstboot"
+rcvar=${name}
+start_cmd="ec2_firstboot_run"
+stop_cmd=":"
+
+ec2_firstboot_run()
+{
+ # The first boot has finished.
+ if [ -f /root/firstboot ]; then
+ rm /root/firstboot
+ fi
+}
+
+load_rc_config $name
+run_rc_command "$1"
Index: othersrc/share/examples/ec2/files/ec2_init
diff -u /dev/null othersrc/share/examples/ec2/files/ec2_init:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/files/ec2_init Fri Apr 20 21:01:03 2012
@@ -0,0 +1,65 @@
+#!/bin/sh
+#
+# PROVIDE: ec2_init
+# REQUIRE: NETWORKING sshd
+# BEFORE: LOGIN
+
+#
+# DO NOT modify the output of this file without understanding the
+# implications. The format is used to check SSH fingerprints during
+# initial boot.
+#
+$_rc_subr_loaded . /etc/rc.subr
+
+name="ec2_init"
+rcvar=${name}
+start_cmd="ec2_init"
+stop_cmd=":"
+
+METADATA_URL="http://169.254.169.254/latest/meta-data/";
+SSH_KEY_URL="public-keys/0/openssh-key"
+HOSTNAME_URL="hostname"
+
+SSH_KEY_FILE="/root/.ssh/authorized_keys"
+
+ec2_init()
+{
+ (
+ umask 022
+ # fetch the key pair from Amazon Web Services
+ EC2_SSH_KEY=$(ftp -o - "${METADATA_URL}${SSH_KEY_URL}")
+
+ if [ -n "$EC2_SSH_KEY" ]; then
+ # A key pair is associated with this instance, add it
+ # to root 'authorized_keys' file
+ mkdir -p $(dirname "$SSH_KEY_FILE")
+ touch "$SSH_KEY_FILE"
+ cd $(dirname "$SSH_KEY_FILE")
+
+ grep -q "$EC2_SSH_KEY" "$SSH_KEY_FILE"
+ if [ $? -ne 0 ]; then
+ echo "ec2: Setting EC2 SSH key pair: ${EC2_SSH_KEY##* }"
+ echo "$EC2_SSH_KEY" >> "$SSH_KEY_FILE"
+ fi
+ fi
+
+ # set hostname
+ HOSTNAME=$(ftp -o - "${METADATA_URL}${HOSTNAME_URL}")
+ echo "ec2: Setting EC2 hostname: ${HOSTNAME}"
+ echo "$HOSTNAME" > /etc/myname
+ hostname "$HOSTNAME"
+
+ # Output the SSH host keys
+ echo "ec2: ###########################################################"
+ echo "ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----"
+ for FILE in /etc/ssh/ssh_host_*_key.pub; do
+ echo -n "ec2: "
+ ssh-keygen -l -f "$FILE"
+ done
+ echo "ec2: -----END SSH HOST KEY FINGERPRINTS-----"
+ echo "ec2: ###########################################################"
+ )
+}
+
+load_rc_config $name
+run_rc_command "$1"
Index: othersrc/share/examples/ec2/files/motd
diff -u /dev/null othersrc/share/examples/ec2/files/motd:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/files/motd Fri Apr 20 21:01:03 2012
@@ -0,0 +1,25 @@
+Welcome to NetBSD - Amazon EC2 image!
+
+This system is running a snapshot of a stable branch of the NetBSD
+operating system, adapted for running on the Amazon EC2 infrastructure.
+
+The environment is very similar to one provided within a typical Xen domU
+installation. It contains a small, autonomous environment (including a
+compiler toolchain) that you can run to build your own system.
+
+The file system is lightly populated so you have plenty of space to play with.
+Should you need a src or pkgsrc tree, please use the "bootstrap" script found
+under /usr to download them. You can also use the script to set up
+binary packages using "pkgin":
+
+ /usr/bootstrap.sh [src|pkgsrc|binpkg]
+
+This AMI sends email to the maintainer on first boot, to help get
+an idea of what is in use at any given time.
+
+You are encouraged to test this image as thoroughly as possible. Should you
+encounter any problem, please report it back to the development team using the
+send-pr(1) utility (requires a working MTA). If yours is not properly set up,
+use the web interface at: http://www.NetBSD.org/support/send-pr.html
+
+Thank you for helping us test and improve NetBSD's quality!
Index: othersrc/share/examples/ec2/files/spec.in
diff -u /dev/null othersrc/share/examples/ec2/files/spec.in:1.1
--- /dev/null Fri Apr 20 21:01:03 2012
+++ othersrc/share/examples/ec2/files/spec.in Fri Apr 20 21:01:03 2012
@@ -0,0 +1,10 @@
+./etc/motd type=file mode=0664 uname=root gname=wheel
+./etc/ifconfig.xennet0 type=file mode=0644 uname=root gname=wheel
+./etc/rc.d/ec2_init type=file mode=0555 uname=root gname=wheel
+./etc/rc.d/ec2_bootmail type=file mode=0555 uname=root gname=wheel
+./etc/rc.d/ec2_firstboot type=file mode=0555 uname=root gname=wheel
+./grub type=dir mode=0755 uname=root gname=wheel
+./kern type=dir mode=0755 uname=root gname=wheel
+./proc type=dir mode=0755 uname=root gname=wheel
+./root/firstboot type=file mode=0644 uname=root gname=wheel
+./usr/bootstrap.sh type=file mode=0555 uname=root gname=wheel
