[Bug 634102] Re: t1.micro instances hang on reboot
** Branch linked: lp:ubuntu/cloud-init -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
** Branch linked: lp:~cloud-init-dev/cloud-init/maverick -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
** Changed in: cloud-init (Ubuntu Maverick) Milestone: None => ubuntu-10.10 -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
** Tags added: server-mrs -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
** Branch linked: lp:ubuntu/lucid-proposed/cloud-init ** Branch linked: lp:cloud-init -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
** Description changed: Binary package hint: cloud-init - on Amazon's new t1. instance type, there is no ephemeral storage at all. - If you run a ubuntu ebs image on instance type t1.micro and reboot, it - will not come back up. mountall will wait indefinitely for /dev/sda2, - which is never going to be present. + on Amazon's new t1.micro instance type, there is no ephemeral storage at + all. If you run a ubuntu ebs image on instance type t1.micro and + reboot, it will not come back up. mountall will wait indefinitely for + /dev/sda2, which is never going to be present. cloud-init is basically hard coded to expect an 'ephemeral0', while other ephemeral devices are more dynamic. Our images are registered with block-device-mapping indicating ephemeral0, so the metadata service will include ephemeral0 even though there is not one on the instance itself. We need to do one of 2 things here: a.) add 'nobootwait' for the ephemeral0 device (/dev/sda2 in this case) b.) not write a entry in /etc/fstab (or comment it out) if that device is not present on the first boot. There are 2 easy workarounds for this: 1.) sudo sed -i.dist '\,/dev/sda2,s,^,#,' /etc/fstab 2.) launch instance with cloud-config metadata containing: #cloud-config mounts: - [ ephemeral0 ] ### SRU Information BEGIN 1. This bug affects anyone who is going to run an ec2 instance of type t1.micro . It is expected that this will be lots of people, especially those evaluating EC2 and/or Ubuntu. The bug is that the system will only boot and be reachable one time. On subsequent boots, the ssh service will not start, leaving a cloud instance completely unreachable. That is because on first boot an entry is written to /etc/fstab that will never be present. - 2. The bug if fixed by - a.) carefully updating existing entries in /etc/fstab to add 'nobootwait'. Only ephemeral devices are modified (either /dev/sda2 or /dev/sdb), and only if they contain 'comment=cloudconfig'. - b.) on future first-boots, writing 'nobootwait' for the entry. + 2. The bug if fixed by + a.) carefully updating existing entries in /etc/fstab to add 'nobootwait'. Only ephemeral devices are modified (either /dev/sda2 or /dev/sdb), and only if they contain 'comment=cloudconfig'. + b.) on future first-boots, writing 'nobootwait' for the entry. 3. The patch is available at lp:~cloud-init-dev/cloud-init/lucid, in changes seen at http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/lucid/revision/19?remember=15&compare_revid=15 4. To reproduce: - a.) start ec2 lucid instance of t1.micro - ec2-run-instances --region us-east-1 --key mykey ami-1437dd7d - b.) ssh to instance and reboot - sudo reboot - c.) ssh will not come up, leaving the instance basically dead. + a.) start ec2 lucid instance of t1.micro + ec2-run-instances --region us-east-1 --key mykey ami-1437dd7d + b.) ssh to instance and reboot + sudo reboot + c.) ssh will not come up, leaving the instance basically dead. 5. The opportunity for regression is almost completely contained in the pre-install script, and here it is very small. The only real negative fallout would be adding 'nobootwait' to an entry in /etc/fstab that the user *wanted* to wait on. This is very unlkely. # SRU Information END ## - ProblemType: Bug DistroRelease: Ubuntu 10.04 Package: cloud-init 0.5.10-0ubuntu1.2 ProcVersionSignature: User Name 2.6.32-308.15-ec2 2.6.32.15+drm33.5 Uname: Linux 2.6.32-308-ec2 i686 Architecture: i386 Date: Thu Sep 9 14:42:21 2010 Ec2AMI: ami-1234de7b Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1c Ec2InstanceType: t1.micro Ec2Kernel: aki-5037dd39 Ec2Ramdisk: unavailable PackageArchitecture: all ProcEnviron: PATH=(custom, user) LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: cloud-init -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 634102] Re: t1.micro instances hang on reboot
On Thu, 9 Sep 2010, Ben Howard wrote: > This looks like a problem with the block mapping when the AMI is > registered. Well, yes, sort of. All Ubuntu images are registered with '--block-device-mapping' for the smallest type of the given arch. See lines 391 at [1]. I recently bumped into this and added a comment to that effect at [2]. I had actually considered beginning to register images with 'ephemeral0' through 'ephemeral4', so that by default, the user would at least get all the ephemeral storage they could potentially have. > After firing up an t1-miro instance, the Metadata shows: > block-device-mapping: > ami: /dev/sda1 > ephemeral0: /dev/sda2 > root: /dev/sda1 So, yeah, this is the first instance type where this assumption is now wrong, and 'ephemeral0' device as reported in the metadata service wont be there. The fix that I've added is just to add 'nobootwait' to the entry that is added in /etc/fstab. That way, reboot wont hang, and if the user restarted in a larger instance of the given type, they'd get some ephemeral data. -- [1] http://bazaar.launchpad.net/~ubuntu-on-ec2/ubuntu-on-ec2/ec2-publishing-scripts/annotate/head%3A/ec2-image2ebs [2] http://bazaar.launchpad.net/~ubuntu-on-ec2/ubuntu-on-ec2/ec2-publishing-scripts/revision/239 -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
** Description changed: Binary package hint: cloud-init - on Amazon's new t1.micro instance type, there is no ephemeral storage at - all. If you run a ubuntu ebs image on instance type t1.micro and - reboot, it will not come back up. mountall will wait indefinitely for - /dev/sda2, which is never going to be present. + on Amazon's new t1. instance type, there is no ephemeral storage at all. + If you run a ubuntu ebs image on instance type t1.micro and reboot, it + will not come back up. mountall will wait indefinitely for /dev/sda2, + which is never going to be present. cloud-init is basically hard coded to expect an 'ephemeral0', while other ephemeral devices are more dynamic. Our images are registered with block-device-mapping indicating ephemeral0, so the metadata service will include ephemeral0 even though there is not one on the instance itself. We need to do one of 2 things here: a.) add 'nobootwait' for the ephemeral0 device (/dev/sda2 in this case) b.) not write a entry in /etc/fstab (or comment it out) if that device is not present on the first boot. There are 2 easy workarounds for this: 1.) sudo sed -i.dist '\,/dev/sda2,s,^,#,' /etc/fstab 2.) launch instance with cloud-config metadata containing: #cloud-config mounts: - [ ephemeral0 ] + + ### SRU Information BEGIN + 1. This bug affects anyone who is going to run an ec2 instance of type t1.micro . It is expected that this will be lots of people, especially those evaluating EC2 and/or Ubuntu. The bug is that the system will only boot and be reachable one time. On subsequent boots, the ssh service will not start, leaving a cloud instance completely unreachable. That is because on first boot an entry is written to /etc/fstab that will never be present. + 2. The bug if fixed by + a.) carefully updating existing entries in /etc/fstab to add 'nobootwait'. Only ephemeral devices are modified (either /dev/sda2 or /dev/sdb), and only if they contain 'comment=cloudconfig'. + b.) on future first-boots, writing 'nobootwait' for the entry. + 3. The patch is available at lp:~cloud-init-dev/cloud-init/lucid, in changes seen at http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/lucid/revision/19?remember=15&compare_revid=15 + 4. To reproduce: + a.) start ec2 lucid instance of t1.micro + ec2-run-instances --region us-east-1 --key mykey ami-1437dd7d + b.) ssh to instance and reboot + sudo reboot + c.) ssh will not come up, leaving the instance basically dead. + 5. The opportunity for regression is almost completely contained in the pre-install script, and here it is very small. The only real negative fallout would be adding 'nobootwait' to an entry in /etc/fstab that the user *wanted* to wait on. This is very unlkely. + # SRU Information END ## + ProblemType: Bug DistroRelease: Ubuntu 10.04 Package: cloud-init 0.5.10-0ubuntu1.2 ProcVersionSignature: User Name 2.6.32-308.15-ec2 2.6.32.15+drm33.5 Uname: Linux 2.6.32-308-ec2 i686 Architecture: i386 Date: Thu Sep 9 14:42:21 2010 Ec2AMI: ami-1234de7b Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1c Ec2InstanceType: t1.micro Ec2Kernel: aki-5037dd39 Ec2Ramdisk: unavailable PackageArchitecture: all ProcEnviron: PATH=(custom, user) LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: cloud-init -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
Sorry...the previous comment should read: After firing up an t1-miro instance with Lucid, the Metadata shows: block-device-mapping: ami: /dev/sda1 ephemeral0: /dev/sda2 root: /dev/sda1 While one that does not have an ephemeral mapping shows: block-device-mapping: ami: /dev/sda1 root: /dev/sda1 -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
This looks like a problem with the block mapping when the AMI is registered. After firing up an t1-miro instance, the Metadata shows: block-device-mapping: ami: /dev/sda1 ephemeral0: /dev/sda2 root: /dev/sda1 While one that does have an ephemeral mapping shows: block-device-mapping: ami: /dev/sda1 root: /dev/sda1 -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
** Changed in: cloud-init (Ubuntu Lucid) Importance: Undecided => High ** Changed in: cloud-init (Ubuntu Lucid) Status: New => Triaged ** Changed in: cloud-init (Ubuntu Lucid) Milestone: None => lucid-updates ** Description changed: Binary package hint: cloud-init - on Amazon's new t1. instance type, there is no ephemeral storage at all. - If you run a ubuntu ebs image on instance type t1.micro and reboot, it - will not come back up. mountall will wait indefinitely for /dev/sda2, - which is never going to be present. + on Amazon's new t1.micro instance type, there is no ephemeral storage at + all. If you run a ubuntu ebs image on instance type t1.micro and + reboot, it will not come back up. mountall will wait indefinitely for + /dev/sda2, which is never going to be present. cloud-init is basically hard coded to expect an 'ephemeral0', while other ephemeral devices are more dynamic. Our images are registered with block-device-mapping indicating ephemeral0, so the metadata service will include ephemeral0 even though there is not one on the instance itself. We need to do one of 2 things here: a.) add 'nobootwait' for the ephemeral0 device (/dev/sda2 in this case) b.) not write a entry in /etc/fstab (or comment it out) if that device is not present on the first boot. There are 2 easy workarounds for this: 1.) sudo sed -i.dist '\,/dev/sda2,s,^,#,' /etc/fstab 2.) launch instance with cloud-config metadata containing: #cloud-config mounts: - [ ephemeral0 ] ProblemType: Bug DistroRelease: Ubuntu 10.04 Package: cloud-init 0.5.10-0ubuntu1.2 ProcVersionSignature: User Name 2.6.32-308.15-ec2 2.6.32.15+drm33.5 Uname: Linux 2.6.32-308-ec2 i686 Architecture: i386 Date: Thu Sep 9 14:42:21 2010 Ec2AMI: ami-1234de7b Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1c Ec2InstanceType: t1.micro Ec2Kernel: aki-5037dd39 Ec2Ramdisk: unavailable PackageArchitecture: all ProcEnviron: PATH=(custom, user) LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: cloud-init -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
** Also affects: cloud-init (Ubuntu Maverick) Importance: Undecided Status: New ** Changed in: cloud-init (Ubuntu Maverick) Importance: Undecided => Medium ** Changed in: cloud-init (Ubuntu Maverick) Status: New => Confirmed ** Changed in: cloud-init (Ubuntu Maverick) Assignee: (unassigned) => Scott Moser (smoser) -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 634102] Re: t1.micro instances hang on reboot
** Also affects: cloud-init (Ubuntu Lucid) Importance: Undecided Status: New -- t1.micro instances hang on reboot https://bugs.launchpad.net/bugs/634102 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
