Public bug reported:
Here's the deal: running a stress test, 600 instances. Life is good,
pretty much successes, with the (very) rare failure. Suddenly, pretty
much all we get are failures to start new instances. If I keep on
running, *no* new instance succeeds: they all go into pending, stay
there for a while, and are terminated by timeout.
A quick look at the NC logs shows an error like "Failed to prepare
images for instance <whatever> (error 1)":
On sapodilla:
ubu...@sapodilla:/var/log/eucalyptus$ grep "Failed to prepare images" *
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-50EC08E8 (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-51EF08FC (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-51EF08FC (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance ��d8� (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance `�8� (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-4FC10831 (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-4A8B0811 (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance P�{8� (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-43C50871 (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-456D0882 (error=1)
nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-37B107A3 (error=1)
nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-50EC08E8 (error=1)
nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-397606C6 (error=1)
nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-49B0087A (error=1)
nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-4A8B0811 (error=1)
nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-51EF08FC (error=1)
nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-50EC08E8 (error=1)
nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-33C805F7 (error=1)
nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images
for instance i-46AC06F8 (error=1)
nc.log:[Wed Aug 18 10:37:21 2010][001992][EUCAFATAL ] Failed to prepare images
for instance � (error=1)
ubu...@sapodilla:/var/log/eucalyptus$
An equivalent list can be found on the other test NC.
Following one such instance, we can see (from the nc.log):
[Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] doRunInstance() invoked
(id=i-50EC08E8 cores=1 disk=5 memory=256)
[Wed Aug 18 10:22:08 2010][001992][EUCAINFO ]
image=emi-3C651C4F at
http://10.55.55.5:8773/services/Walrus/maverick-20100817-amd64-20100818080418/maverick-server-uec-amd64.img.manifest.xml
[Wed Aug 18 10:22:08 2010][001992][EUCAINFO ]
krnel=eki-D9512160 at
http://10.55.55.5:8773/services/Walrus/maverick-20100817-amd64-20100818080418/maverick-server-uec-amd64-vmlinuz-virtual.manifest.xml
[Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] vlan=12
priMAC=D0:0D:50:EC:08:E8 privIp=172.19.5.5
[Wed Aug 18 10:22:08 2010][001992][EUCADEBUG ] state change for instance
i-50EC08E8: Unknown -> Staging (Pending)
[Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] network started for instance
i-50EC08E8
[Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] retrieving images for instance
i-50EC08E8 (disk limit=5120MB)...
...
[Wed Aug 18 10:37:11 2010][001992][EUCAERROR ] error: file
/var/lib/eucalyptus/instances//eucalyptus/cache/eki-D9512160/kernel not found
[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for
instance i-50EC08E8 (error=1)
[Wed Aug 18 10:37:11 2010][001992][EUCAERROR ] [Wed Aug 18 10:37:11
2010][001992][EUCADEBUG ] state change for instance i-50EC08E8: Teardown ->
Shutoff (Extant)
I am not sure this is a cause or consequence of a previous error. Logs
are being saved.
** Affects: eucalyptus (Ubuntu)
Importance: High
Status: New
** Changed in: eucalyptus (Ubuntu)
Importance: Undecided => High
--
suddenly no instance starts anymore - walrus issue?
https://bugs.launchpad.net/bugs/619970
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
