Public bug reported: Here's the deal: running a stress test, 600 instances. Life is good, pretty much successes, with the (very) rare failure. Suddenly, pretty much all we get are failures to start new instances. If I keep on running, *no* new instance succeeds: they all go into pending, stay there for a while, and are terminated by timeout.
A quick look at the NC logs shows an error like "Failed to prepare images for instance <whatever> (error 1)": On sapodilla: ubu...@sapodilla:/var/log/eucalyptus$ grep "Failed to prepare images" * nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-50EC08E8 (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-51EF08FC (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-51EF08FC (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance ��d8� (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance `�8� (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-4FC10831 (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-4A8B0811 (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance P�{8� (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-43C50871 (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-456D0882 (error=1) nc.log:[Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-37B107A3 (error=1) nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-50EC08E8 (error=1) nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-397606C6 (error=1) nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-49B0087A (error=1) nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-4A8B0811 (error=1) nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-51EF08FC (error=1) nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-50EC08E8 (error=1) nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-33C805F7 (error=1) nc.log:[Wed Aug 18 10:37:12 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-46AC06F8 (error=1) nc.log:[Wed Aug 18 10:37:21 2010][001992][EUCAFATAL ] Failed to prepare images for instance � (error=1) ubu...@sapodilla:/var/log/eucalyptus$ An equivalent list can be found on the other test NC. Following one such instance, we can see (from the nc.log): [Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] doRunInstance() invoked (id=i-50EC08E8 cores=1 disk=5 memory=256) [Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] image=emi-3C651C4F at http://10.55.55.5:8773/services/Walrus/maverick-20100817-amd64-20100818080418/maverick-server-uec-amd64.img.manifest.xml [Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] krnel=eki-D9512160 at http://10.55.55.5:8773/services/Walrus/maverick-20100817-amd64-20100818080418/maverick-server-uec-amd64-vmlinuz-virtual.manifest.xml [Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] vlan=12 priMAC=D0:0D:50:EC:08:E8 privIp=172.19.5.5 [Wed Aug 18 10:22:08 2010][001992][EUCADEBUG ] state change for instance i-50EC08E8: Unknown -> Staging (Pending) [Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] network started for instance i-50EC08E8 [Wed Aug 18 10:22:08 2010][001992][EUCAINFO ] retrieving images for instance i-50EC08E8 (disk limit=5120MB)... ... [Wed Aug 18 10:37:11 2010][001992][EUCAERROR ] error: file /var/lib/eucalyptus/instances//eucalyptus/cache/eki-D9512160/kernel not found [Wed Aug 18 10:37:11 2010][001992][EUCAFATAL ] Failed to prepare images for instance i-50EC08E8 (error=1) [Wed Aug 18 10:37:11 2010][001992][EUCAERROR ] [Wed Aug 18 10:37:11 2010][001992][EUCADEBUG ] state change for instance i-50EC08E8: Teardown -> Shutoff (Extant) I am not sure this is a cause or consequence of a previous error. Logs are being saved. ** Affects: eucalyptus (Ubuntu) Importance: High Status: New ** Changed in: eucalyptus (Ubuntu) Importance: Undecided => High -- suddenly no instance starts anymore - walrus issue? https://bugs.launchpad.net/bugs/619970 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs