Hi Alex It could. What we did was wrap the entire handler in a try: block, then in the except section set a instance/module level variable to signal a failed startup (error in the handler)
Then if there was time we would try and allocate enough memory to kill the process. After DEE you might have a grace period of just under second before the instance is killed. If the instance didn't get killed then when n the second request hit the server it would see the instance defined flag then immediately allocate enough memory to kill the instance So it wouldn't really matter what type of request (warmup or normal) that hit the instance Cheers Tim On Saturday, March 29, 2014 12:18:03 AM UTC+8, Alex Burgel wrote: > > Thanks Tim. That was the problem. > > I have warmup requests on, but the first request was a non-warmup loading > request which got a DeadlineExceededError. Your fix probably wouldn't work > since its not a warmup request. > > I found this bug report, marked as Won't Fix. It provides some background > in case anyone else runs into it. Looks like there's no good answer. > > https://code.google.com/p/googleappengine/issues/detail?id=1409 > > --Alex > > On Thursday, March 27, 2014 2:27:15 AM UTC-4, timh wrote: >> >> Go back through the logs for the instance, and see if it had a >> DeadlineExceeded Error during startup. >> >> My guess is you are not using warmup requests, and the instance hasn't >> started properly leaving imports in a bad state. >> >> This used to happen a lot in the early days when startup times in python >> could be unreliable on M/S. >> >> The only way to deal with it at the time (because you couldn't kill >> instances) was to wrap the entire handler in a try block, and if you got an >> import error inside the handler >> I would allocate enough memory to kill the instance. >> >> Cheers >> >> Tim >> >> On Thursday, March 27, 2014 9:42:27 AM UTC+8, Alex Burgel wrote: >>> >>> Every so often (once a month or less), I get an instance that just >>> doesn't work. Every request fails with some error in library code that >>> indicates something is profoundly screwed up, like an import fails or some >>> line of obviously correct code throws an exception. >>> >>> When this has happened in the past, I would get a bunch of error emails >>> and I would just shutdown the instance and hope that it was a fluke. Today >>> it happened while I was on a flight, so for about 3 hours, this broken >>> instance was returning errors for most of the requests hitting my app. >>> >>> This is the exception (app named changed to protect the innocent): >>> >>> Traceback (most recent call last): >>> >>> File >>> "/base/data/home/apps/s~xxx/8.374612775992938880/django/core/handlers/base.py", >>> >>> line 101, in get_response >>> resolver_match = resolver.resolve(request.path_info) >>> >>> File >>> "/base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py", >>> >>> line 318, in resolve >>> for pattern in self.url_patterns: >>> >>> File >>> "/base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py", >>> >>> line 346, in url_patterns >>> patterns = getattr(self.urlconf_module, "urlpatterns", >>> self.urlconf_module) >>> >>> File >>> "/base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py", >>> >>> line 341, in urlconf_module >>> self._urlconf_module = import_module(self.urlconf_name) >>> >>> File >>> "/base/data/home/apps/s~xxx/8.374612775992938880/django/utils/importlib.py", >>> >>> line 40, in import_module >>> __import__(name) >>> >>> File "/base/data/home/apps/s~xxx/8.374612775992938880/urls.py", line >>> 4, in <module> >>> from swaagit import admin as swaag_admin, sites >>> >>> File "/base/data/home/apps/s~xx/8.374612775992938880/xxx/admin.py", >>> line 12, in <module> >>> from mapreduce.site import site as mapreduce_site >>> >>> File >>> "/base/data/home/apps/s~xxx/8.374612775992938880/xxx/mapreduce/site.py", >>> line 5, in <module> >>> from mapreduce import base_handler >>> >>> File >>> "/base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/base_handler.py", >>> >>> line 44, in <module> >>> from mapreduce import model >>> >>> File >>> "/base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/model.py", >>> line 61, in <module> >>> from mapreduce import context >>> >>> File >>> "/base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/context.py", >>> line 40, in <module> >>> from google.appengine.ext import ndb >>> >>> File "/base/data/home/runtimes/python27/python27_lib/versions/1/ >>> google/appengine/ext/ndb/__init__.py", line 8, in <module> >>> __all__ += tasklets.__all__ >>> >>> NameError: name 'tasklets' is not defined >>> >>> Clearly a NameError in SDK code means that something is very wrong. Yet, >>> this instance happily continued serving traffic for hours. >>> >>> Are there not some smoke tests or verification tests that are run before >>> an instance is put into production? How does everybody else deal with this >>> problem? >>> >>> --Alex >>> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
