[google-appengine] Re: What to do with a bad instance?
Thanks Tim. That was the problem. I have warmup requests on, but the first request was a non-warmup loading request which got a DeadlineExceededError. Your fix probably wouldn't work since its not a warmup request. I found this bug report, marked as Won't Fix. It provides some background in case anyone else runs into it. Looks like there's no good answer. https://code.google.com/p/googleappengine/issues/detail?id=1409 --Alex On Thursday, March 27, 2014 2:27:15 AM UTC-4, timh wrote: Go back through the logs for the instance, and see if it had a DeadlineExceeded Error during startup. My guess is you are not using warmup requests, and the instance hasn't started properly leaving imports in a bad state. This used to happen a lot in the early days when startup times in python could be unreliable on M/S. The only way to deal with it at the time (because you couldn't kill instances) was to wrap the entire handler in a try block, and if you got an import error inside the handler I would allocate enough memory to kill the instance. Cheers Tim On Thursday, March 27, 2014 9:42:27 AM UTC+8, Alex Burgel wrote: Every so often (once a month or less), I get an instance that just doesn't work. Every request fails with some error in library code that indicates something is profoundly screwed up, like an import fails or some line of obviously correct code throws an exception. When this has happened in the past, I would get a bunch of error emails and I would just shutdown the instance and hope that it was a fluke. Today it happened while I was on a flight, so for about 3 hours, this broken instance was returning errors for most of the requests hitting my app. This is the exception (app named changed to protect the innocent): Traceback (most recent call last): File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/handlers/base.py, line 101, in get_response resolver_match = resolver.resolve(request.path_info) File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py, line 318, in resolve for pattern in self.url_patterns: File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py, line 346, in url_patterns patterns = getattr(self.urlconf_module, urlpatterns, self.urlconf_module) File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py, line 341, in urlconf_module self._urlconf_module = import_module(self.urlconf_name) File /base/data/home/apps/s~xxx/8.374612775992938880/django/utils/importlib.py, line 40, in import_module __import__(name) File /base/data/home/apps/s~xxx/8.374612775992938880/urls.py, line 4, in module from swaagit import admin as swaag_admin, sites File /base/data/home/apps/s~xx/8.374612775992938880/xxx/admin.py, line 12, in module from mapreduce.site import site as mapreduce_site File /base/data/home/apps/s~xxx/8.374612775992938880/xxx/mapreduce/site.py, line 5, in module from mapreduce import base_handler File /base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/base_handler.py, line 44, in module from mapreduce import model File /base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/model.py, line 61, in module from mapreduce import context File /base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/context.py, line 40, in module from google.appengine.ext import ndb File /base/data/home/runtimes/python27/python27_lib/versions/1/ google/appengine/ext/ndb/__init__.py, line 8, in module __all__ += tasklets.__all__ NameError: name 'tasklets' is not defined Clearly a NameError in SDK code means that something is very wrong. Yet, this instance happily continued serving traffic for hours. Are there not some smoke tests or verification tests that are run before an instance is put into production? How does everybody else deal with this problem? --Alex -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: What to do with a bad instance?
Hi Alex It could. What we did was wrap the entire handler in a try: block, then in the except section set a instance/module level variable to signal a failed startup (error in the handler) Then if there was time we would try and allocate enough memory to kill the process. After DEE you might have a grace period of just under second before the instance is killed. If the instance didn't get killed then when n the second request hit the server it would see the instance defined flag then immediately allocate enough memory to kill the instance So it wouldn't really matter what type of request (warmup or normal) that hit the instance Cheers Tim On Saturday, March 29, 2014 12:18:03 AM UTC+8, Alex Burgel wrote: Thanks Tim. That was the problem. I have warmup requests on, but the first request was a non-warmup loading request which got a DeadlineExceededError. Your fix probably wouldn't work since its not a warmup request. I found this bug report, marked as Won't Fix. It provides some background in case anyone else runs into it. Looks like there's no good answer. https://code.google.com/p/googleappengine/issues/detail?id=1409 --Alex On Thursday, March 27, 2014 2:27:15 AM UTC-4, timh wrote: Go back through the logs for the instance, and see if it had a DeadlineExceeded Error during startup. My guess is you are not using warmup requests, and the instance hasn't started properly leaving imports in a bad state. This used to happen a lot in the early days when startup times in python could be unreliable on M/S. The only way to deal with it at the time (because you couldn't kill instances) was to wrap the entire handler in a try block, and if you got an import error inside the handler I would allocate enough memory to kill the instance. Cheers Tim On Thursday, March 27, 2014 9:42:27 AM UTC+8, Alex Burgel wrote: Every so often (once a month or less), I get an instance that just doesn't work. Every request fails with some error in library code that indicates something is profoundly screwed up, like an import fails or some line of obviously correct code throws an exception. When this has happened in the past, I would get a bunch of error emails and I would just shutdown the instance and hope that it was a fluke. Today it happened while I was on a flight, so for about 3 hours, this broken instance was returning errors for most of the requests hitting my app. This is the exception (app named changed to protect the innocent): Traceback (most recent call last): File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/handlers/base.py, line 101, in get_response resolver_match = resolver.resolve(request.path_info) File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py, line 318, in resolve for pattern in self.url_patterns: File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py, line 346, in url_patterns patterns = getattr(self.urlconf_module, urlpatterns, self.urlconf_module) File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py, line 341, in urlconf_module self._urlconf_module = import_module(self.urlconf_name) File /base/data/home/apps/s~xxx/8.374612775992938880/django/utils/importlib.py, line 40, in import_module __import__(name) File /base/data/home/apps/s~xxx/8.374612775992938880/urls.py, line 4, in module from swaagit import admin as swaag_admin, sites File /base/data/home/apps/s~xx/8.374612775992938880/xxx/admin.py, line 12, in module from mapreduce.site import site as mapreduce_site File /base/data/home/apps/s~xxx/8.374612775992938880/xxx/mapreduce/site.py, line 5, in module from mapreduce import base_handler File /base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/base_handler.py, line 44, in module from mapreduce import model File /base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/model.py, line 61, in module from mapreduce import context File /base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/context.py, line 40, in module from google.appengine.ext import ndb File /base/data/home/runtimes/python27/python27_lib/versions/1/ google/appengine/ext/ndb/__init__.py, line 8, in module __all__ += tasklets.__all__ NameError: name 'tasklets' is not defined Clearly a NameError in SDK code means that something is very wrong. Yet, this instance happily continued serving traffic for hours. Are there not some smoke tests or verification tests that are run before an instance is put into production? How does everybody else deal with this problem? --Alex -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to
[google-appengine] Re: What to do with a bad instance?
Go back through the logs for the instance, and see if it had a DeadlineExceeded Error during startup. My guess is you are not using warmup requests, and the instance hasn't started properly leaving imports in a bad state. This used to happen a lot in the early days when startup times in python could be unreliable on M/S. The only way to deal with it at the time (because you couldn't kill instances) was to wrap the entire handler in a try block, and if you got an import error inside the handler I would allocate enough memory to kill the instance. Cheers Tim On Thursday, March 27, 2014 9:42:27 AM UTC+8, Alex Burgel wrote: Every so often (once a month or less), I get an instance that just doesn't work. Every request fails with some error in library code that indicates something is profoundly screwed up, like an import fails or some line of obviously correct code throws an exception. When this has happened in the past, I would get a bunch of error emails and I would just shutdown the instance and hope that it was a fluke. Today it happened while I was on a flight, so for about 3 hours, this broken instance was returning errors for most of the requests hitting my app. This is the exception (app named changed to protect the innocent): Traceback (most recent call last): File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/handlers/base.py, line 101, in get_response resolver_match = resolver.resolve(request.path_info) File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py, line 318, in resolve for pattern in self.url_patterns: File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py, line 346, in url_patterns patterns = getattr(self.urlconf_module, urlpatterns, self.urlconf_module) File /base/data/home/apps/s~xxx/8.374612775992938880/django/core/urlresolvers.py, line 341, in urlconf_module self._urlconf_module = import_module(self.urlconf_name) File /base/data/home/apps/s~xxx/8.374612775992938880/django/utils/importlib.py, line 40, in import_module __import__(name) File /base/data/home/apps/s~xxx/8.374612775992938880/urls.py, line 4, in module from swaagit import admin as swaag_admin, sites File /base/data/home/apps/s~xx/8.374612775992938880/xxx/admin.py, line 12, in module from mapreduce.site import site as mapreduce_site File /base/data/home/apps/s~xxx/8.374612775992938880/xxx/mapreduce/site.py, line 5, in module from mapreduce import base_handler File /base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/base_handler.py, line 44, in module from mapreduce import model File /base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/model.py, line 61, in module from mapreduce import context File /base/data/home/apps/s~xxx/8.374612775992938880/mapreduce/context.py, line 40, in module from google.appengine.ext import ndb File /base/data/home/runtimes/python27/python27_lib/versions/1/ google/appengine/ext/ndb/__init__.py, line 8, in module __all__ += tasklets.__all__ NameError: name 'tasklets' is not defined Clearly a NameError in SDK code means that something is very wrong. Yet, this instance happily continued serving traffic for hours. Are there not some smoke tests or verification tests that are run before an instance is put into production? How does everybody else deal with this problem? --Alex -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
