[google-appengine] Outage of site due to "cannot import name datastore_stub_util"
This is a request for help from someone at google.
Appid: fpgamesserver.appspot.com
At 3.53pm on Oct 13 we had an error that said "cannot import name
datastore_stub_util" due to the launch of a new process. It's not
clear why a new process was launched since the previous requests were
not CPU intensive. It's possible that this one request was, but I
can't quite tell why. Every subsequent request to my application
results in an error: 'NoneType' object has no attribute 'exception'.
This error references a line main.py which says:
logging.exception('Exception in request:')
Requests to the site (fpgamesserver.appspot.com/heist) return this
error:
"Error: Server Error
The server encountered an error and could not complete your request.
If the problem persists, please report your problem and mention this
error message and the query that caused it."
The first error that seems to have been triggered by the launch of a
new process is:
10-13 03:54PM 39.280 /heist/ 500 2722ms 1108cpu_ms 0kb Mozilla/
5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.10) Gecko/
20100914 Firefox/3.6.10,gzip(gfe)
See details
65.46.75.178 - - [13/Oct/2010:15:54:42 -0700] "GET /heist/ HTTP/
1.1" 500 0 - "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10,gzip(gfe)"
"fpgamesserver.appspot.com" ms=2722 cpu_ms=1108 api_cpu_ms=0
cpm_usd=0.030836 loading_request=1
D 10-13 03:54PM 41.340
FILTER_LIST: [{'PATH_INFO': '!.*/admin/.*'}]
D 10-13 03:54PM 41.341
Match on {'PATH_INFO': '!.*/admin/.*'}
I 10-13 03:54PM 41.537
Hook registered
I 10-13 03:54PM 41.538
DatastoreProxy initialized
E 10-13 03:54PM 41.912
Exception in request:
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/core/handlers/base.py", line 68, in get_response
callback, callback_args, callback_kwargs =
resolver.resolve(request.path)
File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/core/urlresolvers.py", line 162, in resolve
sub_match = pattern.resolve(new_path)
File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/core/urlresolvers.py", line 118, in resolve
return self.callback, args, kwargs
File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/core/urlresolvers.py", line 127, in _get_callback
raise ViewDoesNotExist, "Could not import %s. Error was: %s"
% (mod_name, str(e))
ViewDoesNotExist: Could not import freshplanet.framework.views.
Error was: cannot import name datastore_stub_util
E 10-13 03:54PM 41.993
: Tried
freshplanet.framework.views.get500. Error was: cannot import name
datastore_stub_util
Traceback (most recent call last):
File "/base/data/home/apps/fpgamesserver/0-0-4-first-caper-
online.345419715554941267/main.py", line 50, in
main()
File "/base/data/home/apps/fpgamesserver/0-0-4-first-caper-
online.345419715554941267/main.py", line 47, in main
util.run_wsgi_app(application)
File "/base/python_runtime/python_lib/versions/1/google/
appengine/ext/webapp/util.py", line 97, in run_wsgi_app
run_bare_wsgi_app(add_wsgi_middleware(application))
File "/base/python_runtime/python_lib/versions/1/google/
appengine/ext/webapp/util.py", line 115, in run_bare_wsgi_app
result = application(env, _start_response)
File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/core/handlers/wsgi.py", line 189, in __call__
response = self.get_response(request)
File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/core/handlers/base.py", line 125, in get_response
callback, param_dict = resolver.resolve500()
File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/core/urlresolvers.py", line 200, in resolve500
return self._resolve_special('500')
File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/core/urlresolvers.py", line 194, in
_resolve_special
raise ViewDoesNotExist, "Tried %s. Error was: %s" %
(callback, str(e))
I 10-13 03:54PM 42.000
This request caused a new process to be started for your
application, and thus caused your application code to be loaded for
the first time. This request may thus take longer and use more CPU
than a typical request for your application.
It's not clear why a new process was launched since the previous
request was at 3.46pm and does not seem to be overloaded. But it's
possible that the previous request to that, which took 1380cpu_ms
triggered this process to launch?
Also, why is this process corrupted and is there a way to recover it
automatically or avoid this sort of outage? Is this something that I'm
just not handling corre
Re: [google-appengine] Outage of site due to "cannot import name datastore_stub_util"
You might try to deploy your code to a new version, then set the new
version to default. That will cause your app to reload.
I think Tim has a good idea, deploy the same code to two versions.
Then when you get a stuck / corrupt instance you can switch the
default version and be back in business.
Robert
On Thu, Oct 14, 2010 at 10:22, cybertoast wrote:
> This is a request for help from someone at google.
>
> Appid: fpgamesserver.appspot.com
> At 3.53pm on Oct 13 we had an error that said "cannot import name
> datastore_stub_util" due to the launch of a new process. It's not
> clear why a new process was launched since the previous requests were
> not CPU intensive. It's possible that this one request was, but I
> can't quite tell why. Every subsequent request to my application
> results in an error: 'NoneType' object has no attribute 'exception'.
> This error references a line main.py which says:
> logging.exception('Exception in request:')
> Requests to the site (fpgamesserver.appspot.com/heist) return this
> error:
> "Error: Server Error
> The server encountered an error and could not complete your request.
> If the problem persists, please report your problem and mention this
> error message and the query that caused it."
>
> The first error that seems to have been triggered by the launch of a
> new process is:
>
> 10-13 03:54PM 39.280 /heist/ 500 2722ms 1108cpu_ms 0kb Mozilla/
> 5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.10) Gecko/
> 20100914 Firefox/3.6.10,gzip(gfe)
> See details
>
> 65.46.75.178 - - [13/Oct/2010:15:54:42 -0700] "GET /heist/ HTTP/
> 1.1" 500 0 - "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
> rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10,gzip(gfe)"
> "fpgamesserver.appspot.com" ms=2722 cpu_ms=1108 api_cpu_ms=0
> cpm_usd=0.030836 loading_request=1
>
> D 10-13 03:54PM 41.340
>
> FILTER_LIST: [{'PATH_INFO': '!.*/admin/.*'}]
>
> D 10-13 03:54PM 41.341
>
> Match on {'PATH_INFO': '!.*/admin/.*'}
>
> I 10-13 03:54PM 41.537
>
> Hook registered
>
> I 10-13 03:54PM 41.538
>
> DatastoreProxy initialized
>
> E 10-13 03:54PM 41.912
>
> Exception in request:
> Traceback (most recent call last):
> File "/base/python_runtime/python_lib/versions/third_party/
> django-0.96/django/core/handlers/base.py", line 68, in get_response
> callback, callback_args, callback_kwargs =
> resolver.resolve(request.path)
> File "/base/python_runtime/python_lib/versions/third_party/
> django-0.96/django/core/urlresolvers.py", line 162, in resolve
> sub_match = pattern.resolve(new_path)
> File "/base/python_runtime/python_lib/versions/third_party/
> django-0.96/django/core/urlresolvers.py", line 118, in resolve
> return self.callback, args, kwargs
> File "/base/python_runtime/python_lib/versions/third_party/
> django-0.96/django/core/urlresolvers.py", line 127, in _get_callback
> raise ViewDoesNotExist, "Could not import %s. Error was: %s"
> % (mod_name, str(e))
> ViewDoesNotExist: Could not import freshplanet.framework.views.
> Error was: cannot import name datastore_stub_util
>
> E 10-13 03:54PM 41.993
>
> : Tried
> freshplanet.framework.views.get500. Error was: cannot import name
> datastore_stub_util
> Traceback (most recent call last):
> File "/base/data/home/apps/fpgamesserver/0-0-4-first-caper-
> online.345419715554941267/main.py", line 50, in
> main()
> File "/base/data/home/apps/fpgamesserver/0-0-4-first-caper-
> online.345419715554941267/main.py", line 47, in main
> util.run_wsgi_app(application)
> File "/base/python_runtime/python_lib/versions/1/google/
> appengine/ext/webapp/util.py", line 97, in run_wsgi_app
> run_bare_wsgi_app(add_wsgi_middleware(application))
> File "/base/python_runtime/python_lib/versions/1/google/
> appengine/ext/webapp/util.py", line 115, in run_bare_wsgi_app
> result = application(env, _start_response)
> File "/base/python_runtime/python_lib/versions/third_party/
> django-0.96/django/core/handlers/wsgi.py", line 189, in __call__
> response = self.get_response(request)
> File "/base/python_runtime/python_lib/versions/third_party/
> django-0.96/django/core/handlers/base.py", line 125, in get_response
> callback, param_dict = resolver.resolve500()
> File "/base/python_runtime/python_lib/versions/third_party/
> django-0.96/django/core/urlresolvers.py", line 200, in resolve500
> return self._resolve_special('500')
> File "/base/python_runtime/python_lib/versions/third_party/
> django-0.96/django/core/urlresolvers.py", line 194, in
> _resolve_special
> raise ViewDoesNotExist, "Tried %s. Error was: %s" %
> (callback, str(e))
>
> I 10-13 03:54PM 42.000
>
> This request caused a new process to be started for your
> application, and thus caused your application code to be loaded
