And... I have a solution.
It appears that GAE resets sys.path before every invocation of
main(). We were modifying the sys.path (adding a lib directory to it)
at the module-level, so this would only take effect for the first
request on a given Python instance.
On Sep 15, 9:55 am, Tony Arkles <[EMAIL PROTECTED]> wrote:
> I hate to do this. I'm bumping this, in the hopes that someone might
> be experiencing similar problems.
>
> It's super hard to diagnose this, because it works great on the local
> dev server, and works great on app-engine *most* of the time, and then
> it'll get random failures for a few minutes, and then work again.
>
> On Sep 11, 9:46 am, Tony Arkles <[EMAIL PROTECTED]> wrote:
>
> > Hi!
>
> > We're in the process of learning GAE, and over the last two days have
> > hit a wall. Before going too far into the description, here's the
> > versions of things we're using:
>
> > * Django 1.0 beta 2
> > * app-engine-django-helper r58
>
> > It started yesterday; we were getting strange import errors, and only
> > once in a while:
>
> > ----- snip -----
>
> > <class 'django.core.exceptions.ImproperlyConfigured'>: Error importing
> > middleware middleware.redirects: "No module named
> > middleware.redirects"
> > Traceback (most recent call last):
> > File "/base/data/home/apps/vendastaweb/0.39/main.py", line 88, in
> > <module>
> > main()
> > File "/base/data/home/apps/vendastaweb/0.39/main.py", line 58, in
> > real_main
> > util.run_wsgi_app(application)
> > File "/base/python_lib/versions/1/google/appengine/ext/webapp/
> > util.py", line 76, in run_wsgi_app
> > result = application(env, _start_response)
> > File "/base/data/home/apps/vendastaweb/0.39/django/core/handlers/
> > wsgi.py", line 211, in __call__
> > self.load_middleware()
> > File "/base/data/home/apps/vendastaweb/0.39/django/core/handlers/
> > base.py", line 40, in load_middleware
> > raise exceptions.ImproperlyConfigured, 'Error importing middleware
> > %s: "%s"' % (mw_module, e)
>
> > ----- snip -----
>
> > Hitting refresh a few times would make the error go away. These
> > errors weren't consistent; it would just be some random module from
> > our project that would fail to import.
>
> > We also started noticing that we were getting messages like:
>
> > This request used a high amount of CPU, and was roughly 8.2 times over
> > the average request CPU limit. High CPU requests have a small quota,
> > and if you exceed this quota, your app will be temporarily disabled.
>
> > So, of course, the answer is to profile. Profiling is where it got
> > really wierd. Our execution of "real_main()" was a small fraction
> > (about 1/3) of the request time measured by GAE, and most of *that*
> > time was occupied by "__import__":
>
> > ----- snip -----
>
> > : 80216 function calls (76826 primitive calls) in
> > 0.580 CPU seconds
> > :
> > : Ordered by: internal time
> > : List reduced from 1040 to 30 due to restriction <30>
> > :
> > : ncalls tottime percall cumtime percall
> > filename:lineno(function)
> > : 7 0.108 0.015 0.120 0.017
> > {google3.apphosting.runtime._apphosting_runtime___python__apiproxy.Wait}
> > : 48/42 0.060 0.001 0.351 0.008 {__import__}
> > : 1 0.060 0.060 0.183 0.183 /base/data/
> > home/apps/vendastaweb/0.29/vendastaweb/models/feed.py:5(<module>)
> > : 1 0.022 0.022 0.071 0.071 /base/data/
> > home/apps/vendastaweb/0.29/vendastaweb/views/faces.py:5(<module>)
> > : 456/98 0.022 0.000 0.072 0.001 /base/
> > python_dist/lib/python2.5/sre_parse.py:385(_parse)
>
> > ----- snip -----
>
> > Our models (feed, for example) are incredibly simple, and it makes no
> > sense that they'd take any significant amount of CPU time to create
> > the class.
>
> > The profiling log entries (from the admin panel) look like this
> > (trimmed up a bit):
>
> > ----- snip -----
>
> > 09-10 04:54PM 41.119 /projects/ 200 1482ms 8kb
>
> > 71.17.167.74 - - [10/09/2008:16:54:42 -0700] "GET /projects/ HTTP/1.1"
> > 200 8091 - -
>
> > Profile data:
> > Function time:
>
> > 118854 function calls (114692 primitive calls) in 0.638 CPU
> > seconds
>
> > Ordered by: internal time
> > List reduced from 1178 to 30 due to restriction <30>
>
> > ----- snip -----
>
> > To further confuse things, I uploaded the application in parallel, and
> > the parallel copy seems to run flawlessly (no timeouts, no quota
> > violations, etc). I've hit both of them with httperf (from 1-5 conn/
> > sec), and the parallel copy succeeds 100% of the time, while the
> > original copy fails ~30-40% of the time.
>
> > Please let me know if I've missed anything relevant.
>
> > Cheers
> > Tony
>
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
