I've been seeing Timeouts on data put operations about 30-80 times a
day (which accounts for as much as 1-3% of all write requests) ever
since my app went into production on August 18. This is happening
every day and it's very annoying. It's worse during some periods,
(e.g. when the GAE team reports outages), but it's been a constant
factor regardless of these issues.
It would be awesome if the App Engine team could investigate and put
together a set of best practices for coding and data modeling to
reduce the chance of timeouts, if this is at all possible.
Certainly it's a good idea to code defensively and expect these sorts
of exceptions, but there has to be a way to lower the timeout rate.
Everyone's logs are filling up with garbage because of this and making
it hard to find any other important failures in the logs.
Here are some example stack traces I've been getting:
1) On Model.get_or_insert()
a).
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 862, in get_or_insert
return run_in_transaction(txn)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 1407, in RunInTransaction
result = function(*args, **kwargs)
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 857, in txn
entity = cls.get_by_key_name(key_name,
parent=kwds.get('parent'))
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 779, in get_by_key_name
return get(*keys)
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 974, in get
entities = datastore.Get(keys)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 211, in Get
_MaybeSetupTransaction(req, keys)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 1502, in _MaybeSetupTransaction
tx.handle)
File "/base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py", line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 245, in MakeSyncCall
rpc.Wait()
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 161, in Wait
rpc_completed =
_apphosting_runtime___python__apiproxy.Wait(self)
DeadlineExceededError
b).
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 862, in get_or_insert
return run_in_transaction(txn)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 1431, in RunInTransaction
tx.handle, resp)
File "/base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py", line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 245, in MakeSyncCall
rpc.Wait()
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 216, in __MakeCallDone
exception_entry[1] % (self.package, self.call))
DeadlineExceededError
c).
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 862, in get_or_insert
return run_in_transaction(txn)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 1407, in RunInTransaction
result = function(*args, **kwargs)
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 860, in txn
entity.put()
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 618, in put
return datastore.Put(self._entity)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 162, in Put
raise _ToDatastoreError(err)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 1627, in _ToDatastoreError
raise errors[err.application_error](err.error_detail)
Timeout
d). (this is a very frequent trace)
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 862, in get_or_insert
return run_in_transaction(txn)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 1441, in RunInTransaction
raise _ToDatastoreError(err)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 1627, in _ToDatastoreError
raise errors[err.application_error](err.error_detail)
Timeout
2). On Model.save()
a).
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 618, in put
return datastore.Put(self._entity)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 160, in Put
apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Put', req, resp)
File "/base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py", line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 245, in MakeSyncCall
rpc.Wait()
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
DeadlineExceededError
b). (this is a very frequent trace)
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 618, in put
return datastore.Put(self._entity)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 162, in Put
raise _ToDatastoreError(err)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 1627, in _ToDatastoreError
raise errors[err.application_error](err.error_detail)
Alex
On Sep 30, 12:57 pm, johnP <[EMAIL PROTECTED]> wrote:
> For a while today, I started seeing errors in custom form validation.
> Requesting self.instance in ModelForm form resulted in ValueErrors.
> Now, it seems to have recovered a bit...
>
> On Sep 30, 9:43 am, Adam Loving <[EMAIL PROTECTED]> wrote:
>
> > I am seeing the datastoretimeouterror intermittently several times a
> > day (application = toyvirtualgifts) on a fairly simple put operation.
> > Could this be caused by the rest of the request taking too long (like
> > if thetimeoutfor the entire request fires during the put)? It
> > doesn't seem like that would be the case here, but that's the only
> > explanation I can think of.
>
> > File "/base/python_lib/versions/1/google/appengine/ext/db/
> > __init__.py", line 618, in put
> > return datastore.Put(self._entity)
> > File "/base/python_lib/versions/1/google/appengine/api/
> > datastore.py", line 162, in Put
> > raise _ToDatastoreError(err)
> > File "/base/python_lib/versions/1/google/appengine/api/
> > datastore.py", line 1627, in _ToDatastoreError
> > raise errors[err.application_error](err.error_detail)
> >Timeout
>
> > Thanks,
> > Adam
>
> > On Sep 28, 3:19 am, Ronald <[EMAIL PROTECTED]> wrote:
>
> > > Hi,
> > > I don't know how much work has been done on this front, but my app is
> > > still cripple by this issue.
> > > Maybe the amount oftimeouterrors in the log has decrease a little,
> > > but so has the amount of happy users of my site =(
>
> > > /ronald
>
> > > On Sep 19, 9:45 pm, "Marzia Niccolai" <[EMAIL PROTECTED]> wrote:
>
> > > > Hi,
>
> > > > Our engineers are looking in to your reports.
>
> > > > If all of the people experiencing quota issues like this could email me
> > > > directly with your application id, as well as the approximate times
> > > > these
> > > > errors were occurring, that would be helpful!
>
> > > > Thanks,
> > > > Marzia
>
> > > > On Fri, Sep 19, 2008 at 11:17 AM, Michael Bailey <[EMAIL
> > > > PROTECTED]>wrote:
>
> > > > > I got the error below on a app that has almost zero traffic:
> > > > > see screenshot:
> > > > >http://imagebin.ca/img/3rlwzkBy.png
>
> > > > > Traceback (most recent call last):
> > > > > File "/base/python_lib/versions/1/google/appengine/ext/webapp/
> > > > > __init__.py", line 496, in __call__
> > > > > handler.get(*groups)
> > > > > File "/base/data/home/apps/*******/1.4/search.py", line 25, in get
> > > > > for kw in kws:
> > > > > File "/base/python_lib/versions/1/google/appengine/ext/db/
> > > > > __init__.py", line 1257, in __iter__
> > > > > return self.run()
> > > > > File "/base/python_lib/versions/1/google/appengine/ext/db/
> > > > > __init__.py", line 1589, in run
> > > > > query_run = self._proto_query.Run(*self._args, **self._kwds)
> > > > > File "/base/python_lib/versions/1/google/appengine/ext/gql/
> > > > > __init__.py", line 572, in Run
> > > > > it = bind_results.Run()
> > > > > File "/base/python_lib/versions/1/google/appengine/ext/gql/
> > > > > __init__.py", line 1211, in Run
> > > > > results.append(bound_query.Run())
> > > > > File "/base/python_lib/versions/1/google/appengine/api/
> > > > > datastore.py", line 860, in Run
> > > > > return self._Run()
> > > > > File "/base/python_lib/versions/1/google/appengine/api/
> > > > > datastore.py", line 879, in _Run
> > > > > apiproxy_stub_map.MakeSyncCall('datastore_v3', 'RunQuery', pb,
> > > > > result)
> > > > > File "/base/python_lib/versions/1/google/appengine/api/
> > > > > apiproxy_stub_map.py", line 46, in MakeSyncCall
> > > > > stub.MakeSyncCall(service, call, request, response)
> > > > > File "/base/python_lib/versions/1/google/appengine/runtime/
> > > > > apiproxy.py", line 246, in MakeSyncCall
> > > > > rpc.CheckSuccess()
> > > > > File "/base/python_lib/versions/1/google/appengine/runtime/
> > > > > apiproxy.py", line 189, in CheckSuccess
> > > > > raise self.exception
> > > > > OverQuotaError: The API call datastore_v3.RunQuery() required more
> > > > > quota than is available.
>
> > > > > On Sep 10, 1:16 pm, Ronald <[EMAIL PROTECTED]> wrote:
> > > > > > Hi,
> > > > > > I'm seeing a number ofTimeouterrors on my log file:
>
> > > > > > Traceback (most recent call last):
> > > > > > File "/base/python_lib/versions/1/google/appengine/ext/webapp/
> > > > > > __init__.py", line 496, in __call__
> > > > > > handler.get(*groups)
> > > > > > File "/base/data/home/apps/40tazo/2.29/index.py", line 46, in get
> > > > > > moves = g.move(card,pickups)
> > > > > > File "/base/data/home/apps/40tazo/2.29/game.py", line 231, in move
> > > > > > self.save()
> > > > > > File "/base/data/home/apps/40tazo/2.29/game.py", line 290, in save
> > > > > > self.g.put()
> > > > > > File "/base/python_lib/versions/1/google/appengine/ext/db/
> > > > > > __init__.py", line 618, in put
> > > > > > return datastore.Put(self._entity)
> > > > > > File "/base/python_lib/versions/1/google/appengine/api/
> > > > > > datastore.py", line 162, in Put
> > > > > > raise _ToDatastoreError(err)
> > > > > > File "/base/python_lib/versions/1/google/appengine/api/
> > > > > > datastore.py", line 1627, in _ToDatastoreError
> > > > > > raise errors[err.application_error](err.error_detail)
> > > > > >Timeout: datastoretimeout: operation took too long.
>
> > > > > > In this thread:
> > > > >http://groups.google.com/group/google-appengine/browse_thread/thread/...,
> > > > > > someone describes a similar problem and Jeff Scudder reply that we
> > > > > > should spect some increase latency until Tuesday 9/2.
>
> > > > > > Is this increased latency still an issue?
> > > > > > If that's not the case, any suggestions about how to troubleshoot
> > > > > > this
> > > > > > issue?
>
> > > > > > Greetings
> > > > > > Ronald
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
