[google-appengine] Re: Datastore Timeouts Must Die
As an example, of how frequently datastore write timeouts are happening on app engine, here are the last 20 entries in my production error log, showing 20 timeouts in 15 minutes! I'm hoping the Google team will break their silence about this and at least acknowledge the problem. Developers: please star issue http://code.google.com/p/googleappengine/issues/detail?id=764 if you are also experiencing this problem. 10-07 02:20PM 48.172 /games 500 4040ms 11285mcycles 0kb 10-07 02:20PM 48.172 /games 500 4040ms 11285mcycles 0kb 10-07 02:16PM 43.763 /games 500 2813ms 8253mcycles 0kb 10-07 02:16PM 43.410 /games 500 3078ms 8909mcycles 0kb 10-07 02:16PM 40.958 /games 500 3084ms 8915mcycles 0kb 10-07 02:15PM 03.499 /games 500 4109ms 12077mcycles 0kb 10-07 02:15PM 02.584 /games 500 4110ms 12365mcycles 0kb 10-07 02:14PM 21.732 /games 500 3071ms 8900mcycles 0kb 10-07 02:10PM 52.865 /games 500 4045ms 11287mcycles 0kb 10-07 02:10PM 51.875 /games 500 4041ms 11285mcycles 0kb 10-07 02:10PM 41.774 /games 500 4036ms 11282mcycles 0kb 10-07 02:10PM 36.958 /games 500 4033ms 11279mcycles 0kb 10-07 02:10PM 23.148 /games 500 4039ms 11174mcycles 0kb 10-07 02:08PM 31.288 /games 500 4068ms 10555mcycles 0kb 10-07 02:08PM 30.842 /games 500 4449ms 12217mcycles 0kb 10-07 02:07PM 27.179 /games 500 4047ms 11285mcycles 0kb 10-07 02:07PM 24.828 /games 500 4047ms 11284mcycles 0kb 10-07 02:07PM 20.139 /games 500 4061ms 11281mcycles 0kb 10-07 02:07PM 18.532 /games 500 4042ms 11246mcycles 0kb 10-07 02:07PM 08.776 /games 500 4048ms 11285mcycles 0kb 10-07 02:05PM 58.206 /games 500 4047ms 11019mcycles 0kb On Oct 6, 1:53 am, Alex Epshteyn <[EMAIL PROTECTED]> wrote: > Please see:http://code.google.com/p/googleappengine/issues/detail?id=764 > > There is a recent thread on this group titled "Why Google App Engine > is broken and what Google must do to fix it." I personally don't > think that any of the topics raised by the OP of that thread imply > that anything is broken - they are just feature requests. > > What *is* broken, however, is writing to the Datastore. Lots of > developers, not just me, have been complaining about this here for > months. I don't think there's been any real response from the Google > team about this. This issue needs to be investigated and hopefully > fixed before App Engine can be used for production. The probability > of failure for any given data write request is just too high right > now. I actually *am* using App Engine in production and somewhat > regretting it, mostly due to this issue. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore Timeouts Must Die
Hi Alex, sorry for the frustrations this is causing. The first thing I would check for here is to verify that these writes are not failing due to entity group contention. (documentation on entity groups: http://code.google.com/appengine/docs/datastore/keysandentitygroups.html#Entity_Groups_Ancestors_and_Paths). All writes within one entity group are optimistically serialized. You mention that a good write latency is 100ms. So, if you ever try to do more than 10 writes in one second you will experience transaction retries and eventually you will hit the datastore timeout. Note that even put() and get_or_insert() are still considered transactions in our system. On Oct 7, 2:48 pm, Alex Epshteyn <[EMAIL PROTECTED]> wrote: > As an example, of how frequently datastore write timeouts are > happening on app engine, here are the last 20 entries in my production > error log, showing 20 timeouts in 15 minutes! > > I'm hoping the Google team will break their silence about this and at > least acknowledge the problem. > > Developers: please star issue > http://code.google.com/p/googleappengine/issues/detail?id=764 > if you are also experiencing this problem. > > 10-07 02:20PM 48.172 /games 500 4040ms 11285mcycles 0kb > 10-07 02:20PM 48.172 /games 500 4040ms 11285mcycles 0kb > 10-07 02:16PM 43.763 /games 500 2813ms 8253mcycles 0kb > 10-07 02:16PM 43.410 /games 500 3078ms 8909mcycles 0kb > 10-07 02:16PM 40.958 /games 500 3084ms 8915mcycles 0kb > 10-07 02:15PM 03.499 /games 500 4109ms 12077mcycles 0kb > 10-07 02:15PM 02.584 /games 500 4110ms 12365mcycles 0kb > 10-07 02:14PM 21.732 /games 500 3071ms 8900mcycles 0kb > 10-07 02:10PM 52.865 /games 500 4045ms 11287mcycles 0kb > 10-07 02:10PM 51.875 /games 500 4041ms 11285mcycles 0kb > 10-07 02:10PM 41.774 /games 500 4036ms 11282mcycles 0kb > 10-07 02:10PM 36.958 /games 500 4033ms 11279mcycles 0kb > 10-07 02:10PM 23.148 /games 500 4039ms 11174mcycles 0kb > 10-07 02:08PM 31.288 /games 500 4068ms 10555mcycles 0kb > 10-07 02:08PM 30.842 /games 500 4449ms 12217mcycles 0kb > 10-07 02:07PM 27.179 /games 500 4047ms 11285mcycles 0kb > 10-07 02:07PM 24.828 /games 500 4047ms 11284mcycles 0kb > 10-07 02:07PM 20.139 /games 500 4061ms 11281mcycles 0kb > 10-07 02:07PM 18.532 /games 500 4042ms 11246mcycles 0kb > 10-07 02:07PM 08.776 /games 500 4048ms 11285mcycles 0kb > 10-07 02:05PM 58.206 /games 500 4047ms 11019mcycles 0kb > > On Oct 6, 1:53 am, Alex Epshteyn <[EMAIL PROTECTED]> wrote: > > > Please see:http://code.google.com/p/googleappengine/issues/detail?id=764 > > > There is a recent thread on this group titled "Why Google App Engine > > is broken and what Google must do to fix it." I personally don't > > think that any of the topics raised by the OP of that thread imply > > that anything is broken - they are just feature requests. > > > What *is* broken, however, is writing to the Datastore. Lots of > > developers, not just me, have been complaining about this here for > > months. I don't think there's been any real response from the Google > > team about this. This issue needs to be investigated and hopefully > > fixed before App Engine can be used for production. The probability > > of failure for any given data write request is just too high right > > now. I actually *am* using App Engine in production and somewhat > > regretting it, mostly due to this issue. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore Timeouts Must Die
Thanks for your reply, Jon, but I don't have any entity groups with more than one entity right now. I had the same thought a while back, and haven't been using any entity groups or ancestors since 2 months ago, but nothing changed. These timeouts are happening when putting single entities. Alex On Wed, Oct 8, 2008 at 12:38 AM, Jon McAlister <[EMAIL PROTECTED]> wrote: > > Hi Alex, sorry for the frustrations this is causing. The first thing I > would check for here is to verify that these writes are not failing > due to entity group contention. (documentation on entity groups: > http://code.google.com/appengine/docs/datastore/keysandentitygroups.html#Entity_Groups_Ancestors_and_Paths). > All writes within one entity group are optimistically serialized. You > mention that a good write latency is 100ms. So, if you ever try to do > more than 10 writes in one second you will experience transaction > retries and eventually you will hit the datastore timeout. Note that > even put() and get_or_insert() are still considered transactions in > our system. > > On Oct 7, 2:48 pm, Alex Epshteyn <[EMAIL PROTECTED]> wrote: >> As an example, of how frequently datastore write timeouts are >> happening on app engine, here are the last 20 entries in my production >> error log, showing 20 timeouts in 15 minutes! >> >> I'm hoping the Google team will break their silence about this and at >> least acknowledge the problem. >> >> Developers: please star issue >> http://code.google.com/p/googleappengine/issues/detail?id=764 >> if you are also experiencing this problem. >> >> 10-07 02:20PM 48.172 /games 500 4040ms 11285mcycles 0kb >> 10-07 02:20PM 48.172 /games 500 4040ms 11285mcycles 0kb >> 10-07 02:16PM 43.763 /games 500 2813ms 8253mcycles 0kb >> 10-07 02:16PM 43.410 /games 500 3078ms 8909mcycles 0kb >> 10-07 02:16PM 40.958 /games 500 3084ms 8915mcycles 0kb >> 10-07 02:15PM 03.499 /games 500 4109ms 12077mcycles 0kb >> 10-07 02:15PM 02.584 /games 500 4110ms 12365mcycles 0kb >> 10-07 02:14PM 21.732 /games 500 3071ms 8900mcycles 0kb >> 10-07 02:10PM 52.865 /games 500 4045ms 11287mcycles 0kb >> 10-07 02:10PM 51.875 /games 500 4041ms 11285mcycles 0kb >> 10-07 02:10PM 41.774 /games 500 4036ms 11282mcycles 0kb >> 10-07 02:10PM 36.958 /games 500 4033ms 11279mcycles 0kb >> 10-07 02:10PM 23.148 /games 500 4039ms 11174mcycles 0kb >> 10-07 02:08PM 31.288 /games 500 4068ms 10555mcycles 0kb >> 10-07 02:08PM 30.842 /games 500 4449ms 12217mcycles 0kb >> 10-07 02:07PM 27.179 /games 500 4047ms 11285mcycles 0kb >> 10-07 02:07PM 24.828 /games 500 4047ms 11284mcycles 0kb >> 10-07 02:07PM 20.139 /games 500 4061ms 11281mcycles 0kb >> 10-07 02:07PM 18.532 /games 500 4042ms 11246mcycles 0kb >> 10-07 02:07PM 08.776 /games 500 4048ms 11285mcycles 0kb >> 10-07 02:05PM 58.206 /games 500 4047ms 11019mcycles 0kb >> >> On Oct 6, 1:53 am, Alex Epshteyn <[EMAIL PROTECTED]> wrote: >> >> > Please see:http://code.google.com/p/googleappengine/issues/detail?id=764 >> >> > There is a recent thread on this group titled "Why Google App Engine >> > is broken and what Google must do to fix it." I personally don't >> > think that any of the topics raised by the OP of that thread imply >> > that anything is broken - they are just feature requests. >> >> > What *is* broken, however, is writing to the Datastore. Lots of >> > developers, not just me, have been complaining about this here for >> > months. I don't think there's been any real response from the Google >> > team about this. This issue needs to be investigated and hopefully >> > fixed before App Engine can be used for production. The probability >> > of failure for any given data write request is just too high right >> > now. I actually *am* using App Engine in production and somewhat >> > regretting it, mostly due to this issue. > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore Timeouts Must Die
It does seem like the number of timeouts has increased in recent weeks. I just checked my logs, and during the minute of 11:33am this morning I had 43 timeout data store errors: Traceback (most recent call last): File "/base/python_lib/versions/1/google/appengine/ext/webapp/ __init__.py", line 501, in __call__ handler.post(*groups) File "/base/data/home/apps/buddypokeapp/1.416/api.py", line 1296, in post self.poke_main() File "/base/data/home/apps/buddypokeapp/1.416/api.py", line 1196, in poke_main fromAvatar.put() File "/base/python_lib/versions/1/google/appengine/ext/db/ __init__.py", line 618, in put return datastore.Put(self._entity) File "/base/python_lib/versions/1/google/appengine/api/ datastore.py", line 162, in Put raise _ToDatastoreError(err) File "/base/python_lib/versions/1/google/appengine/api/ datastore.py", line 1627, in _ToDatastoreError raise errors[err.application_error](err.error_detail) Timeout: datastore timeout: operation took too long. Just like Alex, this is for a very simple data model with a single db.text and a list of db.text's. Thanks Dave On Oct 8, 11:17 am, "Alex Epshteyn" <[EMAIL PROTECTED]> wrote: > Thanks for your reply, Jon, but I don't have any entity groups with > more than one entity right now. I had the same thought a while back, > and haven't been using any entity groups or ancestors since 2 months > ago, but nothing changed. These timeouts are happening when putting > single entities. > > Alex > > On Wed, Oct 8, 2008 at 12:38 AM, Jon McAlister <[EMAIL PROTECTED]> wrote: > > > Hi Alex, sorry for the frustrations this is causing. The first thing I > > would check for here is to verify that these writes are not failing > > due to entity group contention. (documentation on entity groups: > >http://code.google.com/appengine/docs/datastore/keysandentitygroups.h...). > > All writes within one entity group are optimistically serialized. You > > mention that a good write latency is 100ms. So, if you ever try to do > > more than 10 writes in one second you will experience transaction > > retries and eventually you will hit the datastore timeout. Note that > > even put() and get_or_insert() are still considered transactions in > > our system. > > > On Oct 7, 2:48 pm, Alex Epshteyn <[EMAIL PROTECTED]> wrote: > >> As an example, of how frequently datastore write timeouts are > >> happening on app engine, here are the last 20 entries in my production > >> error log, showing 20 timeouts in 15 minutes! > > >> I'm hoping the Google team will break their silence about this and at > >> least acknowledge the problem. > > >> Developers: please star issue > >> http://code.google.com/p/googleappengine/issues/detail?id=764 > >> if you are also experiencing this problem. > > >> 10-07 02:20PM 48.172 /games 500 4040ms 11285mcycles 0kb > >> 10-07 02:20PM 48.172 /games 500 4040ms 11285mcycles 0kb > >> 10-07 02:16PM 43.763 /games 500 2813ms 8253mcycles 0kb > >> 10-07 02:16PM 43.410 /games 500 3078ms 8909mcycles 0kb > >> 10-07 02:16PM 40.958 /games 500 3084ms 8915mcycles 0kb > >> 10-07 02:15PM 03.499 /games 500 4109ms 12077mcycles 0kb > >> 10-07 02:15PM 02.584 /games 500 4110ms 12365mcycles 0kb > >> 10-07 02:14PM 21.732 /games 500 3071ms 8900mcycles 0kb > >> 10-07 02:10PM 52.865 /games 500 4045ms 11287mcycles 0kb > >> 10-07 02:10PM 51.875 /games 500 4041ms 11285mcycles 0kb > >> 10-07 02:10PM 41.774 /games 500 4036ms 11282mcycles 0kb > >> 10-07 02:10PM 36.958 /games 500 4033ms 11279mcycles 0kb > >> 10-07 02:10PM 23.148 /games 500 4039ms 11174mcycles 0kb > >> 10-07 02:08PM 31.288 /games 500 4068ms 10555mcycles 0kb > >> 10-07 02:08PM 30.842 /games 500 4449ms 12217mcycles 0kb > >> 10-07 02:07PM 27.179 /games 500 4047ms 11285mcycles 0kb > >> 10-07 02:07PM 24.828 /games 500 4047ms 11284mcycles 0kb > >> 10-07 02:07PM 20.139 /games 500 4061ms 11281mcycles 0kb > >> 10-07 02:07PM 18.532 /games 500 4042ms 11246mcycles 0kb > >> 10-07 02:07PM 08.776 /games 500 4048ms 11285mcycles 0kb > >> 10-07 02:05PM 58.206 /games 500 4047ms 11019mcycles 0kb > > >> On Oct 6, 1:53 am, Alex Epshteyn <[EMAIL PROTECTED]> wrote: > > >> > Please see:http://code.google.com/p/googleappengine/issues/detail?id=764 > > >> > There is a recent thread on this group titled "Why Google App Engine > >> > is broken and what Google must do to fix it." I personally don't > >> > think that any of the topics raised by the OP of that thread imply > >> > that anything is broken - they are just feature requests. > > >> > What *is* broken, however, is writing to the Datastore. Lots of > >> > developers, not just me, have been complaining about this here for > >> > months. I don't think there's been any real response from the Google > >> > team about this. This issue needs to be investigated and hopefully > >> > fixed before App Engine can be used for production. The probability > >> > of failure for any given data write request is just too high right > >> > now. I actually *am* using App Engine in
