[google-appengine] Re: Transaction Failed Exception from a non-transactional .put()

2015-09-23 Thread Kartik Domadiya 
We are facing the same issue thought we have non-transnational. Any ideas ? 

Here is the stacktrace:

  File 
"/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init__.py",
 line 1077, in put
return datastore.Put(self._entity, **kwargs)
  File 
"/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/datastore.py",
 line 605, in Put
return PutAsync(entities, **kwargs).get_result()
  File 
"/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
 line 613, in get_result
return self.__get_result_hook(self)
  File 
"/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
 line 1881, in __put_hook
self.check_rpc_success(rpc)
  File 
"/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
 line 1373, in check_rpc_success
raise _ToDatastoreError(err)
TransactionFailedError: too much contention on these datastore entities. please 
try again.



On Thursday, August 4, 2011 at 7:18:37 PM UTC+5:30, Joshua Smith wrote:
>
> This is a weird one.
>
> I have a cron that rolls up some data every few minutes.  Nothing too 
> complicated:
>
> class CollectStatsHandler(webapp.RequestHandler):
>   def get(self):
> nodes = NodeModel.gql("WHERE online = :1 and procs > :2", True, 
> 0).fetch(1000)
> sum = 0
> for n in nodes:
>   sum = sum + n.procs
> now = datetime.datetime.now().timetuple()
> hour = datetime.datetime(now[0], now[1], now[2], now[3])
> hourly = HourlyStatsModel.gql("WHERE hour = :1", hour).get()
> if not hourly:
>   hourly = HourlyStatsModel()
>   hourly.hour = hour
>   hourly.stats = str(sum)
> else:
>   hourly.stats = hourly.stats + "," + str(sum)
> hourly.put()
>
> I'm running on M/S, so every now and then I'll get timeouts or whatever, 
> but that's OK because this is really just for my internal "app health" 
> dashboard.
>
> Anyway, yesterday I got this exception:
>
> google.appengine.api.datastore_errors.TransactionFailedError@/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py:1074
>  
> (at least 2 occurrences)Handler:main.pyURL:
> http://mesonstreaming.appspot.com/admin/collectStatsStacktrace:
>
> Traceback (most recent call last):
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py",
>  line 700, in __call__
> handler.get(*groups)
>   File "/base/data/home/apps/mesonstreaming/51.352130110004909428/main.py", 
> line 1677, in get
> hourly.put()
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py",
>  line 1006, in put
> return datastore.Put(self._entity, config=config)
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py",
>  line 467, in Put
> return PutAsync(entities, **kwargs).get_result()
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>  line 658, in get_result
> results = self.__rpcs[0].get_result()
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
>  line 592, in get_result
> return self.__get_result_hook(self)
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>  line 1385, in __put_hook
> self.check_rpc_success(rpc)
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>  line 1074, in check_rpc_success
> raise _ToDatastoreError(err)
> TransactionFailedError: too much contention on these datastore entities. 
> please try again.
>
>
> So we have a transaction failed error on a put which is NOT a transaction.
>
> And we have too much contention on an entity which is written once every 5 
> minutes by a cron.
>
> Huh?
>
> -Joshua
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/474d3b75-c056-4af1-a8ae-cc6a1f81cb53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Transaction Failed Exception from a non-transactional .put()

2015-09-23 Thread Ryan (Cloud Platform Support) 
Salutations Kartik,

Contention happens when you are doing to many writes per second. This is a 
great article on avoiding it. 
 If 
you give some more details as to what you are doing I might be able to give 
some more specific advice.

On Thursday, August 4, 2011 at 9:48:37 AM UTC-4, Joshua Smith wrote:
>
> This is a weird one.
>
> I have a cron that rolls up some data every few minutes.  Nothing too 
> complicated:
>
> class CollectStatsHandler(webapp.RequestHandler):
>   def get(self):
> nodes = NodeModel.gql("WHERE online = :1 and procs > :2", True, 
> 0).fetch(1000)
> sum = 0
> for n in nodes:
>   sum = sum + n.procs
> now = datetime.datetime.now().timetuple()
> hour = datetime.datetime(now[0], now[1], now[2], now[3])
> hourly = HourlyStatsModel.gql("WHERE hour = :1", hour).get()
> if not hourly:
>   hourly = HourlyStatsModel()
>   hourly.hour = hour
>   hourly.stats = str(sum)
> else:
>   hourly.stats = hourly.stats + "," + str(sum)
> hourly.put()
>
> I'm running on M/S, so every now and then I'll get timeouts or whatever, 
> but that's OK because this is really just for my internal "app health" 
> dashboard.
>
> Anyway, yesterday I got this exception:
>
> google.appengine.api.datastore_errors.TransactionFailedError@/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py:1074
>  
> (at least 2 occurrences)Handler:main.pyURL:
> http://mesonstreaming.appspot.com/admin/collectStatsStacktrace:
>
> Traceback (most recent call last):
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py",
>  line 700, in __call__
> handler.get(*groups)
>   File "/base/data/home/apps/mesonstreaming/51.352130110004909428/main.py", 
> line 1677, in get
> hourly.put()
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py",
>  line 1006, in put
> return datastore.Put(self._entity, config=config)
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py",
>  line 467, in Put
> return PutAsync(entities, **kwargs).get_result()
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>  line 658, in get_result
> results = self.__rpcs[0].get_result()
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
>  line 592, in get_result
> return self.__get_result_hook(self)
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>  line 1385, in __put_hook
> self.check_rpc_success(rpc)
>   File 
> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>  line 1074, in check_rpc_success
> raise _ToDatastoreError(err)
> TransactionFailedError: too much contention on these datastore entities. 
> please try again.
>
>
> So we have a transaction failed error on a put which is NOT a transaction.
>
> And we have too much contention on an entity which is written once every 5 
> minutes by a cron.
>
> Huh?
>
> -Joshua
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9e5c32f0-8d99-4d6a-b56e-1c6dba9e25c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Transaction Failed Exception from a non-transactional .put()

2015-09-28 Thread Karl MacMillan 
That error is about write limits on entities within the same entity group - it 
actually doesn’t have anything to do with a transaction.


The old datastore documentation has a better explanation of this: 
https://cloud.google.com/appengine/docs/python/datastore/structuring_for_strong_consistency


You can also read 
https://cloud.google.com/appengine/articles/scaling/contention?hl=en.


I don’t quite know what to say about the cron job every 5 minutes other than 
since the write limitation is per entity group it could be that writes are 
happening in other places in you app that are pushing you over the limit.


For myself - I’ve just structured my data so that almost nothing is within an 
entity group and most of the time I’m fetching by key. For those times when I 
need to query the nature of my app makes it fine for the results to be 
eventually consistent.


Karl



> On Sep 23, 2015, at 8:42 AM, Kartik Domadiya  
> wrote:
> 
> 
> We are facing the same issue thought we have non-transnational. Any ideas ? 
> 
> Here is the stacktrace:
>   File 
> "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init__.py",
>  line 1077, in put
> return datastore.Put(self._entity, **kwargs)
>   File 
> "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/datastore.py",
>  line 605, in Put
> return PutAsync(entities, **kwargs).get_result()
>   File 
> "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
>  line 613, in get_result
> return self.__get_result_hook(self)
>   File 
> "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>  line 1881, in __put_hook
> self.check_rpc_success(rpc)
>   File 
> "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>  line 1373, in check_rpc_success
> raise _ToDatastoreError(err)
> TransactionFailedError: too much contention on these datastore entities. 
> please try again.
> 
> On Thursday, August 4, 2011 at 7:18:37 PM UTC+5:30, Joshua Smith wrote:> This 
> is a weird one.
>> 
>> I have a cron that rolls up some data every few minutes.  Nothing too 
>> complicated:
>> 
>> 
>> class CollectStatsHandler(webapp.RequestHandler):
>> 
>>   def get(self):
>>     nodes = NodeModel.gql("WHERE online = :1 and procs > :2", True, 
>> 0).fetch(1000)
>>     sum = 0
>>     for n in nodes:
>>       sum = sum + n.procs
>>     now = datetime.datetime.now().timetuple()
>> 
>>     hour = datetime.datetime(now[0], now[1], now[2], now[3])
>>     hourly = HourlyStatsModel.gql("WHERE hour = :1", hour).get()
>>     if not hourly:
>>       hourly = HourlyStatsModel()
>>       hourly.hour = hour
>>       hourly.stats = str(sum)
>>     else:
>>       hourly.stats = hourly.stats + "," + str(sum)
>>     hourly.put()    
>> 
>> 
>> 
>> I'm running on M/S, so every now and then I'll get timeouts or whatever, but 
>> that's OK because this is really just for my internal "app health" dashboard.
>> 
>> 
>> Anyway, yesterday I got this exception:
>> 
>> 
>> google.appengine.api.datastore_errors.TransactionFailedError@/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py:1074
>>  (at least 2 occurrences)
>> 
>> 
>> 
>> 
>> 
>> 
>> Handler:
>> main.py
>> 
>> URL:
>> http://mesonstreaming.appspot.com/admin/collectStats
>> 
>> 
>> 
>> Stacktrace:
>> Traceback (most recent call last):
>>   File 
>> "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py",
>>  line 700, in __call__
>> handler.get(*groups)
>>   File "/base/data/home/apps/mesonstreaming/51.352130110004909428/main.py", 
>> line 1677, in get
>> hourly.put()
>>   File 
>> "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init__.py",
>>  line 1006, in put
>> return datastore.Put(self._entity, config=config)
>>   File 
>> "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py",
>>  line 467, in Put
>> return PutAsync(entities, **kwargs).get_result()
>>   File 
>> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>>  line 658, in get_result
>> results = self.__rpcs[0].get_result()
>>   File 
>> "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
>>  line 592, in get_result
>> return self.__get_result_hook(self)
>>   File 
>> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>>  line 1385, in __put_hook
>> self.check_rpc_success(rpc)
>>   File 
>> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
>>  line 1074, in check_rpc_success
>> raise _ToDatastoreError(err)
>> TransactionFailedError: too much contention on these datastore entities. 
>> please try again.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>