We were migrating one of our applications from US to EU using Google Cloud Storage as backup destination and noticed that LOTS of errors are reported during the backup process (the backup went fine though).
Most of the errors reported seem to be transaction collisions, which sounds strange to us as basically the application had no DataStore activity (with disabled writes) during the time we took the backup. Below are some dashboard values of the ah-builtin-python-bundle, including one sample error: Is this behaviour normal? - Marcel * Current Load [image: help]<https://developers.google.com/appengine/kb/general#currentload> * URI Req/Min current Requests last 23 hrs Runtime MCycles last hr Avg Latency last hr /_ah/mapreduce/worker_callback<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fworker_callback%22&severity_level_override=1&view=Search> 26.0 15.30K 19489 20630 ms /_ah/mapreduce/controller_callback<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fcontroller_callback%22&severity_level_override=1&view=Search> 8.0 1.46K 169 1703 ms /_ah/mapreduce/kickoffjob_callback<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fkickoffjob_callback%22&severity_level_override=1&view=Search> 0.0 144 /_ah/mapreduce/finalizejob_callback<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Ffinalizejob_callback%22&severity_level_override=1&view=Search> 0.0 43 99 967 ms /_ah/datastore_admin/<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2F%22&severity_level_override=1&view=Search> 0.2 41 234 1269 ms /_ah/datastore_admin/mapreduce_done<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2Fmapreduce_done%22&severity_level_override=1&view=Search> 0.0 39 134 252 ms /_ah/datastore_admin/static/css/compiled.css<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2Fstatic%2Fcss%2Fcompiled.css%22&severity_level_override=1&view=Search> 0.0 15 46 394 ms /_ah/mapreduce/detail<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fdetail%22&severity_level_override=1&view=Search> 0.0 14 /_ah/datastore_admin/static/js/compiled.js<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2Fstatic%2Fjs%2Fcompiled.js%22&severity_level_override=1&view=Search> 0.0 14 32 1659 ms /_ah/datastore_admin/static/img/help.gif<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2Fstatic%2Fimg%2Fhelp.gif%22&severity_level_override=1&view=Search> 0.0 14 31 576 ms /_ah/mapreduce/command/get_job_detail<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fcommand%2Fget_job_detail%22&severity_level_override=1&view=Search> 0.0 13 136 298 ms /_ah/mapreduce/status.js<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fstatus.js%22&severity_level_override=1&view=Search> 0.0 5 /_ah/mapreduce/jquery.js<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fjquery.js%22&severity_level_override=1&view=Search> 0.0 5 16 446 ms /_ah/mapreduce/jquery-json.js<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fjquery-json.js%22&severity_level_override=1&view=Search> 0.0 5 152 2089 ms /_ah/mapreduce/base.css<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fbase.css%22&severity_level_override=1&view=Search> 0.0 5 24 187 ms /_ah/datastore_admin/backup.do<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2Fbackup.do%22&severity_level_override=1&view=Search> 0.0 4 /_ah/mapreduce/status<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fstatus%22&severity_level_override=1&view=Search> 0.0 2 44 3582 ms /_ah/mapreduce/command/list_configs<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fcommand%2Flist_configs%22&severity_level_override=1&view=Search> 0.0 2 0 69 ms /_ah/datastore_admin/queue/deferred<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2Fqueue%2Fdeferred%22&severity_level_override=1&view=Search> 0.0 2 /_ah/datastore_admin<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%22&severity_level_override=1&view=Search> 0.0 2 * Errors [image: help]<https://developers.google.com/appengine/kb/general#erroruris> * URI Count % Errors last 23 hrs /_ah/mapreduce/worker_callback<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fworker_callback%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 10292 67% /_ah/mapreduce/controller_callback<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fcontroller_callback%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 450 31% /_ah/mapreduce/kickoffjob_callback<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fkickoffjob_callback%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 106 74% /_ah/mapreduce/finalizejob_callback<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Ffinalizejob_callback%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 14 33% /_ah/datastore_admin/mapreduce_done<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2Fmapreduce_done%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 10 26% /img/icn/warning.png<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2Fimg%2Ficn%2Fwarning.png%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 2 /_ah/mapreduce/base.css<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fbase.css%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 1 20% /_ah/mapreduce/detail<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fmapreduce%2Fdetail%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 1 7.1% /_ah/datastore_admin/static/img/help.gif<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2Fstatic%2Fimg%2Fhelp.gif%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 1 7.1% /_ah/datastore_admin/static/css/compiled.css<https://appengine.google.com/logs?version_id=ah-builtin-python-bundle.352804153354501097&app_id=s%7Etruesplit&filter_type=labels&filter=path%3A%22%2F_ah%2Fdatastore_admin%2Fstatic%2Fcss%2Fcompiled.css%22+status%3A%5B45%5D%5Cd%5Cd&severity_level_override=1&view=Search> 1 6.7% © 2008 Google | Terms of Service<https://developers.google.com/appengine/terms>| Privacy Policy <http://www.google.com/policies/privacy/> | Blog<http://googleappengine.blogspot.com>| Discussion Forums <http://groups.google.com/group/google-appengine> | Project<https://developers.google.com/appengine/>| Docs <https://developers.google.com/appengine/docs/> 1. 1. 2012-11-23 10:02:11.495 /_ah/mapreduce/worker_callback 500 38690ms 1kb AppEngine-Google; (+http://code.google.com/appengine) 0.1.0.2 - - [23/Nov/2012:02:02:11 -0800] "POST /_ah/mapreduce/worker_callback HTTP/1.1" 500 1399 "http://ah-builtin-python-bundle-dot-latest-dot-NAMEHERE.appspot.com/_ah/mapreduce/worker_callback"; "AppEngine-Google; (+http://code.google.com/appengine)" "ah-builtin-python-bundle-dot-latest-dot-NAMEHERE.appspot.com" ms=38690 cpu_ms=24181 cpm_usd=0.000156 queue_name=default task_name=appengine-mrshard-158262053854451F422C9-0-1 instance=00c61b117c40497530af9f0d75b7a691d84a4813 <https://appengine.google.com/instances?app_id=s%7Etruesplit&version_id=ah-builtin-python-bundle.352804153354501097&key=00c61b117c40497530af9f0d75b7a691d84a4813#00c61b117c40497530af9f0d75b7a691d84a4813> 2. D 2012-11-23 10:01:34.969 Could not import map from google.appengine.ext.datastore_admin.backup_handler.BackupEntity. Will try recursively. Traceback (most recent call last): File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/util.py", line 98, in for_name result = __import__(module_name, None, None, [short_name]) ImportError: No module named BackupEntity 3. W 2012-11-23 10:01:55.500 Transaction collision. Retrying... 4. W 2012-11-23 10:02:01.893 Transaction collision. Retrying... 5. W 2012-11-23 10:02:05.847 Transaction collision. Retrying... 6. W 2012-11-23 10:02:11.091 Transaction collision. Retrying... 7. E 2012-11-23 10:02:11.391 The transaction could not be committed. Please try again. Traceback (most recent call last): File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 712, in __call__ handler.post(*groups) File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/base_handler.py", line 81, in post self.handle() File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/handlers.py", line 223, in handle ctx.flush() File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/mapreduce/context.py", line 348, in flush pool.flush() File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/datastore_admin/backup_handler.py", line 1461, in flush db.run_in_transaction(tx) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2433, in RunInTransaction return RunInTransactionOptions(None, function, *args, **kwargs) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 2579, in RunInTransactionOptions 'The transaction could not be committed. Please try again.') TransactionFailedError: The transaction could not be committed. Please try again. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-07aL5kOdh8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
