[google-appengine] backup not complete sometimes. Tasks are stuck.
Hello, I am doing the backup of the datastore using the cron everyday. Somedays, the backup is not complete. The backup will finish fine in the next day. Please see the following stuck tasks. NameETACreation TimeMethod/URLRetriesExecutionsPayloadActions appengine-mrcontrol-15808981980151796BB87-2236 ↓ 2013/06/12 18:28:42 0:37:09 from now2013/06/10 13:24:06 2 days, 4:27:26 agoPOST /_ah/mapreduce/controller_callback671378 bytes 6 HeadersRun Now appengine-mrshard-15808981980151796BB87-11-1-retry-2 ↓ 2013/06/12 18:26:54 0:35:22 from now2013/06/10 16:15:25 2 days, 1:36:07 agoPOST /_ah/mapreduce/worker_callback645019 bytes 6 HeadersRun Now appengine-mrshard-15808981980151796BB87-11-3-retry-0 ↓ 2013/06/12 18:30:53 0:39:20 from now2013/06/10 11:31:00 2 days, 6:20:32 agoPOST /_ah/mapreduce/worker_callback695019 bytes 6 HeadersRun Now Also see the pending backup from the following screen *Pending Backups* Name Start-Time Backup0_2013_06_10 June 10, 2013, 6 p.m. Backup0_2013_06_05 June 5, 2013, 6 p.m.Backup0_2013_05_26 May 26, 2013, 6 p.m. -Aswath -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] backup not complete sometimes. Tasks are stuck.
Some days I've the same problem and I can't see any errors in logs. Is this a real error and the backup doesn't complete or are single tasks errors because tasks can't complete properly but the backup is done? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] backup not complete sometimes. Tasks are stuck.
Can you create an issue at https://code.google.com/p/googleappengine/wiki/FilingIssues for this? Please include your application ID so that we can investigate the cause. On Thu, Jun 13, 2013 at 2:04 AM, Moises Belchin wrote: > Some days I've the same problem and I can't see any errors in logs. > > Is this a real error and the backup doesn't complete or are single tasks > errors because tasks can't complete properly but the backup is done? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] backup not complete sometimes. Tasks are stuck.
I have spent a bunch of time (with support) trying to get to the bottom of this sort of thing. There are 2 major contributors that I've discovered: (1) *Many namespaces.* If you have many namespaces, then Datastore Statistics are no longer computed. You can tell by going to Datastore Statistics. If it says "Last updated: [long time ago]", then you are in this situation. When this happens, the Map-Reduce framework uses only 32 shards, which can greatly extend the time to backup. (2) *General Files API failures.* If the Files API is having difficulty, then the worker shard can fail. If it fails 10 times (I believe), then it simply dies. The M-R jobs never completes and the controllers just keep on retrying. This situation can also occur if you hit the daily limit for Files API transfer (100 GB or so, I think). This is a pain and has caused me to continually manually clean queues, and of course, there are Kinds that we simply can't get backed up. Hopefully this all gets better under the new GCS client j -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] backup not complete sometimes. Tasks are stuck.
The backup job is stuck again. My appid has lot of namespaces. appid: accountingguru-india.appspot.com Please see the issue created. https://code.google.com/p/googleappengine/issues/detail?id=9694&thanks=9694&ts=1374550986 -Aswath On Mon, Jun 17, 2013 at 9:56 PM, Jason Collins wrote: > I have spent a bunch of time (with support) trying to get to the bottom of > this sort of thing. > > There are 2 major contributors that I've discovered: > > (1) *Many namespaces.* If you have many namespaces, then Datastore > Statistics are no longer computed. You can tell by going to Datastore > Statistics. If it says "Last updated: [long time ago]", then you are in > this situation. When this happens, the Map-Reduce framework uses only 32 > shards, which can greatly extend the time to backup. > > (2) *General Files API failures.* If the Files API is having difficulty, > then the worker shard can fail. If it fails 10 times (I believe), then it > simply dies. The M-R jobs never completes and the controllers just keep on > retrying. This situation can also occur if you hit the daily limit for > Files API transfer (100 GB or so, I think). > > This is a pain and has caused me to continually manually clean queues, and > of course, there are Kinds that we simply can't get backed up. > > Hopefully this all gets better under the new GCS client > > j > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine. > > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
