[google-appengine] Re: Bulk data deletion woe
Here's what I have done that seems to work: 1) select the entities you want to delete from the datastore admin page as you described 2) Manually monitor the cup usage in the dashboard or quota page 3) Once the quota reaches a high value (say 75-80% for example), go to the Task Queues page of the admin console 4) You should see a default queue, that contains the worker tasks that are deleting your data 5) Use the Pause Queue button to pause the default queue 6) Wait until your quota is reset (done once per day) 7) With a new quota, go and resume the default task queue. Repeat from step 2. If you are deleting less than 1GB of data, you can probably do this in one or two passes Note: How high you let the quota go in step #3 above depends on how much CPU your app normally needs... make sure to leave enough for your app to function. On Nov 14, 4:25 am, Justin justin.worr...@gmail.com wrote: I've been trying to bulk delete data from my application as described here http://code.google.com/appengine/docs/python/datastore/creatinggettin... This seems to have kicked off a series of mapreduce workers, whose execution is killing my CPU - approximately 5 mins later I have reached 100% CPU time and am locked out for the rest of the day. I figure I'll just delete by hand; create some appropriate :delete controllers and wait till the next day. Unfortunately the mapreduce process still seems to be running - 10 past midnight and my CPU has reached 100% again. Is there some way to kill these processes and get back control of my app? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: App engine down time or problem with CRON requests
appid is gsrmon. More outages today, from more outages today 18:14-18:40 UTC and 17:16-17:35 UTC. This is still with a two-minute cron task. The jobs are completely dropped, there are not a number of make-up requests sent once the system recovers. Honestly I'm not even sure that would be a great thing to do. I am specifically throttling cron- issued requests at 1 or 2 minute intervals, if 30 minutes worth of crons are missed, having a burst of 30 requests doesn't really help. chris On Oct 27, 1:54 am, Mickey Kataria (Google Employee) mic...@google.com wrote: Thanks for the reports, we're looking into it. In general, if there's an app engine issue that causes cron to miss a job, it will be executed once the system recovers, i.e. jobs are not entirely dropped. The exact time the task will execute depends on the frequency and scheduling. If you're seeing an issue where a task was completely dropped and never executed, please provide details about your app id and what you're seeing. Thanks. On Oct 24, 11:53 pm, Chris Prinos chrispri...@gmail.com wrote: Ok thanks for confirming you saw this as well. Looks like the same thing occured today, same time frame as before. Chris On Oct 22, 11:02 pm, sodso sodhisoluti...@gmail.com wrote: Yes, same happened for me as well, looks like CRON jobs arent reliable, so i have decided to implement double-checking logic to check whether the job has run or not by using a datatstore / memcache record On Oct 22, 7:51 pm, Chris Prinos chrispri...@gmail.com wrote: Anyone know if there was scheduled down time 10/22/2010 around 12:30AM PDT? None of my cron-initiated requests were triggered from about 12:33 AM PDT to 1:05 AM PDT, but there were no errors in the logs. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: App engine down time or problem with CRON requests
btw, I have logged these also as issues 3495, 4054. The first of those was closed when the acute issue of 1 minute crons not working was fixed, but the larger issue of cron failures remains. Since keeping track from about November 6th, I've seen six distinct outage periods with my app. thanks, chris On Nov 16, 1:49 pm, Chris Prinos chrispri...@gmail.com wrote: appid is gsrmon. More outages today, from more outages today 18:14-18:40 UTC and 17:16-17:35 UTC. This is still with a two-minute cron task. The jobs are completely dropped, there are not a number of make-up requests sent once the system recovers. Honestly I'm not even sure that would be a great thing to do. I am specifically throttling cron- issued requests at 1 or 2 minute intervals, if 30 minutes worth of crons are missed, having a burst of 30 requests doesn't really help. chris On Oct 27, 1:54 am, Mickey Kataria (Google Employee) mic...@google.com wrote: Thanks for the reports, we're looking into it. In general, if there's an app engine issue that causes cron to miss a job, it will be executed once the system recovers, i.e. jobs are not entirely dropped. The exact time the task will execute depends on the frequency and scheduling. If you're seeing an issue where a task was completely dropped and never executed, please provide details about your app id and what you're seeing. Thanks. On Oct 24, 11:53 pm, Chris Prinos chrispri...@gmail.com wrote: Ok thanks for confirming you saw this as well. Looks like the same thing occured today, same time frame as before. Chris On Oct 22, 11:02 pm, sodso sodhisoluti...@gmail.com wrote: Yes, same happened for me as well, looks like CRON jobs arent reliable, so i have decided to implement double-checking logic to check whether the job has run or not by using a datatstore / memcache record On Oct 22, 7:51 pm, Chris Prinos chrispri...@gmail.com wrote: Anyone know if there was scheduled down time 10/22/2010 around 12:30AM PDT? None of my cron-initiated requests were triggered from about 12:33 AM PDT to 1:05 AM PDT, but there were no errors in the logs. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1-minute cron jobs are not getting executed on all apps
More outages again this morning. All of my crons (not just 1-minute) were dead from about 4-4:30am EST this morning -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Cron jobs not getting executed
I'v had problems as well, 1-minute crons were not being called starting around 2:26:am EST. They started working again around 5am EST, then stopped again a couple of hours later. Currently they are not working. I changed my cron config to 2-minutes and those are working. I had other crons that were at 5 and 10 minute intervals, those had no issues. chris On Nov 11, 10:55 am, dflorey daniel.flo...@gmail.com wrote: Hi, one of my customers has problems with his app: trecontacts We havecronjobs scheduled for every minute, but they have not run for the last hours. Can someone please check? Thanks, Daniel -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: App engine down time or problem with CRON requests
Ok thanks for confirming you saw this as well. Looks like the same thing occured today, same time frame as before. Chris On Oct 22, 11:02 pm, sodso sodhisoluti...@gmail.com wrote: Yes, same happened for me as well, looks like CRON jobs arent reliable, so i have decided to implement double-checking logic to check whether the job has run or not by using a datatstore / memcache record On Oct 22, 7:51 pm, Chris Prinos chrispri...@gmail.com wrote: Anyone know if there was scheduled down time 10/22/2010 around 12:30AM PDT? None of my cron-initiated requests were triggered from about 12:33 AM PDT to 1:05 AM PDT, but there were no errors in the logs. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] App engine down time or problem with CRON requests
Anyone know if there was scheduled down time 10/22/2010 around 12:30AM PDT? None of my cron-initiated requests were triggered from about 12:33 AM PDT to 1:05 AM PDT, but there were no errors in the logs. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE cold start time still too long
I see the same thing (DeadlineExceeded) over the past couple of days with a cron job that writes a few bytes of date to a single table. It had been working fine for weeks, no code changes. Over the past 10 hours, 6% of these cron requests have failed because of the ap engine errors. Chris On Feb 22, 12:42 pm, Anders i...@blabline.com wrote: But are not deadline exceeded errors something that happens only after an application has been instantiated, i.e. the execution of a request taking too long time, not including the startup time for the whole application? And that Django can cause several seconds of extra delay doesn't sound plausible. No, I think this is something in the GAE infrastructure itself, independent of Java or Python etc. Google should look into it, lest IBM will beat them with a WebSphereCloud or something like that, or Sun Microsystem with a JavaCloud, or Sony with a PlayStationCloud, or, God forbid, Microsoft with a DOSCloud. ;-) On Feb 22, 6:11 pm, GAEfan ken...@gmail.com wrote: Yes, I have been getting Deadline Exceeded errors again, starting about Friday. I thought this was fixed, as I hadn't seen them for a while. But they are back again. :-( AFAIK, it is the cold-start loading of Django that is the cause of the delay. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
