Your approach sounds fine. As an alternative you could schedule the next task up front, then check to see if there is any work to be done, then proceed. That way you'll always have the next task scheduled no matter what might go wrong during processing.
On Mon, Dec 7, 2009 at 2:19 PM, Larry Cable <larry.ca...@gmail.com> wrote: > sorry I should have been more precise, essentially I am concerned > about blowing the 30sec per request processing/duration limit > > previously I was bulk uploading csv files and parsing them into POJOs > then persisting them via JDO to pre-populate my app data ... > > I was encountering both the 5sec DS operation and 30sec processing > quota exceptions with this model ... > > So, all I do initially is upload the CSV in Gae VFS ... then I > schedule a task > > the task then batches lines from the file, parses, then pesists > them ... if it gets an exception then it "quickly" (I hope) > (repeatedly) reschedules another task to upload ... > > Thanks > > - Larry > > On Dec 7, 11:22 am, "Max Ross (Google)" > <maxr+appeng...@google.com<maxr%2bappeng...@google.com> > > > wrote: > > Regarding the "per-request quota".... > > > > Are you worried about the number of requests or the duration of those > > requests? > > > > > > > > On Mon, Dec 7, 2009 at 11:17 AM, Larry Cable <larry.ca...@gmail.com> > wrote: > > > Hi Max, after sleeping on it, it became apparent that as you describe > > > the timeout is > > > in fact in the "client" and not the execution of the (servlet) task > > > itself ... > > > On Dec 7, 10:54 am, "Max Ross (Google)" > > > <maxr+appeng...@google.com<maxr%2bappeng...@google.com> > <maxr%2bappeng...@google.com <maxr%252bappeng...@google.com>> > > > > > wrote: > > > > Local task execution is using the local UrlFetchService > implementation > > > with > > > > its default timeout: 5 seconds. So, if the url that your task hits > takes > > > > more than 5 seconds to respond you'll see this exception. However, > this > > > > exception is from the client of the task url, not the task execution > > > > itself. The client will timeout after 5 seconds but the task should > > > still > > > > execute to completion - we don't interrupt it. Please let me know if > > > that's > > > > not the case. > > > > > so I concur with your assertion above ... > > > > > > Now, 5 seconds is clearly not enough time since the request limit for > > > apps > > > > is 30 seconds. This is a bug, and it's already fixed. It should go > out > > > > with the next SDK. In the meantime just try to ignore the timeouts. > > > > > great thanks ... > > > > > Since I am using GAE-VFS and tasks in order to upload, parse and > > > persist data via JDO into > > > the DataStore, I anticipate that the Servlet/Task will potentially > > > exceed the per-request quota > > > and throw ... I was planning on catching this and scheduling > > > additional tasks until all the file > > > processing is complete ... > > > > > any thoughts on this ... do you see any problems in doing so? > > > > > > As for retries, this is a known limitation: > > >http://code.google.com/appengine/docs/java/taskqueue/overview.html#Ta. > .. > > > > > > We'll get this brought in line with production shortly. > > > > > > Thanks, and sorry for the trouble. > > > > > no problem! ... > > > > > Thanks for the response! > > > > > > Max > > > > > > On Sun, Dec 6, 2009 at 9:57 PM, Tristan <sp00s...@gmail.com> wrote: > > > > > I've seen this as well. I've also noticed that if I purposefully > make > > > > > a task fail (by returning any HTTP status code outside of the range > > > > > 200-299), it doesn't get added back to queue for retrying. This > used > > > > > to work in 1.2.6. > > > > > > > -Tristan > > > > > > > On Dec 6, 4:18 pm, Larry Cable <larry.ca...@gmail.com> wrote: > > > > > > I just updated from 1.2.6 to 1.2.8 and my application code has > > > started > > > > > > failing locally. > > > > > > > > I am using task queue's in order to process uploaded files, this > > > > > > worked in 1.2.6 (although you had > > > > > > to manually fire the task queue from the _ah admin console) > > > > > > > > Now, it is failing under 1.2.8 (as it seems as though it is now > > > > > > automatically starting the queue) > > > > > > with the following exception: > > > > > > > > Dec 7, 2009 12:08:57 AM org.quartz.core.JobRunShell run > > > > > > SEVERE: Job default.a7f0abc6-ed81-4fa4-96f1-c09048a99883 threw an > > > > > > unhandled Exception: > > > > > > com.google.apphosting.api.ApiProxy$ApplicationException: > > > > > > ApplicationError: 5: http method POST against > > > > > > URLhttp://localhost:8080/admin/gae-tasks/upload/csv/irs/processor.task > > > > > > timed out. > > > > > > at > > > > > com.google.appengine.api.urlfetch.dev.LocalURLFetchService.fetch > > > > > > (LocalURLFetchService.java:214) > > > > > > at > > > > > com.google.appengine.api.labs.taskqueue.dev.UrlFetchJob.execute > > > > > > (UrlFetchJob.java:53) > > > > > > at > > > > > com.google.appengine.api.labs.taskqueue.dev.UrlFetchJob.execute > > > > > > (UrlFetchJob.java:46) > > > > > > at org.quartz.core.JobRunShell.run(JobRunShell.java:195) > > > > > > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run > > > > > > (SimpleThreadPool.java:520) > > > > > > Dec 7, 2009 12:08:57 AM org.quartz.core.ErrorLogger > schedulerError > > > > > > SEVERE: Job (default.a7f0abc6-ed81-4fa4-96f1-c09048a99883 threw > an > > > > > > exception. > > > > > > org.quartz.SchedulerException: Job threw an unhandled exception. > [See > > > > > > nested exception: com.google.apphosting.api.ApiProxy > > > > > > $ApplicationException: ApplicationError: 5: http method POST > against > > > > > URLhttp://localhost:8080/admin/gae-tasks/upload/csv/irs/processor.task > > > > > > timed out.] > > > > > > at org.quartz.core.JobRunShell.run(JobRunShell.java:206) > > > > > > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run > > > > > > (SimpleThreadPool.java:520) > > > > > > * Nested Exception (Underlying Cause) --------------- > > > > > > com.google.apphosting.api.ApiProxy$ApplicationException: > > > > > > ApplicationError: 5: http method POST against > > > > > > URLhttp://localhost:8080/admin/gae-tasks/upload/csv/irs/processor.task > > > > > > timed out. > > > > > > at > > > > > com.google.appengine.api.urlfetch.dev.LocalURLFetchService.fetch > > > > > > (LocalURLFetchService.java:214) > > > > > > at > > > > > com.google.appengine.api.labs.taskqueue.dev.UrlFetchJob.execute > > > > > > (UrlFetchJob.java:53) > > > > > > at > > > > > com.google.appengine.api.labs.taskqueue.dev.UrlFetchJob.execute > > > > > > (UrlFetchJob.java:46) > > > > > > at org.quartz.core.JobRunShell.run(JobRunShell.java:195) > > > > > > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run > > > > > > (SimpleThreadPool.java:520) > > > > > > > > it implies that there is some form of timeout occuring ... since > the > > > > > > app code has not changed and is not > > > > > > apparently getting invoked at all, I am somewhat at loss to know > how > > > > > > to resolve this! > > > > > > > > Has anyone else seen these problems? > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Google App Engine for Java" group. > > > > > To post to this group, send email to > > > > > google-appengine-j...@googlegroups.com. > > > > > To unsubscribe from this group, send email to > > > > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > <google-appengine-java%2bunsubscr...@googlegroups.com> > > > <google-appengine-java%2bunsubscr...@googlegroups.com> > > > > > . > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine-java?hl=en.-Hide > > > quoted text - > > > > > > - Show quoted text - > > > > > -- > > > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine for Java" group. > > > To post to this group, send email to > > > google-appengine-j...@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > <google-appengine-java%2bunsubscr...@googlegroups.com> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine-java?hl=en.- Hide > quoted text - > > > > - Show quoted text - > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
