Right now, I've basically been doing as little work as possible in a single task-
Fetching an iCal file in one task and persisting it, enqueueing a task to split it into single event snippets, and then each snippet gets processed in it's own task. With a 10-minute task queue, I could easily do all of that in one fell swoop-- in fact, I could loop through some number of feeds in a single task. The question I'm getting at: Is there still a reason to prefer lots of quick tasks over fewer, longer running tasks? On Fri, Nov 19, 2010 at 1:33 PM, Ikai Lan (Google) <ikai.l+gro...@google.com<ikai.l%2bgro...@google.com> > wrote: > Web requests share the fast pool, so there is a chance you can block all > your web facing instances from serving web requests. Additionally, the > latency is averaged when considering whether or not to autoscale your > instances, so putting slow work in a fast queue would prevent App Engine > from adding additional instances as needed. > > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blogger: http://googleappengine.blogspot.com > Reddit: http://www.reddit.com/r/appengine > Twitter: http://twitter.com/app_engine > > > > On Fri, Nov 19, 2010 at 10:27 AM, Ross M Karchner > <rosskarch...@gmail.com>wrote: > >> What are the consequences of a task being in the fast pool or the slow >> pool? >> >> On Fri, Nov 19, 2010 at 1:07 PM, Ikai Lan (Google) < >> ikai.l+gro...@google.com <ikai.l%2bgro...@google.com>> wrote: >> >>> Hi Julian, >>> >>> Long running user facing requests are bad for the ecosystem, yes. Long >>> running background tasks, however, can be placed into a different execution >>> pool. Fast background tasks are placed into the same execution pool as your >>> web facing requests. >>> >>> The rule of thumb here is to create different queues for tasks you know >>> will run fast (sub-second) and queues for slow running jobs. >>> >>> -- >>> Ikai Lan >>> Developer Programs Engineer, Google App Engine >>> Blogger: http://googleappengine.blogspot.com >>> Reddit: http://www.reddit.com/r/appengine >>> Twitter: http://twitter.com/app_engine >>> >>> >>> >>> On Thu, Nov 18, 2010 at 11:52 PM, Julian Namaro >>> <namarojul...@gmail.com>wrote: >>> >>>> Wow a lot of work packed there, congrats App Engine team! >>>> >>>> I'm surprised about the new 10 minutes deadline on Cron and Task >>>> Queues. A short while back you were explaining that long-running >>>> requests are bad for the App Engine ecosystem. Is it not the case for >>>> Task Queues or are you just confident that the system has improved >>>> enough to handle it now ? >>>> >>>> 10 minutes is a lot of time. If it works out well it will open the >>>> door to a lot of new possibilities :) >>>> >>>> >>>> On Nov 19, 7:27 am, "Ikai Lan (Google)" >>>> <ikai.l+gro...@google.com<ikai.l%2bgro...@google.com> >>>> > >>>> wrote: >>>> > Hey everyone, >>>> > >>>> > I just wanted to let everyone know that prerelease SDK 1.4.0 is out! >>>> Get it >>>> > from the Google Code project: >>>> > >>>> > http://code.google.com/p/googleappengine/downloads/list >>>> > >>>> > We're still working on the docs and will have them ready for the final >>>> > release, so if there are any questions about how to use the new >>>> features, >>>> > feel free to ask on this thread and I'll do my best to clarify them. >>>> The >>>> > release notes are below. This is an EXCITING release: >>>> > >>>> > Python >>>> > ------------ >>>> > - The Always On feature allows applications to pay and keep 3 >>>> instances of >>>> > their >>>> > application always running, which can significantly reduce >>>> application >>>> > latency. >>>> > - Developers can now enable Warmup Requests. By specifying a handler >>>> in an >>>> > app's app.yaml, App Engine will attempt to to send a Warmup Request >>>> to >>>> > initialize new instances before a user interacts with it. This can >>>> reduce >>>> > the >>>> > latency an end-user sees for initializing your application. >>>> > - The Channel API is now available for all users. >>>> > - Task Queue has been officially released, and is no longer an >>>> experimental >>>> > feature. The API import paths that use 'labs' have been deprecated. >>>> Task >>>> > queue >>>> > storage will count towards an application's overall storage quota, >>>> and >>>> > will >>>> > thus be charged for. >>>> > - The deadline for Task Queue and Cron requests has been raised to 10 >>>> > minutes. >>>> > Datastore and API deadlines within those requests remain unchanged. >>>> > - For the Task Queue, developers can specify task retry_parameters in >>>> their >>>> > queue.yaml. >>>> > - Metadata Queries on the datastore for datastore kinds, namespaces, >>>> and >>>> > entity >>>> > properties are available. >>>> > - URLFetch allowed response size has been increased, up to 32 MB. >>>> Request >>>> > size >>>> > is still limited to 1 MB. >>>> > - The Admin Console Blacklist page lists the top blacklist rejected >>>> > visitors. >>>> > - The automatic image thumbnailing service supports arbitrary crop >>>> sizes up >>>> > to >>>> > 1600px. >>>> > - Overall average instance latency in the Admin Console is now a >>>> weighted >>>> > average over QPS per instance. >>>> > - The developer who uploaded an app version can download that >>>> version's code >>>> > using the appcfg.py download_app command. This feature can be >>>> disabled on >>>> > a per application basis in the admin console, under the >>>> 'Permissions' tab. >>>> > Once disabled, code download for the application CANNOT be >>>> re-enabled. >>>> > - Fixed an issue where custom Admin Console pages did not work for >>>> Google >>>> > Apps for your Domain users. >>>> > - Allow Django initialization to be moved to appengine_config.py to >>>> avoid >>>> > Django version conflicts when mixing webapp.template with pure >>>> Django. >>>> > http://code.google.com/p/googleappengine/issues/detail?id=1758 >>>> > - Fixed an issue in the dev_appserver where get_serving_url did not >>>> work >>>> > for transparent, cropped PNGs: >>>> > http://code.google.com/p/googleappengine/issues/detail?id=3887 >>>> > - Fixed an issue with the DatastoreFileStub. >>>> > http://code.google.com/p/googleappengine/issues/detail?id=3895 >>>> > >>>> > Java >>>> > --------- >>>> > - The Always On feature allows applications to pay and keep 3 >>>> instances of >>>> > their >>>> > application always running, which can significantly reduce >>>> application >>>> > latency. >>>> > - Developers can now enable Warmup Requests. By specifying a handler >>>> in an >>>> > app's appengine-web.xml, App Engine will attempt to to send a Warmup >>>> > Request >>>> > to initialize new instances before a user interacts with it. This >>>> can >>>> > reduce >>>> > the latency an end-user sees for initializing your application. >>>> > - The Channel API is now available for all users. >>>> > - Task Queue has been officially released, and is no longer an >>>> experimental >>>> > feature. The API import paths that use 'labs' have been deprecated. >>>> Task >>>> > queue >>>> > storage will count towards an application's overall storage quota, >>>> and >>>> > will >>>> > thus be charged for. >>>> > - The deadline for Task Queue and Cron requests has been raised to 10 >>>> > minutes. >>>> > Datastore and API deadlines within those requests remain unchanged. >>>> > - For the Task Queue, developers can specify task retry-parameters in >>>> their >>>> > queue.xml. >>>> > - Metadata Queries on the datastore for datastore kinds, namespaces, >>>> and >>>> > entity >>>> > properties are available. >>>> > - URL Fetch allowed response size has been increased, up to 32 MB. >>>> Request >>>> > size >>>> > is still limited to 1 MB. >>>> > - The Admin Console Blacklist page lists the top blacklist rejected >>>> > visitors. >>>> > - The automatic image thumbnailing service supports arbitrary crop >>>> sizes up >>>> > to >>>> > 1600px. >>>> > - Overall average instance latency in the Admin Console is now a >>>> weighted >>>> > average over QPS per instance. >>>> > - Added a low-level AysncDatastoreService for making calls to the >>>> datastore >>>> > asynchronously. >>>> > - Added a getBodyAsBytes() method to QueueStateInfo.TaskStateInfo, >>>> this >>>> > returns >>>> > the body of the task state as a pure byte-string. >>>> > - The whitelist has been updated to include all classes from >>>> javax.xml.soap. >>>> > - Fixed an issue sending email to multiple recipients. >>>> > http://code.google.com/p/googleappengine/issues/detail?id=1623 >>>> > >>>> > As usual, we value your feedback, so don't hesitate to evaluate these >>>> SDKs >>>> > and let us know. Be mindful that the server-side components have not >>>> been >>>> > deployed yet, so uploaded code shouldn't work. >>>> > >>>> > Happy coding! >>>> > >>>> > -- >>>> > Ikai Lan >>>> > Developer Programs Engineer, Google App Engine >>>> > Blogger:http://googleappengine.blogspot.com >>>> > Reddit:http://www.reddit.com/r/appengine >>>> > Twitter:http://twitter.com/app_engine >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to google-appeng...@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>>> . >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appeng...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>> . >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> >> >> -- >> Ross M Karchner >> http://eventgrinder.com >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Ross M Karchner http://eventgrinder.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
