That's definitely an option - in my app it is common to reach the transaction edge with a unpredictable number of tasks ready to be raised transactionally - not heaps, but potentially more than 5. It would be nice to be able to just put them on the queue. I guess serialising them into another task's payload which then raises them isn't such a bad option - more complex though, and I wonder what the difference in resource usage would be compared to just raising them in a single batch API call - probably marginal, but being a common pattern its worth trying to avoid complexity and/or overhead. Anyway, that was the main reason for the question - understanding the restriction to try and make a choice of architecture. Cheers,
Colin On Jul 7, 7:04 pm, Robert Kluin <robert.kl...@gmail.com> wrote: > Just raise one transactional task that in turn spawns a series of > (non-transactional) _named_ sub-tasks. Just remember to "pass" on a > "TaskAlreadyExists" error. > > See Brett Slatkin's 2010 IO talk on pipelines for additional discussion. > > Robert > > > > On Wed, Jul 7, 2010 at 12:49 PM, hawkett <hawk...@gmail.com> wrote: > > Thanks for the detail - pretty comfortable with the ins and outs of > > the task queue - any ideas about the 5 task limit? > > >> Also, don't confuse transactional tasks with the task queue. Transactional > >> tasks are the those that will all happen simultaneously or will graciously > >> fail without any partial commits. Due to the implementation details you > >> probably want to avoid intentionally doing lots of transaction updates on > >> the same few objects, since it is possible (if you are kicking lots of > >> simultaneous jobs off with the task queue) to shoot yourself in the foot > >> and > >> have a very small success rate. > > > I wasn't confused until I read this paragraph :) Transactional tasks > > do not need to operate on the same entity group as the transaction in > > which they are raised - this is one of their greatest strengths. They > > are fast to raise, so you can actually include quite a lot of work in > > a 'transaction' of this sort, all done in parallel. Generally if I > > want to do something in the same entity group as the originating > > transaction, I would do it right there in the transaction and not > > raise a task. So at least for me (and I expect for many others), > > transactional tasks are almost exclusively *not* operating on the same > > entity group as the transaction in which they are raised, unless > > perhaps they are being used in a continuation style bulk update, in > > which case they are likely to be in series and not cause contention. > > > Anyway, question still there :) Is the 5 task limit a restriction that > > is expected to remain indefinitely? Can I get around the restriction > > by raising more in a single batch API call - i.e. is it a restriction > > on API calls or actual tasks? Cheers, > > > Colin > > > On Jul 7, 4:46 pm, Nate Bauernfeind <nate.bauernfe...@gmail.com> > > wrote: > >> I have noticed that batch datastore calls run within a single transaction. > >> The maximum number of entities that can be added or deleted (and probably > >> modified, though I have not tried) was 500. I'm betting you could probably > >> wrap them around a single transaction. Though, from my experience, I > >> wouldn't really recommend doing this (since trying to commit two batches of > >> 500 to the datastore within the same call tended to time out for me). > > >> Also, don't confuse transactional tasks with the task queue. Transactional > >> tasks are the those that will all happen simultaneously or will graciously > >> fail without any partial commits. Due to the implementation details you > >> probably want to avoid intentionally doing lots of transaction updates on > >> the same few objects, since it is possible (if you are kicking lots of > >> simultaneous jobs off with the task queue) to shoot yourself in the foot > >> and > >> have a very small success rate. > > >> Transactions work with the task queue in such a way that the task will only > >> be added to the queue if no other piece of your commits fail. For example, > >> you wouldn't really want your app to run the "new user" code if you > >> couldn't > >> create the new user account for that user because someone else registered > >> that user name at the same time. I.E. Things on the task queue do not > >> continue running within the same transaction if they were created within > >> one. > > >> On Wed, Jul 7, 2010 at 4:22 AM, hawkett <hawk...@gmail.com> wrote: > >> > Hi, > > >> > This page - > >> >http://code.google.com/appengine/docs/python/datastore/transactions.html > >> > - says that we cannot raise more than 5 transactional tasks in a > >> > single transaction, and I wanted to check if this was a limit that you > >> > were hoping to raise, or if this is likely to be a long term > >> > restriction? Does this restriction limit API calls to the task queue > >> > or actual number of tasks - e.g. could I raise more than 5 by doing > >> > them in batch with a single call to the task queue API? Cheers, > > >> > Colin > > >> > -- > >> > You received this message because you are subscribed to the Google Groups > >> > "Google App Engine" group. > >> > To post to this group, send email to google-appeng...@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib > >> > e...@googlegroups.com> > >> > . > >> > For more options, visit this group at > >> >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
