Steve, I thought I had gotten everything out of that talk but clearly I was sleeping in class!
The memcache fork join queue looks very interesting - I'm definitely giving that a try. Nikolaj On Sep 23, 1:00 am, Steve Sherrie <st...@wasteofpaper.com> wrote: > Definitely checkout Brett Slatkin's Building high-throughput data > pipelines with Google App Engine > <http://www.google.com/events/io/2010/sessions/high-throughput-data-pi...> > I/O Session which should give you a fundamental understanding of the > caveats involved with batch writing. > > You have to do a little work to make sure writes are not added to a > batch that's already applied (or is otherwise invalid), but Brett > explains how to do that with memcache locking. > > Steve > > On 11-09-22 07:11 PM, Brandon Wirtz wrote: > > > > > > > > > Start a another datastore that doesn't use the same entity, but uses an > > incremental system. > > > On Write: > > In a table with format > > Writes to do, Entity to Overwrite, Data for Entity > > 1,Brandon Wirtz, +$120 > > 2,Brandon Wirtz, -$120 > > 3,Brandon Wirtz, +$0.66 > > > Increment Write to do count, and store transaction. > > > If a Read request for the entity comes in: > > Get Entity > > Get Latest committed "Write To Do Value" > > Check Write Cache > > Recalculate Value based on Entity Value and All Changes > > > On Batch: > > For batch size calculate all values, and update entities > > Store Late todo value as Last committed write. > > > You can now batch as often as every 2 seconds and not hit the entity write > > limit. > > > -----Original Message----- > > From: google-appengine@googlegroups.com > > [mailto:google-appengine@googlegroups.com] On Behalf Of Nikolaj > > Sent: Thursday, September 22, 2011 3:59 PM > > To: Google App Engine > > Subject: [google-appengine] How to throttle or batch transactional updates > > to an entity group > > > Hi there, > > > I'm developing an application that requires me to create lots of child > > entities within the same entity group transactionally. The tasks being > > generated to do this come in unpredictable bursts - sometimes 100/s for an > > entity group. This is clearly exceeding the 1/s limit for transaction > > contention and causing some real problems for my poor application. > > > I need a way to either batch the required updates for each entity group so I > > can run through them serially or otherwise schedule each update to run on a > > 1/s basis. The ideas I've come up with so far: > > > 1. Use some kind of memcache'd timestamp or counter to schedule or throttle > > the updates. My question is how to do this in a way that will work with > > 100's of tasks hungrily looking for their time slot? Should I do a CAS for > > when the last task was scheduled (would this even work with the contention > > we're talking about?) or can I use an atomic operation like incr() somehow? > > > 2. Store the work items somewhere and then at some later time query for the > > work items for an entity group and create the children in serial. I'd like > > to avoid this, if possible, as it creates many other headaches. My > > application is already using the taskqueue to great effect. > > > This really seems like a queue problem - if only there was some kind of > > dynamic subqueue where I could add tasks with a 'namespace' of the entity > > group. Then entity groups could run in parallel, but the entity group items > > would still be in serial. Does something like that exist? > > > Any suggestions or experiences on how I can work around this would be > > appreciated. > > > Many thanks, > > > Nikolaj > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
