Re: [google-appengine] Re: How to throttle or batch transactional updates to an entity group
Yeah it's a pretty in depth topic, and I know I've watched the video several times over. I hope it works for you :) Steve On 11-09-23 07:06 PM, Nikolaj wrote: Steve, I thought I had gotten everything out of that talk but clearly I was sleeping in class! The memcache fork join queue looks very interesting - I'm definitely giving that a try. Nikolaj On Sep 23, 1:00 am, Steve Sherriest...@wasteofpaper.com wrote: Definitely checkout Brett Slatkin's Building high-throughput data pipelines with Google App Engine http://www.google.com/events/io/2010/sessions/high-throughput-data-pi... I/O Session which should give you a fundamental understanding of the caveats involved with batch writing. You have to do a little work to make sure writes are not added to a batch that's already applied (or is otherwise invalid), but Brett explains how to do that with memcache locking. Steve On 11-09-22 07:11 PM, Brandon Wirtz wrote: Start a another datastore that doesn't use the same entity, but uses an incremental system. On Write: In a table with format Writes to do, Entity to Overwrite, Data for Entity 1,Brandon Wirtz, +$120 2,Brandon Wirtz, -$120 3,Brandon Wirtz, +$0.66 Increment Write to do count, and store transaction. If a Read request for the entity comes in: Get Entity Get Latest committed Write To Do Value Check Write Cache Recalculate Value based on Entity Value and All Changes On Batch: For batch size calculate all values, and update entities Store Late todo value as Last committed write. You can now batch as often as every 2 seconds and not hit the entity write limit. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Nikolaj Sent: Thursday, September 22, 2011 3:59 PM To: Google App Engine Subject: [google-appengine] How to throttle or batch transactional updates to an entity group Hi there, I'm developing an application that requires me to create lots of child entities within the same entity group transactionally. The tasks being generated to do this come in unpredictable bursts - sometimes 100/s for an entity group. This is clearly exceeding the 1/s limit for transaction contention and causing some real problems for my poor application. I need a way to either batch the required updates for each entity group so I can run through them serially or otherwise schedule each update to run on a 1/s basis. The ideas I've come up with so far: 1. Use some kind of memcache'd timestamp or counter to schedule or throttle the updates. My question is how to do this in a way that will work with 100's of tasks hungrily looking for their time slot? Should I do a CAS for when the last task was scheduled (would this even work with the contention we're talking about?) or can I use an atomic operation like incr() somehow? 2. Store the work items somewhere and then at some later time query for the work items for an entity group and create the children in serial. I'd like to avoid this, if possible, as it creates many other headaches. My application is already using the taskqueue to great effect. This really seems like a queue problem - if only there was some kind of dynamic subqueue where I could add tasks with a 'namespace' of the entity group. Then entity groups could run in parallel, but the entity group items would still be in serial. Does something like that exist? Any suggestions or experiences on how I can work around this would be appreciated. Many thanks, Nikolaj -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: How to throttle or batch transactional updates to an entity group
Steve, I thought I had gotten everything out of that talk but clearly I was sleeping in class! The memcache fork join queue looks very interesting - I'm definitely giving that a try. Nikolaj On Sep 23, 1:00 am, Steve Sherrie st...@wasteofpaper.com wrote: Definitely checkout Brett Slatkin's Building high-throughput data pipelines with Google App Engine http://www.google.com/events/io/2010/sessions/high-throughput-data-pi... I/O Session which should give you a fundamental understanding of the caveats involved with batch writing. You have to do a little work to make sure writes are not added to a batch that's already applied (or is otherwise invalid), but Brett explains how to do that with memcache locking. Steve On 11-09-22 07:11 PM, Brandon Wirtz wrote: Start a another datastore that doesn't use the same entity, but uses an incremental system. On Write: In a table with format Writes to do, Entity to Overwrite, Data for Entity 1,Brandon Wirtz, +$120 2,Brandon Wirtz, -$120 3,Brandon Wirtz, +$0.66 Increment Write to do count, and store transaction. If a Read request for the entity comes in: Get Entity Get Latest committed Write To Do Value Check Write Cache Recalculate Value based on Entity Value and All Changes On Batch: For batch size calculate all values, and update entities Store Late todo value as Last committed write. You can now batch as often as every 2 seconds and not hit the entity write limit. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Nikolaj Sent: Thursday, September 22, 2011 3:59 PM To: Google App Engine Subject: [google-appengine] How to throttle or batch transactional updates to an entity group Hi there, I'm developing an application that requires me to create lots of child entities within the same entity group transactionally. The tasks being generated to do this come in unpredictable bursts - sometimes 100/s for an entity group. This is clearly exceeding the 1/s limit for transaction contention and causing some real problems for my poor application. I need a way to either batch the required updates for each entity group so I can run through them serially or otherwise schedule each update to run on a 1/s basis. The ideas I've come up with so far: 1. Use some kind of memcache'd timestamp or counter to schedule or throttle the updates. My question is how to do this in a way that will work with 100's of tasks hungrily looking for their time slot? Should I do a CAS for when the last task was scheduled (would this even work with the contention we're talking about?) or can I use an atomic operation like incr() somehow? 2. Store the work items somewhere and then at some later time query for the work items for an entity group and create the children in serial. I'd like to avoid this, if possible, as it creates many other headaches. My application is already using the taskqueue to great effect. This really seems like a queue problem - if only there was some kind of dynamic subqueue where I could add tasks with a 'namespace' of the entity group. Then entity groups could run in parallel, but the entity group items would still be in serial. Does something like that exist? Any suggestions or experiences on how I can work around this would be appreciated. Many thanks, Nikolaj -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
