Yeah it's a pretty in depth topic, and I know I've watched the video
several times over. I hope it works for you :)
Steve
On 11-09-23 07:06 PM, Nikolaj wrote:
Steve, I thought I had gotten everything out of that talk but clearly
I was sleeping in class!
The memcache fork join queue looks very interesting - I'm definitely
giving that a try.
Nikolaj
On Sep 23, 1:00 am, Steve Sherrie<st...@wasteofpaper.com> wrote:
Definitely checkout Brett Slatkin's Building high-throughput data
pipelines with Google App Engine
<http://www.google.com/events/io/2010/sessions/high-throughput-data-pi...>
I/O Session which should give you a fundamental understanding of the
caveats involved with batch writing.
You have to do a little work to make sure writes are not added to a
batch that's already applied (or is otherwise invalid), but Brett
explains how to do that with memcache locking.
Steve
On 11-09-22 07:11 PM, Brandon Wirtz wrote:
Start a another datastore that doesn't use the same entity, but uses an
incremental system.
On Write:
In a table with format
Writes to do, Entity to Overwrite, Data for Entity
1,Brandon Wirtz, +$120
2,Brandon Wirtz, -$120
3,Brandon Wirtz, +$0.66
Increment Write to do count, and store transaction.
If a Read request for the entity comes in:
Get Entity
Get Latest committed "Write To Do Value"
Check Write Cache
Recalculate Value based on Entity Value and All Changes
On Batch:
For batch size calculate all values, and update entities
Store Late todo value as Last committed write.
You can now batch as often as every 2 seconds and not hit the entity write
limit.
-----Original Message-----
From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Nikolaj
Sent: Thursday, September 22, 2011 3:59 PM
To: Google App Engine
Subject: [google-appengine] How to throttle or batch transactional updates
to an entity group
Hi there,
I'm developing an application that requires me to create lots of child
entities within the same entity group transactionally. The tasks being
generated to do this come in unpredictable bursts - sometimes 100/s for an
entity group. This is clearly exceeding the 1/s limit for transaction
contention and causing some real problems for my poor application.
I need a way to either batch the required updates for each entity group so I
can run through them serially or otherwise schedule each update to run on a
1/s basis. The ideas I've come up with so far:
1. Use some kind of memcache'd timestamp or counter to schedule or throttle
the updates. My question is how to do this in a way that will work with
100's of tasks hungrily looking for their time slot? Should I do a CAS for
when the last task was scheduled (would this even work with the contention
we're talking about?) or can I use an atomic operation like incr() somehow?
2. Store the work items somewhere and then at some later time query for the
work items for an entity group and create the children in serial. I'd like
to avoid this, if possible, as it creates many other headaches. My
application is already using the taskqueue to great effect.
This really seems like a queue problem - if only there was some kind of
dynamic subqueue where I could add tasks with a 'namespace' of the entity
group. Then entity groups could run in parallel, but the entity group items
would still be in serial. Does something like that exist?
Any suggestions or experiences on how I can work around this would be
appreciated.
Many thanks,
Nikolaj
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups "Google
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
