On Wed, Mar 10, 2010 at 12:35 PM, Nickolas Daskalou <n...@daskalou.com>wrote:
> Hi Nick, > > Thanks for the reply. I've got a few question below that I'm hoping you can > answer. > > On 10 March 2010 22:10, Nick Johnson (Google) <nick.john...@google.com>wrote: > >> Hi Nickolas, >> >> On Wed, Mar 10, 2010 at 12:28 AM, Nickolas Daskalou <n...@daskalou.com>wrote: >> >>> I remember reading a thread not long ago ( >>> http://groups.google.com/group/google-appengine/browse_thread/thread/f69fe7dc4a9bc2ec) >>> about how put()'ing entities with near-sequential key names or IDs could >>> result in Datastore contention, since those entities would most likely be >>> saved on the same Bigtable tablet. >>> >>> One solution in that thread to avoid Datastore contention mentioned >>> setting the key names to a hash (thus achieving non-sequential key names). >>> >>> My question is does this sequential-causes-contention rule also apply to >>> the indexes updated when the entities are put()? I'm assuming the answer is >>> Yes. >>> >> >> In theory, yes. However, it would take a much higher write rate than for >> entity contention before it becomes an issue - so high that you likely do >> not have to worry about it. >> > > Is it higher than the "100s of qps" mentioned in that other thread? > Yes. > Is there a rough figure you can give? > Since it's never been encountered in a live app, it's hard to say for certain - although that should tell you something about the likelihood of it happening at all. :) > I've personally observed instances of write contention with sequential key >> IDs, where a datetime was also being indexed, and contention with entity >> writes was encountered long before any issues with the index would have >> become a problem. >> > > What was your qps rate before you encountered contention issues? Have you > tried the same thing with non-sequential key IDs? If so, what was the qps > rate you were able to achieve without contention issues on the datetime > index? > The event in question is actually what prompted the addition of that part of the article you're referring to. The user subsequently switched to using well-distributed key names, and the issues went away. I don't know what the exact QPS level was for their tests - high hundreds, at a minimum. -Nick Johnson > Nick > > > > >> -Nick Johnson >> >> >>> An example of what I mean. Let's say I'm taking snapshots of integer >>> values at particular points in time (say, every 5 minutes). So my model >>> looks like this: >>> >>> class Snapshot(db.Model): >>> datetime = db.DateTimeProperty() >>> value = db.IntegerProperty() >>> >>> def new_snapshot(d,v): >>> return Snapshot(datetime=d, value=v, key_name=sha1('%s-%s-%s %s:%s,%s' >>> % (d.year,d.month,d.day,d.hour,d.min,v)) >>> >>> If I then create 1000 entities which all have the same datetime value >>> but all have different value values, will I still encounter Datastore >>> contention, due to the datetime index being updated with 1000 values >>> that are identical, and hence (if my understanding of indexes is >>> correct) will most likely be written onto the same tablet? >>> >>> Nick >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appeng...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>> . >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> >> >> -- >> Nick Johnson, Developer Programs Engineer, App Engine >> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: >> 368047 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
