Say I did a batch update of 500 entities all of the same model...could this breach the '100s of qps' requirement that could lead to tablets getting too hot? I've seen benches (http://blog.dantup.com/pi/ bm_put_perf.png) that show 500 entities being batch put in ~4s which suggests an average put rate that might be in the 100+ per second in such a case.
Or would that be 'ok'? :) I'm guessing if I were in a situation where multiple such batch updates could be occurring simultaneously or in a tight timeframe, then I'd be more likely to run into this...depending on how busy my app became, I'd possibly need to start doing that (i.e. lots of such large batch updates happening in short timeframes). On Feb 19, 1:12 pm, "Nick Johnson (Google)" <nick.john...@google.com> wrote: > Hi Peter, > > On Fri, Feb 19, 2010 at 1:04 PM, peterk <peter.ke...@gmail.com> wrote: > > Thanks Nick, I understand now. So I guess the easiest thing to do is > > to have a random component in your keynames...at least for apps I'm > > considering I don't think I'd have any other way to reasonably ensure > > the range of keynames in a given (batch) update were well distributed. > > Bear in mind that you only have to even worry about this if you're expecting > hundreds of QPS of inserts to the same model. > > If you are in this situation, hashing some stable information from your > model may be sufficient to generate a well distributed key name. > > -Nick Johnson > > > > > > > On Feb 19, 11:58 am, "Nick Johnson (Google)" <nick.john...@google.com> > > wrote: > > > On Fri, Feb 19, 2010 at 12:28 AM, peterk <peter.ke...@gmail.com> wrote: > > > > What about keynames like: > > > > > counter_standard_dbf > > > > counter_standard_clo > > > > > or would something like > > > > > dbfo01la_counter_standard > > > > clo091b_counter_standard > > > > > work better? > > > > > I'm thinking of cases where you may use keynames that can in some way > > > > be constructed/predicted for fast access later. > > > > like..<username>_counter_standard > > > > > Would the common pre-fix or post-fix make for close distribution? :| > > > > Either one will work fine - Bigtable will split tablets based on key to > > > ensure no tablet gets too big. Long identical prefixes just mean that the > > > split will be based on later characters in the string. > > > > What's important for key distribution for really high update rates is the > > > distribution of key names/IDs for those updates: If they all go to a > > single > > > tablet (eg, they make up a small proportion of the total range of IDs > > you're > > > employing), they will be limited by what that tablet server can support. > > If > > > they are widely spread out within the range you're using, regardless of > > what > > > that range is, you'll be fine. > > > > -Nick Johnson > > > > > On Feb 18, 5:59 pm, "Nick Johnson (Google)" <nick.john...@google.com> > > > > wrote: > > > > > Hi Eli, > > > > > > Using a randomly generated ID like a uuid is perfectly satisfactory > > to > > > > > achieve an even distribution. > > > > > > On Wed, Feb 17, 2010 at 7:02 PM, Eli Jones <eli.jo...@gmail.com> > > wrote: > > > > > > I understand the process of evenly distributing IDs since they are > > > > Integer > > > > > > values.. is there a canonized appengine way to evenly distribute > > > > key_names? > > > > > > > Just make sure key_name1 and key_name2 don't have their i-th > > letters > > > > "too > > > > > > close" too eachother? How far is far enough? > > > > > > > Does doing even distribution matter if you aren't using > > auto-generated > > > > IDs? > > > > > > It certainly can - if you insert, in order, "aaaa", "aaab", "aaac", > > etc, > > > > > you'll encounter the same problem at very high volumes as you'd see > > with > > > > > auto generated IDs. > > > > > > -Nick Johnson > > > > > > > Thanks for information. > > > > > > > On Wed, Feb 17, 2010 at 1:32 PM, Nick Johnson (Google) < > > > > > > nick.john...@google.com> wrote: > > > > > > >> Hi Ulrich, > > > > > > >> On Wed, Feb 17, 2010 at 5:30 PM, Ulrich < > > mierendo...@googlemail.com > > > > >wrote: > > > > > > >>> Hi, > > > > > > >>> I have read the following > > > > > >>> "Timeouts due to datastore issues --- [...] The most common > > example > > > > of > > > > > >>> this occurs when you are rapidly inserting a large number of > > entities > > > > > >>> of the same kind, with auto-generated IDs. In this case, most > > inserts > > > > > >>> hit the same range of the same tablet, and the single tablet > > server > > > > is > > > > > >>> overwhelmed with writes. [...] If this does affect your app, the > > > > > >>> easiest solution is to use more evenly distributed IDs instead of > > the > > > > > >>> auto-allocated ones [...]" > > > > > >>> ( > > >http://code.google.com/appengine/articles/handling_datastore_errors.html > > > > > >>> ) > > > > > > >>> Let's say I am having a model "Parent" and a model "Child". For > > > > Parent > > > > > >>> entities, I use key names that are evenly distributed. For Child > > > > > >>> entities, I use auto-generated key IDs and _no_ key names, but > > all > > > > > >>> Child entities are children of Parent entities, so the paths to > > the > > > > > >>> children contain the evenly distributes key names of the parents. > > > > > >>> If I have many write operations on children that are in the same > > > > > >>> entity group, the described error could occur. But what happens > > if my > > > > > >>> write operations are on children that are in different entity > > groups? > > > > > >>> Their IDs are auto-generated and not evenly distributed, but > > their > > > > > >>> paths contain the evenly distributed key names. > > > > > > >> Good question! The point being made in the article refers to the > > > > global > > > > > >> distribution of the complete key, so writes to these children will > > be > > > > well > > > > > >> distributed, and you won't have to worry about this source of > > > > contention. > > > > > > >> -Nick Johnson > > > > > > >>> -- > > > > > > >>> You received this message because you are subscribed to the > > Google > > > > Groups > > > > > >>> "Google App Engine" group. > > > > > >>> To post to this group, send email to > > > > google-appeng...@googlegroups.com. > > > > > >>> To unsubscribe from this group, send email to > > > > > >>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib > > > > > >>> e...@googlegroups.com><google-appengine%2Bunsubscrib > > e...@googlegroups.com><google-appengine%2Bunsubscrib > > > > e...@googlegroups.com> > > > > > >>> . > > > > > >>> For more options, visit this group at > > > > > >>>http://groups.google.com/group/google-appengine?hl=en. > > > > > > >> -- > > > > > >> Nick Johnson, Developer Programs Engineer, App Engine > > > > > >> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > > > Number: > > > > > >> 368047 > > > > > > >> -- > > > > > >> You received this message because you are subscribed to the Google > > > > Groups > > > > > >> "Google App Engine" group. > > > > > >> To post to this group, send email to > > > > google-appeng...@googlegroups.com. > > > > > >> To unsubscribe from this group, send email to > > > > > >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib > > > > > >> e...@googlegroups.com><google-appengine%2Bunsubscrib > > e...@googlegroups.com><google-appengine%2Bunsubscrib > > > > e...@googlegroups.com> > > > > > >> . > > > > > >> For more options, visit this group at > > > > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > > > "Google App Engine" group. > > > > > > To post to this group, send email to > > google-appengine@googlegroups.com > > > > . > > > > > > To unsubscribe from this group, send email to > > > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib > > > > > > e...@googlegroups.com><google-appengine%2Bunsubscrib > > e...@googlegroups.com><google-appengine%2Bunsubscrib > > > > e...@googlegroups.com> > > > > > > . > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > > > > > Nick Johnson, Developer Programs Engineer, App Engine > > > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > > > Number: > > > > > 368047 > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to google-appengine@googlegroups.com > > . > > > > To unsubscribe from this group, send email to > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib > > > > e...@googlegroups.com><google-appengine%2Bunsubscrib > > e...@googlegroups.com> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > > Nick Johnson, Developer Programs Engineer, App Engine > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > Number: > > > 368047 > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib > > e...@googlegroups.com> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Nick Johnson, Developer Programs Engineer, App Engine > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
