On Fri, Feb 19, 2010 at 12:28 AM, peterk <peter.ke...@gmail.com> wrote:
> What about keynames like: > > counter_standard_dbf > counter_standard_clo > > or would something like > > dbfo01la_counter_standard > clo091b_counter_standard > > work better? > > I'm thinking of cases where you may use keynames that can in some way > be constructed/predicted for fast access later. > like..<username>_counter_standard > > Would the common pre-fix or post-fix make for close distribution? :| > Either one will work fine - Bigtable will split tablets based on key to ensure no tablet gets too big. Long identical prefixes just mean that the split will be based on later characters in the string. What's important for key distribution for really high update rates is the distribution of key names/IDs for those updates: If they all go to a single tablet (eg, they make up a small proportion of the total range of IDs you're employing), they will be limited by what that tablet server can support. If they are widely spread out within the range you're using, regardless of what that range is, you'll be fine. -Nick Johnson > > > > On Feb 18, 5:59 pm, "Nick Johnson (Google)" <nick.john...@google.com> > wrote: > > Hi Eli, > > > > Using a randomly generated ID like a uuid is perfectly satisfactory to > > achieve an even distribution. > > > > On Wed, Feb 17, 2010 at 7:02 PM, Eli Jones <eli.jo...@gmail.com> wrote: > > > I understand the process of evenly distributing IDs since they are > Integer > > > values.. is there a canonized appengine way to evenly distribute > key_names? > > > > > Just make sure key_name1 and key_name2 don't have their i-th letters > "too > > > close" too eachother? How far is far enough? > > > > > Does doing even distribution matter if you aren't using auto-generated > IDs? > > > > It certainly can - if you insert, in order, "aaaa", "aaab", "aaac", etc, > > you'll encounter the same problem at very high volumes as you'd see with > > auto generated IDs. > > > > -Nick Johnson > > > > > > > > > > > > > > > > > Thanks for information. > > > > > On Wed, Feb 17, 2010 at 1:32 PM, Nick Johnson (Google) < > > > nick.john...@google.com> wrote: > > > > >> Hi Ulrich, > > > > >> On Wed, Feb 17, 2010 at 5:30 PM, Ulrich <mierendo...@googlemail.com > >wrote: > > > > >>> Hi, > > > > >>> I have read the following > > >>> "Timeouts due to datastore issues --- [...] The most common example > of > > >>> this occurs when you are rapidly inserting a large number of entities > > >>> of the same kind, with auto-generated IDs. In this case, most inserts > > >>> hit the same range of the same tablet, and the single tablet server > is > > >>> overwhelmed with writes. [...] If this does affect your app, the > > >>> easiest solution is to use more evenly distributed IDs instead of the > > >>> auto-allocated ones [...]" > > >>> ( > > >>> > http://code.google.com/appengine/articles/handling_datastore_errors.html > > >>> ) > > > > >>> Let's say I am having a model "Parent" and a model "Child". For > Parent > > >>> entities, I use key names that are evenly distributed. For Child > > >>> entities, I use auto-generated key IDs and _no_ key names, but all > > >>> Child entities are children of Parent entities, so the paths to the > > >>> children contain the evenly distributes key names of the parents. > > >>> If I have many write operations on children that are in the same > > >>> entity group, the described error could occur. But what happens if my > > >>> write operations are on children that are in different entity groups? > > >>> Their IDs are auto-generated and not evenly distributed, but their > > >>> paths contain the evenly distributed key names. > > > > >> Good question! The point being made in the article refers to the > global > > >> distribution of the complete key, so writes to these children will be > well > > >> distributed, and you won't have to worry about this source of > contention. > > > > >> -Nick Johnson > > > > >>> -- > > > > >>> You received this message because you are subscribed to the Google > Groups > > >>> "Google App Engine" group. > > >>> To post to this group, send email to > google-appeng...@googlegroups.com. > > >>> To unsubscribe from this group, send email to > > >>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > >>> . > > >>> For more options, visit this group at > > >>>http://groups.google.com/group/google-appengine?hl=en. > > > > >> -- > > >> Nick Johnson, Developer Programs Engineer, App Engine > > >> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > Number: > > >> 368047 > > > > >> -- > > >> You received this message because you are subscribed to the Google > Groups > > >> "Google App Engine" group. > > >> To post to this group, send email to > google-appeng...@googlegroups.com. > > >> To unsubscribe from this group, send email to > > >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > >> . > > >> For more options, visit this group at > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to google-appengine@googlegroups.com > . > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > Nick Johnson, Developer Programs Engineer, App Engine > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > Number: > > 368047 > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
