Hi David, I was the Googler to originally observe this, in the form of 'hot tablets' on Bigtable. The specific problem only occurs, in my experience, at write rates in excess of a couple of hundred QPS, and in situations where all of those writes are being directed to a single tablet (row range). This is a level of write traffic that few, users are likely to encounter.
The issue also occurs at slightly lower traffic levels if the ID of the entity is being auto-allocated, due to the higher overhead of writing the entire entity instead of just an index row. This is usually trivially alleviated by switching to key names - either randomly allocated (such as a UUID) or based on something that isn't monotonic, such as email address. What level of writes are you experiencing the issue at, and how did you determine that this was the cause of your issue? -Nick Johnson On Wed, Jan 19, 2011 at 5:30 AM, David Mora <dla.m...@gmail.com> wrote: > Quoting Ikai > > "Just be aware of monotonically increasing indexes like timestamps: if > you have an application with a high write rate that has a timestamp, > this will cause the persistence unit storing the indexes to be unable > to be autosplit easily across multiple hardware instances and you will > take a performance hit. The solution here is, again, to shard the > timestamp by prefixing a value to distribute the writes." > > This was in another threat and was left out of the conversation. > > From my perspective, this is a huge improvement since most of the > times we have to deal timestamps. We handle an application that uses > the problematic approach and it's just a bottle neck. We are starting > to switch this and notice two things: > > - it will be nice if the data store implementation can supply the > functionality needed to shard it (since seems like a change down deep > in the layer). > - What would be the best approach to solve this sharding? would unix > timestamps would be enough for this? > > > Appreciate your feedback > > > -- > http://about.me/david.mora > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
