Hi Jason, The current implementation of the datastore mapper uses lexicographical sharding over keys to assign datastore shards. Unfortunately, this can lead to very inconsistent shard sizes, as you observe.
-Nick Johnson On Fri, Jun 11, 2010 at 4:17 PM, Jason C <jason.a.coll...@gmail.com> wrote: > We've been using MapReduce for App Engine for a couple of different > jobs. > > Typically, we use 8 shards (the default), but it seems that only 3, > sometime 4, of the shards have any items in them? E.g., we're > currently running one job and three of the shards have >218,000 items > processed, but the other 5 shards appear to have zero. > > I can understand that a particular key distribution would have > different amounts in each shard, but with so many at zero, I suspect > there is something else happening? > > BTW, we have applied the mapreduce-recommended __key__ DESC index, but > we still see this strange shard distribution. > > Is anyone else seeing this? > > j > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
