Nick,
Please make sure this index definition gets added to the MapReduce
documentation (unless it's not that big of deal). I just happened to
stumble across the log entry; I'm sure many wouldn't notice it.
j
On Jun 9, 8:33 am, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:
> 2010/6/9 Jason C <jason.a.coll...@gmail.com>
>
>
>
> > > Why do you need to walk your data in __key__ reverse order?
>
> > This seems to be something that the MapReduce for App Engine does. It
> > looks like, when sharding, half the shards are from the asc index and
> > half are from the desc index, presumably to spread the scan load over
> > a pair of indexes instead of just hammering one.
>
> No, the reverse index is required in order to determine what the maximum key
> is, so it can lexicographically shard the work between minimum and maximum
> key. Without a descending index, the mapreduce is forced to do a binary
> search to find the maximum key.
>
> -Nick Johnson
>
>
>
>
>
>
>
> > j
>
> > On Jun 9, 2:05 am, Alkis Evlogimenos ('Αλκης Ευλογημένος)
> > <al...@evlogimenos.com> wrote:
> > > Why do you need to walk your data in __key__ reverse order?
>
> > > - alkis
>
> > > 2010/6/9 Jason C <jason.a.coll...@gmail.com>
>
> > > > Ok, thanks.
>
> > > > So it seems for MapReduce for App Engine to work optimally, I would
> > > > need to manually add the requested index (and then, somewhat
> > > > ironically, use MapReduce for App Engine to walk the entire data set
> > > > and put each entity to get the new index populated).
>
> > > > j
>
> > > > On Jun 8, 5:19 pm, Alkis Evlogimenos ('Αλκης Ευλογημένος)
> > > > <evlogime...@gmail.com> wrote:
> > > > > There is no desc index on __key__. Each indexed property has an asc
> > and
> > > > desc
> > > > > index on it but that doesn't apply for keys.
>
> > > > > - alkis
>
> > > > > On Wed, Jun 9, 2010 at 12:39 AM, Jason C <jason.a.coll...@gmail.com>
> > > > wrote:
> > > > > > I'm attempting to use mapreduce for app engine as presented at I/O
> > > > > > 2010.
>
> > > > > > I just noticed the following warning message in our logs (on /
> > > > > > mapreduce/command/start_job):
>
> > > > > > Cannot create accurate approximation of keyspace, guessing instead.
> > > > > > Please address this problem: no matching index found.
> > > > > > This query needs this index:
> > > > > > - kind: AccountKeyword
> > > > > > properties:
> > > > > > - name: __key__
> > > > > > direction: desc
>
> > > > > > This seems to suggest that there is no (desc) index for the key of
> > our
> > > > > > Model. However, isn't this created automatically? Indeed, I can't
> > even
> > > > > > think of a way to _not_ have it created (i.e., I can't think of a
> > way
> > > > > > to specify indexed=False for the key).
>
> > > > > > Is this an error in reporting, or is something very wrong with our
> > > > > > Datastore?
>
> > > > > > Thanks,
> > > > > > j
>
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > Groups
> > > > > > "Google App Engine" group.
> > > > > > To post to this group, send email to
> > google-appengine@googlegroups.com
> > > > .
> > > > > > To unsubscribe from this group, send email to
> > > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib
> > > > > > e...@googlegroups.com><google-appengine%2Bunsubscrib
> > e...@googlegroups.com><google-appengine%2Bunsubscrib
> > > > e...@googlegroups.com>
> > > > > > .
> > > > > > For more options, visit this group at
> > > > > >http://groups.google.com/group/google-appengine?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Google App Engine" group.
> > > > To post to this group, send email to google-appengine@googlegroups.com
> > .
> > > > To unsubscribe from this group, send email to
> > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib
> > > > e...@googlegroups.com><google-appengine%2Bunsubscrib
> > e...@googlegroups.com>
> > > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appeng...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com<google-appengine%2Bunsubscrib
> > e...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. ::
> Registered in Dublin, Ireland, Registration Number: 368047
> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
> 368047
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
