For most queries the process goes something like this:
Index Scan -> Fetch Entity -> Send to user
For any query with a sort order or inequality filter on any property
other than __key__ we are still actually fetching the entity in the
datastore, so the only savings is the serialization and transmission
cost of sending the result to the user. This means that offset is
still typically quite costly.
On Apr 6, 11:42 am, "Ikai L (Google)" <ika...@google.com> wrote:
> Offset doesn't fetch data - it does an index scan. Going to the 10,000th
> result using offset will require us to pass 10,000 results first in our
> index, but we won't be retrieving those objects. Here's a bit of a
> simplification of indexes, entities and how these queries work. As you know,
> Bigtable is a key-value store with the ability to perform range scans.
> Suppose you have a Person entity. Your entries may look like this:
>
> Key
> Value
> --------------------------------------------------------------------------------------------------------------
> APPID:Person:1 { name:
> Ikai, favorite_food: donuts }
> APPID:Person:2 { name:
> Wesley, favorite_food: tacos }
> Index:APPID:Person:favorite_food:ASC:donuts:1 [ empty ]
> Index:APPID:Person:favorite_food:ASC:tacos:2 [ empty ]
>
> *order of values in keys may not match how we actually do it, it's just to
> illustrate my point
> ** argh, non-monospaced fonts. Sorry for the hideous table
>
> When we look for Person entities sorted by favorite_food, we essentially ask
> Bigtable to return all Keys that match:
>
> Index:APPID:Person:favorite_food:ASC*
>
> We get a list of Keys back and figure out keys. Using offsets, we would just
> pass the first N entries per the supplied offset.
>
> Cursors work by serializing the query position via the last index, allow us
> to go straight to the index in constant time and continue the range query.
>
> We've got a lot of content about how this works, though we probably haven't
> done the best job organizing it. I'd start
> here:http://code.google.com/appengine/articles/storage_breakdown.html
>
> I've been meaning to collect all of our talks and place them into a set of
> YouTube playlists. Why don't I do that right now ...
>
> On Sat, Apr 3, 2010 at 8:16 AM, John Patterson <jdpatter...@gmail.com>wrote:
>
>
>
> > How many pages do your users really want to see? Even Google search sets a
> > limit - I think 1000 results. Although the number of results can exceed
> > 1000 the offset is still limited so you would need to filter out results to
> > continue past 1K. That would be slow.
>
> > If you really need to return more than 1000 results and access them in
> > forward and backward directions you could sort them by __key__ or some other
> > unique combination and use that as you own "cursor". You could define both
> > ascending and descending indexes on your chosen property(s) to let you
> > iterate in both directions.
>
> > On 3 Apr 2010, at 19:32, Arny wrote:
>
> > But isn't it getting slower and slower on higher pages, since it
> >> fetches ALL data (according to docs) and discards the offset value?
> >> So Range(0,10) is faster than Range(10000,10) ?
>
> >> Anyone did some performance tests?
> >> Regards
>
> >> On Apr 3, 7:49 am, John Patterson <jdpatter...@gmail.com> wrote:
>
> >>> Probably you should set an offset and limit instead for your paging.
> >>> I believe cursors are really intended for processing a lot of data off-
> >>> line. Remember there is no longer a 1000 result limit on fetches.
>
> >>> On 3 Apr 2010, at 02:57, Arny wrote:
>
> >>> Hi,
>
> >>> Is there a way to get a cursor to page back to a previous page?
> >>>> If not, whats the point of paging forward only? I'm not building an
> >>>> ajax page.
>
> >>> Regards
>
> >>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "Google App Engine for Java" group.
> >>>> To post to this group, send email to
> >>>> google-appengine-java@googlegroups.com
> >>>> .
> >>>> To unsubscribe from this group, send email to
> >>>> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com>
> >>>> .
> >>>> For more options, visit this group athttp://
> >>>> groups.google.com/group/google-appengine-java?hl=en
> >>>> .
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine for Java" group.
> >> To post to this group, send email to
> >> google-appengine-j...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com>
> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine-java?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine for Java" group.
> > To post to this group, send email to
> > google-appengine-j...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine-java?hl=en.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App
> Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
