I'd want to do this so that I could include parts of the cursor (such as the
offset) into a URL without including other parts (eg. the model kind and
filters). I could then reconstruct the cursor on the server side based on
what was passed into the URL.
For example, if I was searching for blog comments that contained the word
"the" (with the default order being the creation time, descending), the URL
might look like this:
myblog.com/comments/?q=the
With model:
class Comment(db.Model):
....
created_at = db.DateTimeProperty(auto_now_add=True)
words = db.StringListProperty() # A list of all the words in a comment
(forget about exploding indexes for now)
...
The query object for this URL might look something like:
....
q = Comment.all().filter('words',self.request.get('q')).order('-created_at')
....
To get to the 1001st comment, it'd be good if the URL looked something like
this:
myblog.com/comments/?q=the&skip=1000
instead of:
myblog.com/comments/?q=the&cursor=[something ugly]
so that when the request comes in, I can do this:
....
q = Comment.all().filter('words',self.request.get('q')).order('-created_at')
cursor_template = q.cursor_template()
cursor =
db.Cursor.from_template(cursor_template,offset=int(self.request.get('skip')))
....
(or something along these lines)
Does that make sense?
On 10 February 2010 01:03, Nick Johnson (Google) <nick.john...@google.com>wrote:
> Hi Nickolas,
>
> 2010/2/9 Nickolas Daskalou <n...@daskalou.com>
>
> Will we be able to construct our own cursors much the same way that we are
>> able to construct our own Datastore keys (Key.from_path())?
>>
>
> No, not practically speaking.
>
>
>>
>> Also along the same lines, will we be able to "deconstruct" a cursor to
>> get its components (offset, start_inclusive etc.), as we can now do with
>> keys (key.name(), key.id(), key.kind() etc.)?
>>
>
> While you could do this, there's no guarantees that it'll work (or continue
> to work), as you'd be digging into internal implementation details. Why do
> you want to do this?
>
> -Nick Johnson
>
>
>>
>> 2010/2/9 Nick Johnson (Google) <nick.john...@google.com>
>>
>>> 2010/2/9 Stephen <sdea...@gmail.com>
>>>
>>>
>>>> I'm asking if it's wise to store it as a query parameter embedded in a
>>>> web page.
>>>>
>>>
>>> You're right that it's unwise. Depending on how you construct your query,
>>> a user could potentially modify the cursor they send to you to return
>>> results from any query your datastore is capable of performing, which could
>>> result in you revealing information to the user that they shouldn't know.
>>> You should either store the cursor on the server-side, or encrypt it before
>>> sending it to the client.
>>>
>>> I was going to mention something about this in my post, but it slipped my
>>> mind.
>>>
>>> -Nick Johnson
>>>
>>>>
>>>>
>>>> On Feb 9, 12:26 am, "Ikai L (Google)" <ika...@google.com> wrote:
>>>> > A cursor serializes to a Base64 encoded String, so you can store it
>>>> anywhere
>>>> > you want to store strings: Memcached, Datastore, etc. You can even
>>>> pass it
>>>> > as an URL parameter to task queues.
>>>> >
>>>> > 2010/2/8 Stephen <sdea...@gmail.com>
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > > Ah right, Nick's blog does say start_key and not offset. My bad.
>>>> >
>>>> > > Maybe there will be warnings in the upcoming documentation, but my
>>>> > > first instinct was to embed the serialised cursor in the HTML as the
>>>> > > 'next' link. But that doesn't look like a good idea as Nick's
>>>> decoded
>>>> > > query shows what's embedded:
>>>> >
>>>> > > PrimaryScan {
>>>> > > start_key: "shell\000TestModel\000foo\000\232bar\000\200"
>>>> > > start_inclusive: true
>>>> > > }
>>>> > > keys_only: false
>>>> >
>>>> > > First, you may or may not want to leak this info. Second, could this
>>>> > > be altered on the client to change the query in any way that's
>>>> > > undesirable?
>>>> >
>>>> > > Once you have a cursor, where do you store it so you can use it
>>>> again?
>>>> >
>>>> > > On Feb 8, 10:17 pm, "Ikai L (Google)" <ika...@google.com> wrote:
>>>> > > > I got beaten to this answer. No, there is no traversal to get to
>>>> the
>>>> > > offset.
>>>> >
>>>> > > > BigTable has an underlying mechanism for range queries on keys.
>>>> Indexes
>>>> > > are
>>>> > > > essentially a key comprised of a concatenation of application ID,
>>>> entity
>>>> > > > type, column, value. When a filter operation is performed, the
>>>> datastore
>>>> > > > looks for a range matching this criteria, returning the set of
>>>> keys. A
>>>> > > > cursor also adds the datastore key of the entity so it is possible
>>>> to
>>>> > > > serialize where to begin the query. This is actually a bit awkward
>>>> to
>>>> > > > explain without visuals. You can watch Ryan Barrett's talk here:
>>>> >
>>>> > > >http://www.youtube.com/watch?v=tx5gdoNpcZM
>>>> >
>>>> > > > Hopefully, we'll be able to post an article at some point in the
>>>> future
>>>> > > > explaining how cursors work.
>>>> >
>>>> > > > 2010/2/8 Alkis Evlogimenos ('Αλκης Ευλογημένος) <
>>>> evlogime...@gmail.com>
>>>> >
>>>> > > > > There is no offset. The protocol buffer stores a start_key and a
>>>> > > boolean
>>>> > > > > denoting if this start key is inclusive or not. The performance
>>>> of
>>>> > > > > continuing the fetch from a cursor should be the same as the
>>>> > > performance of
>>>> > > > > the first entities you got from a query.
>>>> >
>>>> > > > > On Mon, Feb 8, 2010 at 4:33 PM, Stephen <sdea...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > > >> On Feb 8, 7:06 pm, "Ikai L (Google)" <ika...@google.com>
>>>> wrote:
>>>> > > > >> > The official docs are pending, but here's Nick Johnson to the
>>>> > > rescue:
>>>> >
>>>> > >
>>>> http://blog.notdot.net/2010/02/New-features-in-1-3-1-prerelease-Cursors
>>>> >
>>>> > > > >> What are the performance characteristics of cursors?
>>>> >
>>>> > > > >> The serialised cursor shows that it stores an offset. Does that
>>>> mean
>>>> > > > >> that if the offset is one million, one million rows will have
>>>> to be
>>>> > > > >> skipped before the next 10 are returned? This will be faster
>>>> than
>>>> > > > >> doing it in your app, but not as quick as the existing bookmark
>>>> > > > >> techniques which use the primary key index.
>>>> >
>>>> > > > >> Or is the server-side stateful, like a typical SQL
>>>> implementation of
>>>> > > > >> cursors? In which case, are there any limits to the number of
>>>> active
>>>> > > > >> cursors? Or what if a cursor is resumed some time in the
>>>> future; will
>>>> > > > >> it work at all, or work slower?
>>>> >
>>>> > > > >> --
>>>> > > > >> You received this message because you are subscribed to the
>>>> Google
>>>> > > Groups
>>>> > > > >> "Google App Engine" group.
>>>> > > > >> To post to this group, send email to
>>>> > > google-appeng...@googlegroups.com.
>>>> > > > >> To unsubscribe from this group, send email to
>>>> > > > >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>> <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>> >
>>>> > > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>> <google-appengine%252bunsubscr...@googlegroups.com<google-appengine%25252bunsubscr...@googlegroups.com>
>>>> >
>>>> >
>>>> > > > >> .
>>>> > > > >> For more options, visit this group at
>>>> > > > >>http://groups.google.com/group/google-appengine?hl=en.
>>>> >
>>>> > > > > --
>>>> >
>>>> > > > > Alkis
>>>> >
>>>> > > > > --
>>>> > > > > You received this message because you are subscribed to the
>>>> Google
>>>> > > Groups
>>>> > > > > "Google App Engine" group.
>>>> > > > > To post to this group, send email to
>>>> google-appengine@googlegroups.com
>>>> > > .
>>>> > > > > To unsubscribe from this group, send email to
>>>> > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>> <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>> >
>>>> > > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>> <google-appengine%252bunsubscr...@googlegroups.com<google-appengine%25252bunsubscr...@googlegroups.com>
>>>> >
>>>> >
>>>> > > > > .
>>>> > > > > For more options, visit this group at
>>>> > > > >http://groups.google.com/group/google-appengine?hl=en.
>>>> >
>>>> > > > --
>>>> > > > Ikai Lan
>>>> > > > Developer Programs Engineer, Google App Enginehttp://
>>>> > > googleappengine.blogspot.com|http://twitter.com/app_engine
>>>> >
>>>> > > --
>>>> > > You received this message because you are subscribed to the Google
>>>> Groups
>>>> > > "Google App Engine" group.
>>>> > > To post to this group, send email to
>>>> google-appeng...@googlegroups.com.
>>>> > > To unsubscribe from this group, send email to
>>>> > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>> <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>> >
>>>> > > .
>>>> > > For more options, visit this group at
>>>> > >http://groups.google.com/group/google-appengine?hl=en.
>>>> >
>>>> > --
>>>> > Ikai Lan
>>>> > Developer Programs Engineer, Google App Enginehttp://
>>>> googleappengine.blogspot.com|http://twitter.com/app_engine
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Google App Engine" group.
>>>> To post to this group, send email to google-appeng...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>> .
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-appengine?hl=en.
>>>>
>>>>
>>>
>>>
>>> --
>>> Nick Johnson, Developer Programs Engineer, App Engine
>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
>>> Number: 368047
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appeng...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appeng...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>
>
> --
> Nick Johnson, Developer Programs Engineer, App Engine
> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
> 368047
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
