Hey David,
I'm not sure where to point you for a simple example, but I'm sure
there are some out there. You just need to build your query, fetch
some results, grab the cursor, pass it into the same query, then fetch
more results, then repeat. Make sure you check the number of results
returned so that you know if you need to continue fetching more
results or not.
Here is one example of using cursors to continue processing results
(just the get_work method):
https://bitbucket.org/thebobert/slagg/src/a41cc67326ce/slagg/__init__.py#cl-324
The items you should pay special attention to are 'work_query' and
'continue_cursor.'
Robert
On Mon, May 30, 2011 at 17:27, David Stone <minpeng2...@gmail.com> wrote:
> Hello Robert,
>
> Thanks for your idea! I try to find the cursor coding example from the
> internet, but what i have found is really little and GAE site does not
> say much about it.
>
> Do you know where can i find coding example in Python for cursor of
> GAE?
>
> Thanks!
>
> David
>
> On May 29, 5:31 am, Robert Kluin <robert.kl...@gmail.com> wrote:
>> HiDavid,
>> There is no way you're going to return a million entities in a
>> single request. You should look into cursors. Cursors allow you to
>> fetch results from a query, then resume fetching more results where
>> you left off. Cursors are nice because you can use them within the
>> same request to efficiently fetch a larger number of entities in
>> smaller batches, or you can use them across requests (by passing them
>> between requests). You could also potentially use the new files API
>> to dump your entities to the blobstore (maybe as CSV) then download
>> the blob.
>>
>> http://code.google.com/appengine/docs/python/datastore/queryclass.htm...
>> http://code.google.com/appengine/docs/python/blobstore/overview.html#...
>>
>> Within a request, you can use a cursor something like this:
>>
>> query = YourKind.all().order('some_prop')
>> cursor = None
>> while True:
>> if cursor:
>> query.with_cursor(cursor)
>> entities = query.fetch(500)
>> # do stuff with entities
>> if len(entities) < 500:
>> break
>> cursor = query.cursor()
>>
>> Across requests you'd just need to pass the cursor to the next
>> request (perhaps in a form field, for example).
>>
>> Robert
>>
>>
>>
>> On Sat, May 28, 2011 at 04:06,DavidStone<minpeng2...@gmail.com> wrote:
>> > Hello,
>>
>> > Suppose i have 4,5 or even more kinds (tables) stored in GAE data
>> > store and each table has quite a lot of data, let's say each table has
>> > 5 million rows (entities) or even more. Now i am facing a big problem
>> > with generating and returning a big size result for a query. The
>> > reason is that a request handler will time out if takes more than 30
>> > second to generate the query result and i think there is a probably
>> > 10MB limitation for a http response to return the query result.
>>
>> > So i am wondering is there a way to send the query result as small
>> > portion (data streaming) each time back to me.
>>
>> > I have read from this link:
>> >http://code.google.com/appengine/docs/python/runtime.html
>>
>> > The link mentions "App Engine collects all of the data the request
>> > handler script writes to the standard output stream, then waits for
>> > the script to exit. When the script exits, all of the output data is
>> > sent to the user.
>>
>> > App Engine does not support sending data to the user's browser before
>> > exiting the handler. Some web servers use this technique to "stream"
>> > data to the user's browser over a period of time in response to a
>> > single request. App Engine does not support this streaming technique."
>>
>> > As far as i understand their information is that there is no way to
>> > send the query result as many small portions by time and each time
>> > sends out for example 5000 rows.
>>
>> > At the same time, this link (http://arstechnica.com/web/news/2010/12/
>> > app-engine-gets-streaming-api-and-longer-background-tasks.ars)
>> > mentions there is a streaming API from GAE.
>>
>> > Now i am confused by these messages. Any idea or suggestion about this
>> > issue would be well appreciated.
>>
>> > Another question is that once i have a query generating a lot of
>> > result might cause time out to the request handler due to the 30
>> > seconds time limitation. How could i overcome this problem?
>>
>> > Thank you very much for all the answers from you.
>>
>> >David
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group
>> > athttp://groups.google.com/group/google-appengine?hl=en.- Hide quoted text
>> > -
>>
>> - Show quoted text -
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
