Tasks are now allowed up to 10 minutes to complete their work. You still have to deal with the 30 seconds max per datastore api call but that shouldn't be a problem.
Use tasks to handle the heavy lifting and like Wim suggested use a cursor to maintain your position in the result set between task invocations. Between the 2 as well as using parallel batch gets you should easily be able to handle million + records in your result set. Once the tasks have completed it should write some marker entity out to the datastore which can be used by a forward facing request to determine if the result set is available or not. Jeff On Tue, Feb 1, 2011 at 1:24 PM, Prashanth <prashanth.b...@gmail.com> wrote: > Thanks Jeff and Wim for the responses. > > I missed out giving the number of records in the datastore. > The datastore has ~1 million records and ever increasing. > The solution which we have now is, get all the records greater than > the input date [most restrictive] in a query and then loop thru the > records for records greater than x_items. > This solution hit performance badly. Hence looking for an alternative > solution. > > Any suggestions please. > Thanks for reading this. > > On Feb 1, 11:05 pm, Jeff Schwartz <jefftschwa...@gmail.com> wrote: > > If you intend to make multiple calls passing the cursor back and forth > > between client and server then that would work but if you intend to do it > > all in one request then I think the poster of the original question would > > need to define how much data they were talking about which brings me back > to > > what I suggested earlier which is: if you are talking about merely > trivial > > amounts of data then it would work but if he is talking about anything > more > > than merely trivial amounts of data then it wont work and another > approach > > would be needed. > > > > Jeff > > > > I'd prefer a neater package: begin a task that kicks of multiple queries > to > > retrieve all the results - the tasks can use a cursor. When the tasks > have > > completed their work > > > > On Tue, Feb 1, 2011 at 12:28 PM, Wim den Ouden <wdenou...@gmail.com> > wrote: > > > > > > > > > > > > > > > > > > > > > no, no, first the query on is item_names and more than x_items, > > > then a loop on the result from this query (for items in resultquery:) > > > if some_input_date > date: append to list > > > when ready show list > > > > > you can play with fetch(..) to get optimum speed and cpu usage. > > > ofcourse not a top solution, but it works till?? > > > gr > > > wim > > > > > 2011/2/1 Jeff Schwartz <jefftschwa...@gmail.com>: > > > > This would require 2 inequality filters so no luck. > > > > > > On Tue, Feb 1, 2011 at 7:17 AM, Prashanth <prashanth.b...@gmail.com> > > > wrote: > > > > > >> Hi, > > > > > >> Am trying to query my Google App Engine datastore [Python], which > has > > > >> a item_name, manufacturing_date and number_of_items_shipped. > > > >> The scenario: > > > >> Get all the item_names which has been shipped more than x_items > [user > > > >> input] and manufactured after some_input_date [user input]. > > > >> Basically, kind of inventory check. > > > > > >> But due to restrictions on queries in GAE, am not able to do this. > > > > >http://code.google.com/appengine/docs/python/datastore/queries.html#R. > .. > > > > > >> I tried searching on the internet for this issue. But, no luck till > > > >> now. Did you come across this issue, if so, were you able to resolve > > > >> this? Please let me know. > > > > > >> Also in Google I/O 2010, "Next Gen Queries" [ > http://www.youtube.com/ > > > >> watch?v=ofhEyDBpngM#t=3m17s], Alfred Fuller mentioned that they are > > > >> going to remove this restriction soon. Its been more than 8 months, > > > >> but this restriction is in place even now. Makes life very > difficult. > > > > > >> Appreciate if anyone can post an answer if they were able to > > > >> circumvent this restriciton. > > > >> Thanks a lot. > > > > > >> Regards, > > > >> Prashanth. > > > > > >> -- > > > >> You received this message because you are subscribed to the Google > > > Groups > > > >> "Google App Engine" group. > > > >> To post to this group, send email to > google-appengine@googlegroups.com. > > > >> To unsubscribe from this group, send email to > > > >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > > . > > > >> For more options, visit this group at > > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > > > > Jeff Schwartz > > > > > > -- > > > > You received this message because you are subscribed to the Google > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to > google-appengine@googlegroups.com. > > > > To unsubscribe from this group, send email to > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > gr > > > Wim den Ouden > > > Custom applications,https://e-comm.appspot.com/ > > > Free open source E-commerce/E-bookkeeping/E-business framework (web) > > > apps,http://code.google.com/p/relat/ > > > Gae developer tips,http://code.google.com/p/relat/wiki/gaetips > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to google-appengine@googlegroups.com > . > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > *Jeff Schwartz* > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- *Jeff Schwartz* -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
