btw the datastore in wich i was ask to do a ask and for which i developed the tool, i was told it holded million records initially but it had 45 million after all, so the meta refresh trick didnt hold here.
my colleague ended up not using the tool :/ No dia 6 de Abril de 2010 23:41, José Moreira <matrixowns...@gmail.com> escreveu: > Hi > > i posed the question to optimize this tool i was working on > http://github.com/matrixownsyou/MultiTaskBob > > it "works" by serially segmenting records in buckets of X records, by the > keys: > > segemnt1, startkey1, endkey2 > segemnt2, startkey3, endkey4 > <....> > > this should be fast since it only selects keys and it uses the > memcached cursor for the result set, but getting the last key of a > fetch() is taking some time : > > http://github.com/matrixownsyou/MultiTaskBob/blob/master/MultiTaskBob/base.py#L188 > > the idea is that after the records are segmented, each segment (and > its contained records) can be processed concurrently, for example in > an htttp threading tool. > > the reason i didnt use something like httpmr was because the data > model (wich needed to be updated due to new properties ) didnt had an > unique property other than the key or anything else i tought > manageable to map upon. > > > > > > 2010/4/6 Wesley C (Google) <wesc+...@google.com>: >> agreeing with ryan here that you'll need to do a pair of questions, >> regardless of whether you do it via a Query or GqlQuery. >> >> however, a better question would be: what are you trying to do, and >> why do you need the first and last keys? non-relational datastores are >> generally not geared up for doing counting (thus fetching the "last >> key"). perhaps there is another way to accomplish your task without >> doing exactly what you've described. >> >> for example, an alternative to querying is to create another type of >> entity in the datastore that stores the first key and the (current) >> last key, and update that data structure as you continue to add more >> applications entities into your datastore. this way you'll have a much >> shorter query to perform (only one object that contains exactly what >> you need). >> >> just curious... >> -- wesley >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> "Core Python Programming", Prentice Hall, (c)2007,2001 >> "Python Fundamentals", Prentice Hall, (c)2009 >> http://corepython.com >> >> wesley.j.chun :: wesc+api-at-google-dot-com >> developer relations :: google app engine >> @app_engine :: googleappengine.blogspot.com >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > > > -- > josemore...@irc.freenode.net > http://beta.icodedhere.com > http://pt.linkedin.com/in/josemoreira > http://djangopeople.net/josemoreira > http://www.crunchbase.com/user/josemoreira > -- josemore...@irc.freenode.net http://beta.icodedhere.com http://pt.linkedin.com/in/josemoreira http://djangopeople.net/josemoreira http://www.crunchbase.com/user/josemoreira -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
