[google-appengine] Re: Expected Database Performance with Millions of Rows?

2009-03-10 Thread peterk
As far as I know: query = A.all().filter('intProperty', val).filter('date >=', date) This is one read. Hence the relatively fast performance for getting itemAs. But this: itemBs = [itemA.typeB for itemA in itemAs] ..is n reads, where n is the number of itemAs you have. In your case, an extra

[google-appengine] Re: Expected Database Performance with Millions of Rows?

2009-03-10 Thread peterk
Doh, I just thought of this as one potential simple way to improve your itemB dereferencing: http://code.google.com/appengine/docs/python/datastore/modelclass.html#Model_get Create a list of the keys to your itemBs without dereferencing them, then pass the list to itemB.get(). I think this perfo

[google-appengine] Re: Expected Database Performance with Millions of Rows?

2009-03-10 Thread lenza
Thanks for the response peter. It is my understanding that a ReferenceProperty IS actually just a Key (http://code.google.com/ appengine/docs/python/datastore/ typesandpropertyclasses.html#ReferenceProperty). If it isn't, how do I get the key of itemB without dereferencing itemB? There is no Ke

[google-appengine] Re: Expected Database Performance with Millions of Rows?

2009-03-10 Thread peterk
Yeah, sorry, you're right..with regard to a batch get, I'm not sure there's any way to just get the itemB keys without dereferencing the whole object.. A thought crossed my mind that you could try storing the key id or name as strings in itemA and then construct a list of keys by casting the id s

[google-appengine] Re: Expected Database Performance with Millions of Rows?

2009-03-10 Thread ryan
On Mar 10, 11:10 am, peterk wrote: > A survey of read cost as entity numbers increase would indeed be > interesting however..I've been making the assumption that read speed > is roughly constant regardless of the number of entities > underneath...but that is just an assumption. Maybe a google pe

[google-appengine] Re: Expected Database Performance with Millions of Rows?

2009-03-10 Thread peterk
This is useful stuff Ryan, thanks very much for weighing in here. Sorry if some redundant points are being raised here.. my lingering question in light of your post is regarding the variation in performance between itemAs' construction and itemBs' if both boil down to the same read behaviour on t

[google-appengine] Re: Expected Database Performance with Millions of Rows?

2009-03-10 Thread ryan
On Mar 10, 4:38 pm, peterk wrote: > Sorry if some redundant points are being raised here.. my lingering > question in light of your post is regarding the variation in > performance between itemAs' construction and itemBs' if both boil down > to the same read behaviour on the datastore. To walk th

[google-appengine] Re: Expected Database Performance with Millions of Rows?

2009-03-11 Thread peterk
Ahh, thanks Ryan..that makes sense. So for lenza...you need to try and squeeze your itemBs construction into one RPC call..so..something like this might do the trick: def myQuery(): query = A.all().filter('intProperty', val).filter('date >=', date) itemAs = [itemA for itemA in query] it

[google-appengine] Re: Expected Database Performance with Millions of Rows?

2009-03-19 Thread djidjadji
You can eliminate the batch get when iterating the query, by fetching all the itemAs in one call. If you will process ALL the objects from a query, get them ALL, and then iterate. def myQuery(): query = A.all().filter('intProperty', val).filter('date >=', date).fetch(1000) itemBkeys = [A.type