Doh, I just thought of this as one potential simple way to improve your itemB dereferencing:
http://code.google.com/appengine/docs/python/datastore/modelclass.html#Model_get Create a list of the keys to your itemBs without dereferencing them, then pass the list to itemB.get(). I think this performs one single batch get..so it should be faster then 36 seperate reads. Try that out and let me know if it improves performance.. :) On Mar 10, 12:03 pm, peterk <peter.ke...@gmail.com> wrote: > As far as I know: > > query = A.all().filter('intProperty', val).filter('date >=', date) > > This is one read. Hence the relatively fast performance for getting > itemAs. > > But this: > > itemBs = [itemA.typeB for itemA in itemAs] > > ..is n reads, where n is the number of itemAs you have. In your case, > an extra 36 reads. > > That lines up with the figures you're providing..0.4s to get all > itemAs, then 2.2 for the (36) itemB reads. > > There may be more optimal ways others can share for dereferencing your > itemBs to reduce the number of reads required, but in terms of > scalibility..if the number of itemAs returned will remain the same, > then your execution time for this computation should remain the same > regardless of the size of the database underneath. Datastore reads > are, I think, supposed to be fairly constant regardless of the number > of entities..so if you're always doing 36+1 reads then the cost should > always be roughly the same regardless of how many itemAs and itemBs > are in the datastore. I could be wrong on that..but that's my > expectation. > > In terms of optimising your itemB dereferencing..do you need all those > itemBs at the point you're adding them to your list at the moment? > What is itemB made up of? > > On Mar 10, 1:23 am, lenza <le...@lenza.org> wrote: > > > App Engine. Here is the situation: > > > I have a database with about about 50k rows of type A, and maybe 500k > > rows of type B. > > > Type A contains: > > - ReferenceProperty (to type B) > > - DateTimeProperty > > - IntegerPropery > > - 3 x StringProperty (at most 5 bytes each) > > > I am performing the following query: > > > def myQuery(): > > query = A.all().filter('intProperty', val).filter('date >=', date) > > itemAs = [itemA for itemA in query] > > itemBs = [itemA.typeB for itemA in itemAs] > > return itemBs > > > Even under no load, when retuning a modest number of elements, this > > function takes so long! For example, I'm looking at a log entry where > > for 36 items the database took 0.4s to create the list of itemAs and > > 2.2s to create the list of itemB's. Does this seem excessive? What > > can I do to speed this up? I was planing on these databases to grow > > 10x in size (although the queries will return the same number of > > items) so this is concerning. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
