Interesting - in retrospect, of course it isn't realloc overhead. There would be at most 9 reallocs, and there's no way that could take hundreds of milliseconds.
I'm not sure that this optimization of Python will be as effective in Javaland. The pattern of fetching is totally different in Python - they fetch N entities at a time into a list, so it makes sense to turn that list into an async iterator behind the scenes. Java already has iterators, so people probably only use Lists when they want to iterate the dataset multiple times. It'll be a less common case. I have mixed feelings about converting the Objectify Query.list() method to return a lazy List instead of an eagerly-converted List of POJOs. It would mean the List is no longer serializable. I've always thought that the Query methods that return Iterator are there for performance; the Query methods that return List are there for convenience. In this case, seems like convenience (ie serializability) should trump. But I'm not certain of that. Thanks for clearing this up! Jeff On Thu, Jun 9, 2011 at 10:29 AM, Alfred Fuller <arfuller+appeng...@google.com> wrote: > If you are going to just iterate through a list with out doing any work, > fetching everything up front is always going to be faster. However, we > expect that you are going to be doing something with the entities you fetch. > The lazy list tries to hid the cost of fetching the entities by doing it > asynchronously while you are 'processing' the previous batch of entities. In > my experiments (in Python) where work was actually being done (namely > thread.sleep(X)) it gave a 10-15% speed up (depending on how much work you > are doing, as the fetch time is a constant value). > I would not have expected the difference to be this significant in Java > though. We never know the # of results ahead of time so the difference > couldn't be related to growing the List organically (as it always grows > organically). > One thing that should be noted, a wrapper that proactively converts entities > (instead of doing the conversion on demand) will negate any benefit from > async prefetching. Also a realistic benchmark should tak into account that > some amount of work will be done on the entities fetched. > > On Thu, Jun 9, 2011 at 9:51 AM, Alfred Fuller > <arfuller+appeng...@google.com> wrote: >> >> It does uses a lazy list to do asynchronous prefetching: >> >> http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/LazyList.java >> >> On Tue, Jun 7, 2011 at 3:19 AM, Anders <blabl...@gmail.com> wrote: >>> >>> I doubt that the difference can be that large. The performance test code >>> uses the low-level PreparedQuery#asList call. The question is if the list >>> (List<Entity>) contains entities loaded with data or if the list returned >>> has a lazy loading implementation so that the actual data from the the >>> datastore only gets loaded when entity properties are accessed. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-appengine/-/WVhBRXBqMWFMZ3dK. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
