On Thu, Jun 9, 2011 at 2:56 PM, Jeff Schnitzer <j...@infohazard.org> wrote:
> Interesting - in retrospect, of course it isn't realloc overhead. > There would be at most 9 reallocs, and there's no way that could take > hundreds of milliseconds. > > I'm not sure that this optimization of Python will be as effective in > Javaland. The pattern of fetching is totally different in Python - > they fetch N entities at a time into a list, so it makes sense to turn > that list into an async iterator behind the scenes. Java already has > iterators, so people probably only use Lists when they want to iterate > the dataset multiple times. It'll be a less common case. > Funny because we actually don't have this feature in python (only iterators async prefetch). Ya, I would hope coders would use asIterable() when doing a single for loop. The real win for asList() async prefetch is in these situations: List l = q.asList(); // returns immediately .... other work/rpcs... ... use l ... and // Launches and runs both queries in parallel List l1 = q1.asList(); // returns immediately List l2 = d2.asList(); // returns immediately ... use l1 and l2 ... > I have mixed feelings about converting the Objectify Query.list() > method to return a lazy List instead of an eagerly-converted List of > POJOs. It would mean the List is no longer serializable. I've always > thought that the Query methods that return Iterator are there for > performance; the Query methods that return List are there for > convenience. In this case, seems like convenience (ie > serializability) should trump. But I'm not certain of that. > The lazy list implementation in the low level is both lazy and serializable. You should be able to do something similar in Objectify. > > Thanks for clearing this up! > > Jeff > > On Thu, Jun 9, 2011 at 10:29 AM, Alfred Fuller > <arfuller+appeng...@google.com> wrote: > > If you are going to just iterate through a list with out doing any work, > > fetching everything up front is always going to be faster. However, we > > expect that you are going to be doing something with the entities you > fetch. > > The lazy list tries to hid the cost of fetching the entities by doing it > > asynchronously while you are 'processing' the previous batch of entities. > In > > my experiments (in Python) where work was actually being done (namely > > thread.sleep(X)) it gave a 10-15% speed up (depending on how much work > you > > are doing, as the fetch time is a constant value). > > I would not have expected the difference to be this significant in Java > > though. We never know the # of results ahead of time so the difference > > couldn't be related to growing the List organically (as it always grows > > organically). > > One thing that should be noted, a wrapper that proactively converts > entities > > (instead of doing the conversion on demand) will negate any benefit from > > async prefetching. Also a realistic benchmark should tak into account > that > > some amount of work will be done on the entities fetched. > > > > On Thu, Jun 9, 2011 at 9:51 AM, Alfred Fuller > > <arfuller+appeng...@google.com> wrote: > >> > >> It does uses a lazy list to do asynchronous prefetching: > >> > >> > http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/LazyList.java > >> > >> On Tue, Jun 7, 2011 at 3:19 AM, Anders <blabl...@gmail.com> wrote: > >>> > >>> I doubt that the difference can be that large. The performance test > code > >>> uses the low-level PreparedQuery#asList call. The question is if the > list > >>> (List<Entity>) contains entities loaded with data or if the list > returned > >>> has a lazy loading implementation so that the actual data from the the > >>> datastore only gets loaded when entity properties are accessed. > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups > >>> "Google App Engine" group. > >>> To view this discussion on the web visit > >>> https://groups.google.com/d/msg/google-appengine/-/WVhBRXBqMWFMZ3dK. > >>> To post to this group, send email to google-appengine@googlegroups.com > . > >>> To unsubscribe from this group, send email to > >>> google-appengine+unsubscr...@googlegroups.com. > >>> For more options, visit this group at > >>> http://groups.google.com/group/google-appengine?hl=en. > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
