Ok - so what you're saying is that the extra call to list.size() before iterating through the list makes list iteration faster? Oddly enough, this does seem to make a difference.
This looks like some sort of performance bug in the Low-Level API. It's clearly not related to Slim3... except in as much as Slim3 inadvertently works around this bug. It's also very peculiar to working with List results - you obviously can't call size() when you're iterating through an unbounded result set. The benchmark is bogus... which is to say, it does not show what you think it shows. Jeff On Thu, Jun 9, 2011 at 12:09 AM, Yasuo Higa <higaya...@gmail.com> wrote: > It's *NO* bogus benchmark because the sample iterates the results. > > http://code.google.com/p/slim3/source/browse/trunk/slim3demo/src/slim3/demo/controller/performance/GetLLController.java > > Yasuo Higa > >> >> The code linked from http://slim3demo.appspot.com/performance/ has >> just gone back to calling list.size() without iterating the results, >> which once again goes back to a bogus benchmark. >> >> Jeff >> >> On Wed, Jun 8, 2011 at 9:30 PM, Yasuo Higa <higaya...@gmail.com> wrote: >>> Slim3 uses LL API. >>> >>> To resolve a strange issue that slim3 is faster than LL, I tried the >>> following samples: >>> >>> One: >>> AsyncDatastoreService ds = >>> DatastoreServiceFactory.getAsyncDatastoreService(); >>> Query q = new Query("Bar"); >>> PreparedQuery pq = ds.prepare(q); >>> List<Entity> list = >>> pq.asList(FetchOptions.Builder.withDefaults().limit( >>> Integer.MAX_VALUE)); >>> for (Entity e : service.getBarListUsingLL()) { >>> e.getKey(); >>> e.getProperty("sortValue"); >>> } >>> >>> Two: >>> AsyncDatastoreService ds = >>> DatastoreServiceFactory.getAsyncDatastoreService(); >>> Query q = new Query("Bar"); >>> PreparedQuery pq = ds.prepare(q); >>> List<Entity> list = >>> pq.asList(FetchOptions.Builder.withDefaults().limit( >>> Integer.MAX_VALUE)); >>> >>> // VERY IMPORTANT >>> list.size(); >>> >>> for (Entity e : service.getBarListUsingLL()) { >>> e.getKey(); >>> e.getProperty("sortValue"); >>> } >>> >>> The second one is much faster than the first one. >>> I fixed the samples to call list.size(). >>> http://slim3demo.appspot.com/performance/ >>> >>> As a result, LL is as fast as slim3 (^^; >>> >>> Yasuo Higa >>> >>> >>> >>> On Thu, Jun 9, 2011 at 10:17 AM, Jeff Schnitzer <j...@infohazard.org> wrote: >>>> Thank you for fixing the benchmark. >>>> >>>> I am very curious. According to this new benchmark - it's hard to >>>> tell without pushing the buttons a lot of times, but there seems to be >>>> a trend - Slim3 is somewhat faster than the Low Level API. >>>> >>>> Doesn't Slim3 use the Low Level API underneath? How can it possibly be >>>> faster? >>>> >>>> Jeff >>>> >>>> On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa <higaya...@gmail.com> wrote: >>>>> What I want to provide is a fair and casual benchmark. >>>>> >>>>> As jeff advised, I modified samples as follows: >>>>> for (Entity e : service.getBarListUsingLL()) { >>>>> e.getKey(); >>>>> e.getProperty("sortValue"); >>>>> } >>>>> >>>>> for (Bar bar : service.getBarListUsingSlim3()) { >>>>> bar.getKey(); >>>>> bar.getSortValue(); >>>>> } >>>>> >>>>> for (BarObjectify bar : service.getBarListUsingObjectify()) { >>>>> bar.getKey(); >>>>> bar.getSortValue(); >>>>> } >>>>> >>>>> for (BarJDO bar : service.getBarListUsingJDO()) { >>>>> bar.getKey(); >>>>> bar.getSortValue(); >>>>> } >>>>> >>>>> LL API is much slower than before. >>>>> http://slim3demo.appspot.com/performance/ >>>>> >>>>> Yasuo Higa >>>>> >>>>> >>>>> On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer <j...@infohazard.org> >>>>> wrote: >>>>>> Slim3 may be a nice piece of software, but it has not been >>>>>> demonstrated to be faster than anything (including JDO). It might or >>>>>> might not be faster - I don't know - but based on the sloppy >>>>>> benchmarking, I'm pretty certain that the people making this claim >>>>>> don't know either. >>>>>> >>>>>> There's another ill-conceived performance claim on the Slim3 website: >>>>>> "You may worry about the overhead of global transactions. Don't worry. >>>>>> It is not very expensive." There are three problems with their little >>>>>> performance test: >>>>>> >>>>>> 1) It only measures wall-clock time, not cost. >>>>>> 2) It does not measure what happens under contention. >>>>>> 3) The numbers are obviously wrong - they don't even pass a smoke test. >>>>>> >>>>>> Look at these numbers (from the Slim3 home page): >>>>>> >>>>>> Entity Groups Local Transaction(millis) Global >>>>>> Transaction(millis) >>>>>> 1 67 70 >>>>>> 2 450 415 >>>>>> 3 213 539 >>>>>> 4 1498 981 >>>>>> 5 447 781 >>>>>> >>>>>> Just like the 1ms low-level API query performance in the benchmark >>>>>> that spawned this thread, even a casual observer should be able to >>>>>> recognize the obvious flaw - the numbers don't show any expected >>>>>> relationship between # of entity groups or the use of global >>>>>> transactions. Interpreted literally, you would assume that local >>>>>> transactions are much faster for 5 entity groups, but global >>>>>> transactions are much faster for 4 entity groups. >>>>>> >>>>>> It's pretty obvious that the benchmark author just ran one pass and >>>>>> took the numbers as-is. If you actually run multiple passes, you'll >>>>>> find that there is enormous variance in the timing. The only way you >>>>>> can realistically measure results like this on appengine is to execute >>>>>> the test 100 times and take a median. >>>>>> >>>>>> FWIW, I deliberately haven't made any performance claims for Objectify >>>>>> because I haven't put the necessary amount of due diligence into >>>>>> creating a proper set of benchmarks. It is not easy to measure >>>>>> performance, especially in a dynamic environment like appengine. I >>>>>> only hope that the Slim3 authors have put more care and thought into >>>>>> crafting their library than they have their benchmarks. >>>>>> >>>>>> Jeff >>>>>> >>>>>> >>>>>> On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber <gal.dol...@gmail.com> wrote: >>>>>>> Slim3 is not only fast, the api is completely awesome. It has been my >>>>>>> choice >>>>>>> for a year now for all gae projects. >>>>>>> It includes "name safety" and and amazing querying utils. >>>>>>> Very recommendable! >>>>>>> >>>>>>> On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer <j...@infohazard.org> >>>>>>> wrote: >>>>>>>> >>>>>>>> You are wrong. >>>>>>>> >>>>>>>> Try adding getProperty() calls to your LL performance test, and the >>>>>>>> speed advantage of the LL API goes away. I don't know what to say >>>>>>>> about Slim3, but here's my test case: >>>>>>>> >>>>>>>> >>>>>>>> http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test >>>>>>>> >>>>>>>> I created 10,000 entities in the datastore that have the same format >>>>>>>> as your test case - a single string property. Here's what happens >>>>>>>> (try it - and remember to reload the urls several times to get a >>>>>>>> realistic median value): >>>>>>>> >>>>>>>> ###### Low Level API with just .size() >>>>>>>> >>>>>>>> http://voodoodyne.appspot.com/fetchLLSize >>>>>>>> >>>>>>>> The code: >>>>>>>> >>>>>>>> List<Entity> things = >>>>>>>> DatastoreServiceFactory.getDatastoreService() >>>>>>>> .prepare(new >>>>>>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >>>>>>>> .asList(FetchOptions.Builder.withDefaults()); >>>>>>>> things.size(); >>>>>>>> >>>>>>>> Note that results are almost always under 2000ms. Wild guess I'd say >>>>>>>> the median elapsed is ~1900, just like your example. >>>>>>>> >>>>>>>> ###### Low Level API with actual fetch of the data >>>>>>>> >>>>>>>> http://voodoodyne.appspot.com/fetchLL >>>>>>>> >>>>>>>> The code: >>>>>>>> >>>>>>>> List<Entity> things = >>>>>>>> DatastoreServiceFactory.getDatastoreService() >>>>>>>> .prepare(new >>>>>>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >>>>>>>> .asList(FetchOptions.Builder.withDefaults()); >>>>>>>> for (Entity ent: things) >>>>>>>> { >>>>>>>> ent.getKey(); >>>>>>>> ent.getProperty("value"); >>>>>>>> } >>>>>>>> >>>>>>>> Note that the duration is now considerably longer. Eyeballing the >>>>>>>> median elapsed time, I'd say somewhere around 3000ms. >>>>>>>> >>>>>>>> ###### Objectify fetching from datastore >>>>>>>> >>>>>>>> http://voodoodyne.appspot.com/fetch >>>>>>>> >>>>>>>> Objectify ofy = ObjectifyService.begin(); >>>>>>>> List<Thing> things = ofy.query(Thing.class).list(); >>>>>>>> for (Thing thing: things) >>>>>>>> { >>>>>>>> thing.getId(); >>>>>>>> thing.getValue(); >>>>>>>> } >>>>>>>> >>>>>>>> Note that the timing is pretty much the same as the LL API when it >>>>>>>> includes actual fetches of the entity values. It is, no doubt, just a >>>>>>>> little higher. >>>>>>>> >>>>>>>> ###### A pure measurement of Objectify's overhead >>>>>>>> >>>>>>>> http://voodoodyne.appspot.com/fakeFetch >>>>>>>> >>>>>>>> This causes Objectify to translate 10,000 statically-created Entity >>>>>>>> objects to POJOs. You can see the code here: >>>>>>>> >>>>>>>> >>>>>>>> http://code.google.com/p/scratchmonkey/source/browse/appengine/performance-test/src/test/FakeFetchServlet.java >>>>>>>> >>>>>>>> You'll notice (after you hit the URL a couple times to warm up the >>>>>>>> JIT) that elapsed time converges to somewhere around 120ms. >>>>>>>> >>>>>>>> ----------- >>>>>>>> >>>>>>>> Conclusion: >>>>>>>> >>>>>>>> The numbers in the original benchmark are a result of improper >>>>>>>> measurements. The actual wall-clock overhead for Objectify in this >>>>>>>> test is ~4% (120ms out of 3000ms). >>>>>>>> >>>>>>>> Further speculation on my part, but probably correct: The overhead of >>>>>>>> reflection is unlikely to be a significant part of that 4%. >>>>>>>> >>>>>>>> Sloppy work. >>>>>>>> >>>>>>>> Jeff >>>>>>>> >>>>>>>> On Wed, Jun 8, 2011 at 7:55 AM, Yasuo Higa <higaya...@gmail.com> wrote: >>>>>>>> > It is not bogus. >>>>>>>> > LazyList#size() fetches all data as follows: >>>>>>>> > public int size() { >>>>>>>> > resolveAllData(); >>>>>>>> > return results.size(); >>>>>>>> > } >>>>>>>> > >>>>>>>> > Yasuo Higa >>>>>>>> > >>>>>>>> > On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson >>>>>>>> > <dennisbpeter...@gmail.com> wrote: >>>>>>>> >> It's not my benchmark, it's Slim3's :) ...but you're right, it's >>>>>>>> >> bogus. >>>>>>>> >> I >>>>>>>> >> asked on the main appengine group too, and it turns out the >>>>>>>> >> low-level >>>>>>>> >> benchmark is doing lazy loading. With that fixed, their numbers come >>>>>>>> >> out >>>>>>>> >> like yours. >>>>>>>> >> I found this one too, which also gets results like yours: >>>>>>>> >> http://gaejava.appspot.com/ >>>>>>>> >> >>>>>>>> >> On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur >>>>>>>> >> <erwin.str...@gmail.com> >>>>>>>> >> wrote: >>>>>>>> >>> >>>>>>>> >>> Indeed Dennis's measurements are very suspicious. First you should >>>>>>>> >>> do >>>>>>>> >>> a couple of warming ups on each of the implementations to prevent >>>>>>>> >>> pollution like the JDO classpath scan for enhanced classes (which >>>>>>>> >>> is >>>>>>>> >>> one of the reasons for the high initial run). Then do a couple of >>>>>>>> >>> run >>>>>>>> >>> to determine a range of measurements to spot outlyers. your >>>>>>>> >>> low-level >>>>>>>> >>> API 2millis is definately one. >>>>>>>> >>> >>>>>>>> >>> When I did the measurements I got the following results >>>>>>>> >>> low-level: 1150-1550 >>>>>>>> >>> Slim3: 1150-1600 >>>>>>>> >>> Objectify: 1950-2400 >>>>>>>> >>> JDO: 2100-2700 >>>>>>>> >>> >>>>>>>> >>> These measurements confirm that GAE designed implementations are >>>>>>>> >>> faster then the GAE implementation of a generic data access layer >>>>>>>> >>> (JDO), but not so extrem as initially posted. >>>>>>>> >>> >>>>>>>> >>> The initial response using JDO is a known issue and especially low >>>>>>>> >>> trafic website should not use it or use the always on feature >>>>>>>> >>> (maybe >>>>>>>> >>> this will change in the new pricing model) >>>>>>>> >>> >>>>>>>> >>> Regards, >>>>>>>> >>> >>>>>>>> >>> Erwin >>>>>>>> >>> >>>>>>>> >>> On Jun 7, 11:00 am, Ian Marshall <ianmarshall...@gmail.com> wrote: >>>>>>>> >>> > The low-level API does indeed look very fast. >>>>>>>> >>> > >>>>>>>> >>> > Just a comment on JDO: repeat runs roughly halve the JDO run >>>>>>>> >>> > time. I >>>>>>>> >>> > presume that this is because for repeat runs the JDO persistence >>>>>>>> >>> > manager factory has already been constructed. >>>>>>>> >>> > >>>>>>>> >>> > On Jun 6, 8:44 pm, DennisP <dennisbpeter...@gmail.com> wrote: >>>>>>>> >>> > >>>>>>>> >>> > > I'm looking at this online >>>>>>>> >>> > > demo:http://slim3demo.appspot.com/performance/ >>>>>>>> >>> > >>>>>>>> >>> > > Sample run: >>>>>>>> >>> > > The number of entities: 10000 >>>>>>>> >>> > > low-level API:get: 2 millis >>>>>>>> >>> > > Slim3: 2490 millis >>>>>>>> >>> > > JDO: 6030 millis >>>>>>>> >>> > >>>>>>>> >>> > > Is the low-level API really that much faster? >>>>>>>> >>> >>>>>>>> >>> -- >>>>>>>> >>> You received this message because you are subscribed to the Google >>>>>>>> >>> Groups >>>>>>>> >>> "Google App Engine for Java" group. >>>>>>>> >>> To post to this group, send email to >>>>>>>> >>> google-appengine-java@googlegroups.com. >>>>>>>> >>> To unsubscribe from this group, send email to >>>>>>>> >>> google-appengine-java+unsubscr...@googlegroups.com. >>>>>>>> >>> For more options, visit this group at >>>>>>>> >>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>>>> >>> >>>>>>>> >> >>>>>>>> >> -- >>>>>>>> >> You received this message because you are subscribed to the Google >>>>>>>> >> Groups >>>>>>>> >> "Google App Engine for Java" group. >>>>>>>> >> To post to this group, send email to >>>>>>>> >> google-appengine-java@googlegroups.com. >>>>>>>> >> To unsubscribe from this group, send email to >>>>>>>> >> google-appengine-java+unsubscr...@googlegroups.com. >>>>>>>> >> For more options, visit this group at >>>>>>>> >> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>>>> >> >>>>>>>> > >>>>>>>> > -- >>>>>>>> > You received this message because you are subscribed to the Google >>>>>>>> > Groups "Google App Engine for Java" group. >>>>>>>> > To post to this group, send email to >>>>>>>> > google-appengine-java@googlegroups.com. >>>>>>>> > To unsubscribe from this group, send email to >>>>>>>> > google-appengine-java+unsubscr...@googlegroups.com. >>>>>>>> > For more options, visit this group at >>>>>>>> > http://groups.google.com/group/google-appengine-java?hl=en. >>>>>>>> > >>>>>>>> > >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups >>>>>>>> "Google App Engine for Java" group. >>>>>>>> To post to this group, send email to >>>>>>>> google-appengine-java@googlegroups.com. >>>>>>>> To unsubscribe from this group, send email to >>>>>>>> google-appengine-java+unsubscr...@googlegroups.com. >>>>>>>> For more options, visit this group at >>>>>>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Guit: Elegant, beautiful, modular and *production ready* gwt >>>>>>> applications. >>>>>>> >>>>>>> http://code.google.com/p/guit/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups >>>>>>> "Google App Engine for Java" group. >>>>>>> To post to this group, send email to >>>>>>> google-appengine-java@googlegroups.com. >>>>>>> To unsubscribe from this group, send email to >>>>>>> google-appengine-java+unsubscr...@googlegroups.com. >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine for Java" group. >>>>>> To post to this group, send email to >>>>>> google-appengine-java@googlegroups.com. >>>>>> To unsubscribe from this group, send email to >>>>>> google-appengine-java+unsubscr...@googlegroups.com. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>> >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Google App Engine for Java" group. >>>>> To post to this group, send email to >>>>> google-appengine-java@googlegroups.com. >>>>> To unsubscribe from this group, send email to >>>>> google-appengine-java+unsubscr...@googlegroups.com. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine for Java" group. >>>> To post to this group, send email to >>>> google-appengine-java@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> google-appengine-java+unsubscr...@googlegroups.com. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine for Java" group. >>> To post to this group, send email to google-appengine-java@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine-java+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine-java?hl=en. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to google-appengine-java@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
