It's *NO* bogus benchmark because the sample iterates the results. http://code.google.com/p/slim3/source/browse/trunk/slim3demo/src/slim3/demo/controller/performance/GetLLController.java
Yasuo Higa > > The code linked from http://slim3demo.appspot.com/performance/ has > just gone back to calling list.size() without iterating the results, > which once again goes back to a bogus benchmark. > > Jeff > > On Wed, Jun 8, 2011 at 9:30 PM, Yasuo Higa <higaya...@gmail.com> wrote: >> Slim3 uses LL API. >> >> To resolve a strange issue that slim3 is faster than LL, I tried the >> following samples: >> >> One: >> AsyncDatastoreService ds = >> DatastoreServiceFactory.getAsyncDatastoreService(); >> Query q = new Query("Bar"); >> PreparedQuery pq = ds.prepare(q); >> List<Entity> list = >> pq.asList(FetchOptions.Builder.withDefaults().limit( >> Integer.MAX_VALUE)); >> for (Entity e : service.getBarListUsingLL()) { >> e.getKey(); >> e.getProperty("sortValue"); >> } >> >> Two: >> AsyncDatastoreService ds = >> DatastoreServiceFactory.getAsyncDatastoreService(); >> Query q = new Query("Bar"); >> PreparedQuery pq = ds.prepare(q); >> List<Entity> list = >> pq.asList(FetchOptions.Builder.withDefaults().limit( >> Integer.MAX_VALUE)); >> >> // VERY IMPORTANT >> list.size(); >> >> for (Entity e : service.getBarListUsingLL()) { >> e.getKey(); >> e.getProperty("sortValue"); >> } >> >> The second one is much faster than the first one. >> I fixed the samples to call list.size(). >> http://slim3demo.appspot.com/performance/ >> >> As a result, LL is as fast as slim3 (^^; >> >> Yasuo Higa >> >> >> >> On Thu, Jun 9, 2011 at 10:17 AM, Jeff Schnitzer <j...@infohazard.org> wrote: >>> Thank you for fixing the benchmark. >>> >>> I am very curious. According to this new benchmark - it's hard to >>> tell without pushing the buttons a lot of times, but there seems to be >>> a trend - Slim3 is somewhat faster than the Low Level API. >>> >>> Doesn't Slim3 use the Low Level API underneath? How can it possibly be >>> faster? >>> >>> Jeff >>> >>> On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa <higaya...@gmail.com> wrote: >>>> What I want to provide is a fair and casual benchmark. >>>> >>>> As jeff advised, I modified samples as follows: >>>> for (Entity e : service.getBarListUsingLL()) { >>>> e.getKey(); >>>> e.getProperty("sortValue"); >>>> } >>>> >>>> for (Bar bar : service.getBarListUsingSlim3()) { >>>> bar.getKey(); >>>> bar.getSortValue(); >>>> } >>>> >>>> for (BarObjectify bar : service.getBarListUsingObjectify()) { >>>> bar.getKey(); >>>> bar.getSortValue(); >>>> } >>>> >>>> for (BarJDO bar : service.getBarListUsingJDO()) { >>>> bar.getKey(); >>>> bar.getSortValue(); >>>> } >>>> >>>> LL API is much slower than before. >>>> http://slim3demo.appspot.com/performance/ >>>> >>>> Yasuo Higa >>>> >>>> >>>> On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer <j...@infohazard.org> wrote: >>>>> Slim3 may be a nice piece of software, but it has not been >>>>> demonstrated to be faster than anything (including JDO). It might or >>>>> might not be faster - I don't know - but based on the sloppy >>>>> benchmarking, I'm pretty certain that the people making this claim >>>>> don't know either. >>>>> >>>>> There's another ill-conceived performance claim on the Slim3 website: >>>>> "You may worry about the overhead of global transactions. Don't worry. >>>>> It is not very expensive." There are three problems with their little >>>>> performance test: >>>>> >>>>> 1) It only measures wall-clock time, not cost. >>>>> 2) It does not measure what happens under contention. >>>>> 3) The numbers are obviously wrong - they don't even pass a smoke test. >>>>> >>>>> Look at these numbers (from the Slim3 home page): >>>>> >>>>> Entity Groups Local Transaction(millis) Global Transaction(millis) >>>>> 1 67 70 >>>>> 2 450 415 >>>>> 3 213 539 >>>>> 4 1498 981 >>>>> 5 447 781 >>>>> >>>>> Just like the 1ms low-level API query performance in the benchmark >>>>> that spawned this thread, even a casual observer should be able to >>>>> recognize the obvious flaw - the numbers don't show any expected >>>>> relationship between # of entity groups or the use of global >>>>> transactions. Interpreted literally, you would assume that local >>>>> transactions are much faster for 5 entity groups, but global >>>>> transactions are much faster for 4 entity groups. >>>>> >>>>> It's pretty obvious that the benchmark author just ran one pass and >>>>> took the numbers as-is. If you actually run multiple passes, you'll >>>>> find that there is enormous variance in the timing. The only way you >>>>> can realistically measure results like this on appengine is to execute >>>>> the test 100 times and take a median. >>>>> >>>>> FWIW, I deliberately haven't made any performance claims for Objectify >>>>> because I haven't put the necessary amount of due diligence into >>>>> creating a proper set of benchmarks. It is not easy to measure >>>>> performance, especially in a dynamic environment like appengine. I >>>>> only hope that the Slim3 authors have put more care and thought into >>>>> crafting their library than they have their benchmarks. >>>>> >>>>> Jeff >>>>> >>>>> >>>>> On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber <gal.dol...@gmail.com> wrote: >>>>>> Slim3 is not only fast, the api is completely awesome. It has been my >>>>>> choice >>>>>> for a year now for all gae projects. >>>>>> It includes "name safety" and and amazing querying utils. >>>>>> Very recommendable! >>>>>> >>>>>> On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer <j...@infohazard.org> >>>>>> wrote: >>>>>>> >>>>>>> You are wrong. >>>>>>> >>>>>>> Try adding getProperty() calls to your LL performance test, and the >>>>>>> speed advantage of the LL API goes away. I don't know what to say >>>>>>> about Slim3, but here's my test case: >>>>>>> >>>>>>> >>>>>>> http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test >>>>>>> >>>>>>> I created 10,000 entities in the datastore that have the same format >>>>>>> as your test case - a single string property. Here's what happens >>>>>>> (try it - and remember to reload the urls several times to get a >>>>>>> realistic median value): >>>>>>> >>>>>>> ###### Low Level API with just .size() >>>>>>> >>>>>>> http://voodoodyne.appspot.com/fetchLLSize >>>>>>> >>>>>>> The code: >>>>>>> >>>>>>> List<Entity> things = >>>>>>> DatastoreServiceFactory.getDatastoreService() >>>>>>> .prepare(new >>>>>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >>>>>>> .asList(FetchOptions.Builder.withDefaults()); >>>>>>> things.size(); >>>>>>> >>>>>>> Note that results are almost always under 2000ms. Wild guess I'd say >>>>>>> the median elapsed is ~1900, just like your example. >>>>>>> >>>>>>> ###### Low Level API with actual fetch of the data >>>>>>> >>>>>>> http://voodoodyne.appspot.com/fetchLL >>>>>>> >>>>>>> The code: >>>>>>> >>>>>>> List<Entity> things = >>>>>>> DatastoreServiceFactory.getDatastoreService() >>>>>>> .prepare(new >>>>>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >>>>>>> .asList(FetchOptions.Builder.withDefaults()); >>>>>>> for (Entity ent: things) >>>>>>> { >>>>>>> ent.getKey(); >>>>>>> ent.getProperty("value"); >>>>>>> } >>>>>>> >>>>>>> Note that the duration is now considerably longer. Eyeballing the >>>>>>> median elapsed time, I'd say somewhere around 3000ms. >>>>>>> >>>>>>> ###### Objectify fetching from datastore >>>>>>> >>>>>>> http://voodoodyne.appspot.com/fetch >>>>>>> >>>>>>> Objectify ofy = ObjectifyService.begin(); >>>>>>> List<Thing> things = ofy.query(Thing.class).list(); >>>>>>> for (Thing thing: things) >>>>>>> { >>>>>>> thing.getId(); >>>>>>> thing.getValue(); >>>>>>> } >>>>>>> >>>>>>> Note that the timing is pretty much the same as the LL API when it >>>>>>> includes actual fetches of the entity values. It is, no doubt, just a >>>>>>> little higher. >>>>>>> >>>>>>> ###### A pure measurement of Objectify's overhead >>>>>>> >>>>>>> http://voodoodyne.appspot.com/fakeFetch >>>>>>> >>>>>>> This causes Objectify to translate 10,000 statically-created Entity >>>>>>> objects to POJOs. You can see the code here: >>>>>>> >>>>>>> >>>>>>> http://code.google.com/p/scratchmonkey/source/browse/appengine/performance-test/src/test/FakeFetchServlet.java >>>>>>> >>>>>>> You'll notice (after you hit the URL a couple times to warm up the >>>>>>> JIT) that elapsed time converges to somewhere around 120ms. >>>>>>> >>>>>>> ----------- >>>>>>> >>>>>>> Conclusion: >>>>>>> >>>>>>> The numbers in the original benchmark are a result of improper >>>>>>> measurements. The actual wall-clock overhead for Objectify in this >>>>>>> test is ~4% (120ms out of 3000ms). >>>>>>> >>>>>>> Further speculation on my part, but probably correct: The overhead of >>>>>>> reflection is unlikely to be a significant part of that 4%. >>>>>>> >>>>>>> Sloppy work. >>>>>>> >>>>>>> Jeff >>>>>>> >>>>>>> On Wed, Jun 8, 2011 at 7:55 AM, Yasuo Higa <higaya...@gmail.com> wrote: >>>>>>> > It is not bogus. >>>>>>> > LazyList#size() fetches all data as follows: >>>>>>> > public int size() { >>>>>>> > resolveAllData(); >>>>>>> > return results.size(); >>>>>>> > } >>>>>>> > >>>>>>> > Yasuo Higa >>>>>>> > >>>>>>> > On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson >>>>>>> > <dennisbpeter...@gmail.com> wrote: >>>>>>> >> It's not my benchmark, it's Slim3's :) ...but you're right, it's >>>>>>> >> bogus. >>>>>>> >> I >>>>>>> >> asked on the main appengine group too, and it turns out the low-level >>>>>>> >> benchmark is doing lazy loading. With that fixed, their numbers come >>>>>>> >> out >>>>>>> >> like yours. >>>>>>> >> I found this one too, which also gets results like yours: >>>>>>> >> http://gaejava.appspot.com/ >>>>>>> >> >>>>>>> >> On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur <erwin.str...@gmail.com> >>>>>>> >> wrote: >>>>>>> >>> >>>>>>> >>> Indeed Dennis's measurements are very suspicious. First you should >>>>>>> >>> do >>>>>>> >>> a couple of warming ups on each of the implementations to prevent >>>>>>> >>> pollution like the JDO classpath scan for enhanced classes (which is >>>>>>> >>> one of the reasons for the high initial run). Then do a couple of >>>>>>> >>> run >>>>>>> >>> to determine a range of measurements to spot outlyers. your >>>>>>> >>> low-level >>>>>>> >>> API 2millis is definately one. >>>>>>> >>> >>>>>>> >>> When I did the measurements I got the following results >>>>>>> >>> low-level: 1150-1550 >>>>>>> >>> Slim3: 1150-1600 >>>>>>> >>> Objectify: 1950-2400 >>>>>>> >>> JDO: 2100-2700 >>>>>>> >>> >>>>>>> >>> These measurements confirm that GAE designed implementations are >>>>>>> >>> faster then the GAE implementation of a generic data access layer >>>>>>> >>> (JDO), but not so extrem as initially posted. >>>>>>> >>> >>>>>>> >>> The initial response using JDO is a known issue and especially low >>>>>>> >>> trafic website should not use it or use the always on feature (maybe >>>>>>> >>> this will change in the new pricing model) >>>>>>> >>> >>>>>>> >>> Regards, >>>>>>> >>> >>>>>>> >>> Erwin >>>>>>> >>> >>>>>>> >>> On Jun 7, 11:00 am, Ian Marshall <ianmarshall...@gmail.com> wrote: >>>>>>> >>> > The low-level API does indeed look very fast. >>>>>>> >>> > >>>>>>> >>> > Just a comment on JDO: repeat runs roughly halve the JDO run >>>>>>> >>> > time. I >>>>>>> >>> > presume that this is because for repeat runs the JDO persistence >>>>>>> >>> > manager factory has already been constructed. >>>>>>> >>> > >>>>>>> >>> > On Jun 6, 8:44 pm, DennisP <dennisbpeter...@gmail.com> wrote: >>>>>>> >>> > >>>>>>> >>> > > I'm looking at this online >>>>>>> >>> > > demo:http://slim3demo.appspot.com/performance/ >>>>>>> >>> > >>>>>>> >>> > > Sample run: >>>>>>> >>> > > The number of entities: 10000 >>>>>>> >>> > > low-level API:get: 2 millis >>>>>>> >>> > > Slim3: 2490 millis >>>>>>> >>> > > JDO: 6030 millis >>>>>>> >>> > >>>>>>> >>> > > Is the low-level API really that much faster? >>>>>>> >>> >>>>>>> >>> -- >>>>>>> >>> You received this message because you are subscribed to the Google >>>>>>> >>> Groups >>>>>>> >>> "Google App Engine for Java" group. >>>>>>> >>> To post to this group, send email to >>>>>>> >>> google-appengine-java@googlegroups.com. >>>>>>> >>> To unsubscribe from this group, send email to >>>>>>> >>> google-appengine-java+unsubscr...@googlegroups.com. >>>>>>> >>> For more options, visit this group at >>>>>>> >>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>>> >>> >>>>>>> >> >>>>>>> >> -- >>>>>>> >> You received this message because you are subscribed to the Google >>>>>>> >> Groups >>>>>>> >> "Google App Engine for Java" group. >>>>>>> >> To post to this group, send email to >>>>>>> >> google-appengine-java@googlegroups.com. >>>>>>> >> To unsubscribe from this group, send email to >>>>>>> >> google-appengine-java+unsubscr...@googlegroups.com. >>>>>>> >> For more options, visit this group at >>>>>>> >> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>>> >> >>>>>>> > >>>>>>> > -- >>>>>>> > You received this message because you are subscribed to the Google >>>>>>> > Groups "Google App Engine for Java" group. >>>>>>> > To post to this group, send email to >>>>>>> > google-appengine-java@googlegroups.com. >>>>>>> > To unsubscribe from this group, send email to >>>>>>> > google-appengine-java+unsubscr...@googlegroups.com. >>>>>>> > For more options, visit this group at >>>>>>> > http://groups.google.com/group/google-appengine-java?hl=en. >>>>>>> > >>>>>>> > >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups >>>>>>> "Google App Engine for Java" group. >>>>>>> To post to this group, send email to >>>>>>> google-appengine-java@googlegroups.com. >>>>>>> To unsubscribe from this group, send email to >>>>>>> google-appengine-java+unsubscr...@googlegroups.com. >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Guit: Elegant, beautiful, modular and *production ready* gwt >>>>>> applications. >>>>>> >>>>>> http://code.google.com/p/guit/ >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google Groups >>>>>> "Google App Engine for Java" group. >>>>>> To post to this group, send email to >>>>>> google-appengine-java@googlegroups.com. >>>>>> To unsubscribe from this group, send email to >>>>>> google-appengine-java+unsubscr...@googlegroups.com. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Google App Engine for Java" group. >>>>> To post to this group, send email to >>>>> google-appengine-java@googlegroups.com. >>>>> To unsubscribe from this group, send email to >>>>> google-appengine-java+unsubscr...@googlegroups.com. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine for Java" group. >>>> To post to this group, send email to >>>> google-appengine-java@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> google-appengine-java+unsubscr...@googlegroups.com. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine for Java" group. >>> To post to this group, send email to google-appengine-java@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine-java+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine-java?hl=en. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to google-appengine-java@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
