I don't understand this code at all. You're querying the data and then iterating over a separate call? And what does the extra call to list.size() have anything to do with this?
The code linked from http://slim3demo.appspot.com/performance/ has just gone back to calling list.size() without iterating the results, which once again goes back to a bogus benchmark. Jeff On Wed, Jun 8, 2011 at 9:30 PM, Yasuo Higa <higaya...@gmail.com> wrote: > Slim3 uses LL API. > > To resolve a strange issue that slim3 is faster than LL, I tried the > following samples: > > One: > AsyncDatastoreService ds = > DatastoreServiceFactory.getAsyncDatastoreService(); > Query q = new Query("Bar"); > PreparedQuery pq = ds.prepare(q); > List<Entity> list = > pq.asList(FetchOptions.Builder.withDefaults().limit( > Integer.MAX_VALUE)); > for (Entity e : service.getBarListUsingLL()) { > e.getKey(); > e.getProperty("sortValue"); > } > > Two: > AsyncDatastoreService ds = > DatastoreServiceFactory.getAsyncDatastoreService(); > Query q = new Query("Bar"); > PreparedQuery pq = ds.prepare(q); > List<Entity> list = > pq.asList(FetchOptions.Builder.withDefaults().limit( > Integer.MAX_VALUE)); > > // VERY IMPORTANT > list.size(); > > for (Entity e : service.getBarListUsingLL()) { > e.getKey(); > e.getProperty("sortValue"); > } > > The second one is much faster than the first one. > I fixed the samples to call list.size(). > http://slim3demo.appspot.com/performance/ > > As a result, LL is as fast as slim3 (^^; > > Yasuo Higa > > > > On Thu, Jun 9, 2011 at 10:17 AM, Jeff Schnitzer <j...@infohazard.org> wrote: >> Thank you for fixing the benchmark. >> >> I am very curious. According to this new benchmark - it's hard to >> tell without pushing the buttons a lot of times, but there seems to be >> a trend - Slim3 is somewhat faster than the Low Level API. >> >> Doesn't Slim3 use the Low Level API underneath? How can it possibly be >> faster? >> >> Jeff >> >> On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa <higaya...@gmail.com> wrote: >>> What I want to provide is a fair and casual benchmark. >>> >>> As jeff advised, I modified samples as follows: >>> for (Entity e : service.getBarListUsingLL()) { >>> e.getKey(); >>> e.getProperty("sortValue"); >>> } >>> >>> for (Bar bar : service.getBarListUsingSlim3()) { >>> bar.getKey(); >>> bar.getSortValue(); >>> } >>> >>> for (BarObjectify bar : service.getBarListUsingObjectify()) { >>> bar.getKey(); >>> bar.getSortValue(); >>> } >>> >>> for (BarJDO bar : service.getBarListUsingJDO()) { >>> bar.getKey(); >>> bar.getSortValue(); >>> } >>> >>> LL API is much slower than before. >>> http://slim3demo.appspot.com/performance/ >>> >>> Yasuo Higa >>> >>> >>> On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer <j...@infohazard.org> wrote: >>>> Slim3 may be a nice piece of software, but it has not been >>>> demonstrated to be faster than anything (including JDO). It might or >>>> might not be faster - I don't know - but based on the sloppy >>>> benchmarking, I'm pretty certain that the people making this claim >>>> don't know either. >>>> >>>> There's another ill-conceived performance claim on the Slim3 website: >>>> "You may worry about the overhead of global transactions. Don't worry. >>>> It is not very expensive." There are three problems with their little >>>> performance test: >>>> >>>> 1) It only measures wall-clock time, not cost. >>>> 2) It does not measure what happens under contention. >>>> 3) The numbers are obviously wrong - they don't even pass a smoke test. >>>> >>>> Look at these numbers (from the Slim3 home page): >>>> >>>> Entity Groups Local Transaction(millis) Global Transaction(millis) >>>> 1 67 70 >>>> 2 450 415 >>>> 3 213 539 >>>> 4 1498 981 >>>> 5 447 781 >>>> >>>> Just like the 1ms low-level API query performance in the benchmark >>>> that spawned this thread, even a casual observer should be able to >>>> recognize the obvious flaw - the numbers don't show any expected >>>> relationship between # of entity groups or the use of global >>>> transactions. Interpreted literally, you would assume that local >>>> transactions are much faster for 5 entity groups, but global >>>> transactions are much faster for 4 entity groups. >>>> >>>> It's pretty obvious that the benchmark author just ran one pass and >>>> took the numbers as-is. If you actually run multiple passes, you'll >>>> find that there is enormous variance in the timing. The only way you >>>> can realistically measure results like this on appengine is to execute >>>> the test 100 times and take a median. >>>> >>>> FWIW, I deliberately haven't made any performance claims for Objectify >>>> because I haven't put the necessary amount of due diligence into >>>> creating a proper set of benchmarks. It is not easy to measure >>>> performance, especially in a dynamic environment like appengine. I >>>> only hope that the Slim3 authors have put more care and thought into >>>> crafting their library than they have their benchmarks. >>>> >>>> Jeff >>>> >>>> >>>> On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber <gal.dol...@gmail.com> wrote: >>>>> Slim3 is not only fast, the api is completely awesome. It has been my >>>>> choice >>>>> for a year now for all gae projects. >>>>> It includes "name safety" and and amazing querying utils. >>>>> Very recommendable! >>>>> >>>>> On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer <j...@infohazard.org> >>>>> wrote: >>>>>> >>>>>> You are wrong. >>>>>> >>>>>> Try adding getProperty() calls to your LL performance test, and the >>>>>> speed advantage of the LL API goes away. I don't know what to say >>>>>> about Slim3, but here's my test case: >>>>>> >>>>>> >>>>>> http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test >>>>>> >>>>>> I created 10,000 entities in the datastore that have the same format >>>>>> as your test case - a single string property. Here's what happens >>>>>> (try it - and remember to reload the urls several times to get a >>>>>> realistic median value): >>>>>> >>>>>> ###### Low Level API with just .size() >>>>>> >>>>>> http://voodoodyne.appspot.com/fetchLLSize >>>>>> >>>>>> The code: >>>>>> >>>>>> List<Entity> things = >>>>>> DatastoreServiceFactory.getDatastoreService() >>>>>> .prepare(new >>>>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >>>>>> .asList(FetchOptions.Builder.withDefaults()); >>>>>> things.size(); >>>>>> >>>>>> Note that results are almost always under 2000ms. Wild guess I'd say >>>>>> the median elapsed is ~1900, just like your example. >>>>>> >>>>>> ###### Low Level API with actual fetch of the data >>>>>> >>>>>> http://voodoodyne.appspot.com/fetchLL >>>>>> >>>>>> The code: >>>>>> >>>>>> List<Entity> things = >>>>>> DatastoreServiceFactory.getDatastoreService() >>>>>> .prepare(new >>>>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >>>>>> .asList(FetchOptions.Builder.withDefaults()); >>>>>> for (Entity ent: things) >>>>>> { >>>>>> ent.getKey(); >>>>>> ent.getProperty("value"); >>>>>> } >>>>>> >>>>>> Note that the duration is now considerably longer. Eyeballing the >>>>>> median elapsed time, I'd say somewhere around 3000ms. >>>>>> >>>>>> ###### Objectify fetching from datastore >>>>>> >>>>>> http://voodoodyne.appspot.com/fetch >>>>>> >>>>>> Objectify ofy = ObjectifyService.begin(); >>>>>> List<Thing> things = ofy.query(Thing.class).list(); >>>>>> for (Thing thing: things) >>>>>> { >>>>>> thing.getId(); >>>>>> thing.getValue(); >>>>>> } >>>>>> >>>>>> Note that the timing is pretty much the same as the LL API when it >>>>>> includes actual fetches of the entity values. It is, no doubt, just a >>>>>> little higher. >>>>>> >>>>>> ###### A pure measurement of Objectify's overhead >>>>>> >>>>>> http://voodoodyne.appspot.com/fakeFetch >>>>>> >>>>>> This causes Objectify to translate 10,000 statically-created Entity >>>>>> objects to POJOs. You can see the code here: >>>>>> >>>>>> >>>>>> http://code.google.com/p/scratchmonkey/source/browse/appengine/performance-test/src/test/FakeFetchServlet.java >>>>>> >>>>>> You'll notice (after you hit the URL a couple times to warm up the >>>>>> JIT) that elapsed time converges to somewhere around 120ms. >>>>>> >>>>>> ----------- >>>>>> >>>>>> Conclusion: >>>>>> >>>>>> The numbers in the original benchmark are a result of improper >>>>>> measurements. The actual wall-clock overhead for Objectify in this >>>>>> test is ~4% (120ms out of 3000ms). >>>>>> >>>>>> Further speculation on my part, but probably correct: The overhead of >>>>>> reflection is unlikely to be a significant part of that 4%. >>>>>> >>>>>> Sloppy work. >>>>>> >>>>>> Jeff >>>>>> >>>>>> On Wed, Jun 8, 2011 at 7:55 AM, Yasuo Higa <higaya...@gmail.com> wrote: >>>>>> > It is not bogus. >>>>>> > LazyList#size() fetches all data as follows: >>>>>> > public int size() { >>>>>> > resolveAllData(); >>>>>> > return results.size(); >>>>>> > } >>>>>> > >>>>>> > Yasuo Higa >>>>>> > >>>>>> > On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson >>>>>> > <dennisbpeter...@gmail.com> wrote: >>>>>> >> It's not my benchmark, it's Slim3's :) ...but you're right, it's >>>>>> >> bogus. >>>>>> >> I >>>>>> >> asked on the main appengine group too, and it turns out the low-level >>>>>> >> benchmark is doing lazy loading. With that fixed, their numbers come >>>>>> >> out >>>>>> >> like yours. >>>>>> >> I found this one too, which also gets results like yours: >>>>>> >> http://gaejava.appspot.com/ >>>>>> >> >>>>>> >> On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur <erwin.str...@gmail.com> >>>>>> >> wrote: >>>>>> >>> >>>>>> >>> Indeed Dennis's measurements are very suspicious. First you should do >>>>>> >>> a couple of warming ups on each of the implementations to prevent >>>>>> >>> pollution like the JDO classpath scan for enhanced classes (which is >>>>>> >>> one of the reasons for the high initial run). Then do a couple of run >>>>>> >>> to determine a range of measurements to spot outlyers. your low-level >>>>>> >>> API 2millis is definately one. >>>>>> >>> >>>>>> >>> When I did the measurements I got the following results >>>>>> >>> low-level: 1150-1550 >>>>>> >>> Slim3: 1150-1600 >>>>>> >>> Objectify: 1950-2400 >>>>>> >>> JDO: 2100-2700 >>>>>> >>> >>>>>> >>> These measurements confirm that GAE designed implementations are >>>>>> >>> faster then the GAE implementation of a generic data access layer >>>>>> >>> (JDO), but not so extrem as initially posted. >>>>>> >>> >>>>>> >>> The initial response using JDO is a known issue and especially low >>>>>> >>> trafic website should not use it or use the always on feature (maybe >>>>>> >>> this will change in the new pricing model) >>>>>> >>> >>>>>> >>> Regards, >>>>>> >>> >>>>>> >>> Erwin >>>>>> >>> >>>>>> >>> On Jun 7, 11:00 am, Ian Marshall <ianmarshall...@gmail.com> wrote: >>>>>> >>> > The low-level API does indeed look very fast. >>>>>> >>> > >>>>>> >>> > Just a comment on JDO: repeat runs roughly halve the JDO run time. >>>>>> >>> > I >>>>>> >>> > presume that this is because for repeat runs the JDO persistence >>>>>> >>> > manager factory has already been constructed. >>>>>> >>> > >>>>>> >>> > On Jun 6, 8:44 pm, DennisP <dennisbpeter...@gmail.com> wrote: >>>>>> >>> > >>>>>> >>> > > I'm looking at this online >>>>>> >>> > > demo:http://slim3demo.appspot.com/performance/ >>>>>> >>> > >>>>>> >>> > > Sample run: >>>>>> >>> > > The number of entities: 10000 >>>>>> >>> > > low-level API:get: 2 millis >>>>>> >>> > > Slim3: 2490 millis >>>>>> >>> > > JDO: 6030 millis >>>>>> >>> > >>>>>> >>> > > Is the low-level API really that much faster? >>>>>> >>> >>>>>> >>> -- >>>>>> >>> You received this message because you are subscribed to the Google >>>>>> >>> Groups >>>>>> >>> "Google App Engine for Java" group. >>>>>> >>> To post to this group, send email to >>>>>> >>> google-appengine-java@googlegroups.com. >>>>>> >>> To unsubscribe from this group, send email to >>>>>> >>> google-appengine-java+unsubscr...@googlegroups.com. >>>>>> >>> For more options, visit this group at >>>>>> >>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>> >>> >>>>>> >> >>>>>> >> -- >>>>>> >> You received this message because you are subscribed to the Google >>>>>> >> Groups >>>>>> >> "Google App Engine for Java" group. >>>>>> >> To post to this group, send email to >>>>>> >> google-appengine-java@googlegroups.com. >>>>>> >> To unsubscribe from this group, send email to >>>>>> >> google-appengine-java+unsubscr...@googlegroups.com. >>>>>> >> For more options, visit this group at >>>>>> >> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>> >> >>>>>> > >>>>>> > -- >>>>>> > You received this message because you are subscribed to the Google >>>>>> > Groups "Google App Engine for Java" group. >>>>>> > To post to this group, send email to >>>>>> > google-appengine-java@googlegroups.com. >>>>>> > To unsubscribe from this group, send email to >>>>>> > google-appengine-java+unsubscr...@googlegroups.com. >>>>>> > For more options, visit this group at >>>>>> > http://groups.google.com/group/google-appengine-java?hl=en. >>>>>> > >>>>>> > >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google Groups >>>>>> "Google App Engine for Java" group. >>>>>> To post to this group, send email to >>>>>> google-appengine-java@googlegroups.com. >>>>>> To unsubscribe from this group, send email to >>>>>> google-appengine-java+unsubscr...@googlegroups.com. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Guit: Elegant, beautiful, modular and *production ready* gwt applications. >>>>> >>>>> http://code.google.com/p/guit/ >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Google App Engine for Java" group. >>>>> To post to this group, send email to >>>>> google-appengine-java@googlegroups.com. >>>>> To unsubscribe from this group, send email to >>>>> google-appengine-java+unsubscr...@googlegroups.com. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine for Java" group. >>>> To post to this group, send email to >>>> google-appengine-java@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> google-appengine-java+unsubscr...@googlegroups.com. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine-java?hl=en. >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine for Java" group. >>> To post to this group, send email to google-appengine-java@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine-java+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine-java?hl=en. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to google-appengine-java@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
