Slim3 may be a nice piece of software, but it has not been demonstrated to be faster than anything (including JDO). It might or might not be faster - I don't know - but based on the sloppy benchmarking, I'm pretty certain that the people making this claim don't know either.
There's another ill-conceived performance claim on the Slim3 website: "You may worry about the overhead of global transactions. Don't worry. It is not very expensive." There are three problems with their little performance test: 1) It only measures wall-clock time, not cost. 2) It does not measure what happens under contention. 3) The numbers are obviously wrong - they don't even pass a smoke test. Look at these numbers (from the Slim3 home page): Entity Groups Local Transaction(millis) Global Transaction(millis) 1 67 70 2 450 415 3 213 539 4 1498 981 5 447 781 Just like the 1ms low-level API query performance in the benchmark that spawned this thread, even a casual observer should be able to recognize the obvious flaw - the numbers don't show any expected relationship between # of entity groups or the use of global transactions. Interpreted literally, you would assume that local transactions are much faster for 5 entity groups, but global transactions are much faster for 4 entity groups. It's pretty obvious that the benchmark author just ran one pass and took the numbers as-is. If you actually run multiple passes, you'll find that there is enormous variance in the timing. The only way you can realistically measure results like this on appengine is to execute the test 100 times and take a median. FWIW, I deliberately haven't made any performance claims for Objectify because I haven't put the necessary amount of due diligence into creating a proper set of benchmarks. It is not easy to measure performance, especially in a dynamic environment like appengine. I only hope that the Slim3 authors have put more care and thought into crafting their library than they have their benchmarks. Jeff On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber <gal.dol...@gmail.com> wrote: > Slim3 is not only fast, the api is completely awesome. It has been my choice > for a year now for all gae projects. > It includes "name safety" and and amazing querying utils. > Very recommendable! > > On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer <j...@infohazard.org> wrote: >> >> You are wrong. >> >> Try adding getProperty() calls to your LL performance test, and the >> speed advantage of the LL API goes away. I don't know what to say >> about Slim3, but here's my test case: >> >> >> http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test >> >> I created 10,000 entities in the datastore that have the same format >> as your test case - a single string property. Here's what happens >> (try it - and remember to reload the urls several times to get a >> realistic median value): >> >> ###### Low Level API with just .size() >> >> http://voodoodyne.appspot.com/fetchLLSize >> >> The code: >> >> List<Entity> things = >> DatastoreServiceFactory.getDatastoreService() >> .prepare(new >> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >> .asList(FetchOptions.Builder.withDefaults()); >> things.size(); >> >> Note that results are almost always under 2000ms. Wild guess I'd say >> the median elapsed is ~1900, just like your example. >> >> ###### Low Level API with actual fetch of the data >> >> http://voodoodyne.appspot.com/fetchLL >> >> The code: >> >> List<Entity> things = >> DatastoreServiceFactory.getDatastoreService() >> .prepare(new >> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name())) >> .asList(FetchOptions.Builder.withDefaults()); >> for (Entity ent: things) >> { >> ent.getKey(); >> ent.getProperty("value"); >> } >> >> Note that the duration is now considerably longer. Eyeballing the >> median elapsed time, I'd say somewhere around 3000ms. >> >> ###### Objectify fetching from datastore >> >> http://voodoodyne.appspot.com/fetch >> >> Objectify ofy = ObjectifyService.begin(); >> List<Thing> things = ofy.query(Thing.class).list(); >> for (Thing thing: things) >> { >> thing.getId(); >> thing.getValue(); >> } >> >> Note that the timing is pretty much the same as the LL API when it >> includes actual fetches of the entity values. It is, no doubt, just a >> little higher. >> >> ###### A pure measurement of Objectify's overhead >> >> http://voodoodyne.appspot.com/fakeFetch >> >> This causes Objectify to translate 10,000 statically-created Entity >> objects to POJOs. You can see the code here: >> >> >> http://code.google.com/p/scratchmonkey/source/browse/appengine/performance-test/src/test/FakeFetchServlet.java >> >> You'll notice (after you hit the URL a couple times to warm up the >> JIT) that elapsed time converges to somewhere around 120ms. >> >> ----------- >> >> Conclusion: >> >> The numbers in the original benchmark are a result of improper >> measurements. The actual wall-clock overhead for Objectify in this >> test is ~4% (120ms out of 3000ms). >> >> Further speculation on my part, but probably correct: The overhead of >> reflection is unlikely to be a significant part of that 4%. >> >> Sloppy work. >> >> Jeff >> >> On Wed, Jun 8, 2011 at 7:55 AM, Yasuo Higa <higaya...@gmail.com> wrote: >> > It is not bogus. >> > LazyList#size() fetches all data as follows: >> > public int size() { >> > resolveAllData(); >> > return results.size(); >> > } >> > >> > Yasuo Higa >> > >> > On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson >> > <dennisbpeter...@gmail.com> wrote: >> >> It's not my benchmark, it's Slim3's :) ...but you're right, it's bogus. >> >> I >> >> asked on the main appengine group too, and it turns out the low-level >> >> benchmark is doing lazy loading. With that fixed, their numbers come >> >> out >> >> like yours. >> >> I found this one too, which also gets results like yours: >> >> http://gaejava.appspot.com/ >> >> >> >> On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur <erwin.str...@gmail.com> >> >> wrote: >> >>> >> >>> Indeed Dennis's measurements are very suspicious. First you should do >> >>> a couple of warming ups on each of the implementations to prevent >> >>> pollution like the JDO classpath scan for enhanced classes (which is >> >>> one of the reasons for the high initial run). Then do a couple of run >> >>> to determine a range of measurements to spot outlyers. your low-level >> >>> API 2millis is definately one. >> >>> >> >>> When I did the measurements I got the following results >> >>> low-level: 1150-1550 >> >>> Slim3: 1150-1600 >> >>> Objectify: 1950-2400 >> >>> JDO: 2100-2700 >> >>> >> >>> These measurements confirm that GAE designed implementations are >> >>> faster then the GAE implementation of a generic data access layer >> >>> (JDO), but not so extrem as initially posted. >> >>> >> >>> The initial response using JDO is a known issue and especially low >> >>> trafic website should not use it or use the always on feature (maybe >> >>> this will change in the new pricing model) >> >>> >> >>> Regards, >> >>> >> >>> Erwin >> >>> >> >>> On Jun 7, 11:00 am, Ian Marshall <ianmarshall...@gmail.com> wrote: >> >>> > The low-level API does indeed look very fast. >> >>> > >> >>> > Just a comment on JDO: repeat runs roughly halve the JDO run time. I >> >>> > presume that this is because for repeat runs the JDO persistence >> >>> > manager factory has already been constructed. >> >>> > >> >>> > On Jun 6, 8:44 pm, DennisP <dennisbpeter...@gmail.com> wrote: >> >>> > >> >>> > > I'm looking at this online >> >>> > > demo:http://slim3demo.appspot.com/performance/ >> >>> > >> >>> > > Sample run: >> >>> > > The number of entities: 10000 >> >>> > > low-level API:get: 2 millis >> >>> > > Slim3: 2490 millis >> >>> > > JDO: 6030 millis >> >>> > >> >>> > > Is the low-level API really that much faster? >> >>> >> >>> -- >> >>> You received this message because you are subscribed to the Google >> >>> Groups >> >>> "Google App Engine for Java" group. >> >>> To post to this group, send email to >> >>> google-appengine-java@googlegroups.com. >> >>> To unsubscribe from this group, send email to >> >>> google-appengine-java+unsubscr...@googlegroups.com. >> >>> For more options, visit this group at >> >>> http://groups.google.com/group/google-appengine-java?hl=en. >> >>> >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "Google App Engine for Java" group. >> >> To post to this group, send email to >> >> google-appengine-java@googlegroups.com. >> >> To unsubscribe from this group, send email to >> >> google-appengine-java+unsubscr...@googlegroups.com. >> >> For more options, visit this group at >> >> http://groups.google.com/group/google-appengine-java?hl=en. >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Google App Engine for Java" group. >> > To post to this group, send email to >> > google-appengine-java@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine-java+unsubscr...@googlegroups.com. >> > For more options, visit this group at >> > http://groups.google.com/group/google-appengine-java?hl=en. >> > >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-java@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> > > > > -- > Guit: Elegant, beautiful, modular and *production ready* gwt applications. > > http://code.google.com/p/guit/ > > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
