Slim3 uses LL API.
To resolve a strange issue that slim3 is faster than LL, I tried the
following samples:
One:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query("Bar");
PreparedQuery pq = ds.prepare(q);
List<Entity> list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty("sortValue");
}
Two:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query("Bar");
PreparedQuery pq = ds.prepare(q);
List<Entity> list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
// VERY IMPORTANT
list.size();
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty("sortValue");
}
The second one is much faster than the first one.
I fixed the samples to call list.size().
http://slim3demo.appspot.com/performance/
As a result, LL is as fast as slim3 (^^;
Yasuo Higa
On Thu, Jun 9, 2011 at 10:17 AM, Jeff Schnitzer <j...@infohazard.org> wrote:
> Thank you for fixing the benchmark.
>
> I am very curious. According to this new benchmark - it's hard to
> tell without pushing the buttons a lot of times, but there seems to be
> a trend - Slim3 is somewhat faster than the Low Level API.
>
> Doesn't Slim3 use the Low Level API underneath? How can it possibly be
> faster?
>
> Jeff
>
> On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa <higaya...@gmail.com> wrote:
>> What I want to provide is a fair and casual benchmark.
>>
>> As jeff advised, I modified samples as follows:
>> for (Entity e : service.getBarListUsingLL()) {
>> e.getKey();
>> e.getProperty("sortValue");
>> }
>>
>> for (Bar bar : service.getBarListUsingSlim3()) {
>> bar.getKey();
>> bar.getSortValue();
>> }
>>
>> for (BarObjectify bar : service.getBarListUsingObjectify()) {
>> bar.getKey();
>> bar.getSortValue();
>> }
>>
>> for (BarJDO bar : service.getBarListUsingJDO()) {
>> bar.getKey();
>> bar.getSortValue();
>> }
>>
>> LL API is much slower than before.
>> http://slim3demo.appspot.com/performance/
>>
>> Yasuo Higa
>>
>>
>> On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer <j...@infohazard.org> wrote:
>>> Slim3 may be a nice piece of software, but it has not been
>>> demonstrated to be faster than anything (including JDO). It might or
>>> might not be faster - I don't know - but based on the sloppy
>>> benchmarking, I'm pretty certain that the people making this claim
>>> don't know either.
>>>
>>> There's another ill-conceived performance claim on the Slim3 website:
>>> "You may worry about the overhead of global transactions. Don't worry.
>>> It is not very expensive." There are three problems with their little
>>> performance test:
>>>
>>> 1) It only measures wall-clock time, not cost.
>>> 2) It does not measure what happens under contention.
>>> 3) The numbers are obviously wrong - they don't even pass a smoke test.
>>>
>>> Look at these numbers (from the Slim3 home page):
>>>
>>> Entity Groups Local Transaction(millis) Global Transaction(millis)
>>> 1 67 70
>>> 2 450 415
>>> 3 213 539
>>> 4 1498 981
>>> 5 447 781
>>>
>>> Just like the 1ms low-level API query performance in the benchmark
>>> that spawned this thread, even a casual observer should be able to
>>> recognize the obvious flaw - the numbers don't show any expected
>>> relationship between # of entity groups or the use of global
>>> transactions. Interpreted literally, you would assume that local
>>> transactions are much faster for 5 entity groups, but global
>>> transactions are much faster for 4 entity groups.
>>>
>>> It's pretty obvious that the benchmark author just ran one pass and
>>> took the numbers as-is. If you actually run multiple passes, you'll
>>> find that there is enormous variance in the timing. The only way you
>>> can realistically measure results like this on appengine is to execute
>>> the test 100 times and take a median.
>>>
>>> FWIW, I deliberately haven't made any performance claims for Objectify
>>> because I haven't put the necessary amount of due diligence into
>>> creating a proper set of benchmarks. It is not easy to measure
>>> performance, especially in a dynamic environment like appengine. I
>>> only hope that the Slim3 authors have put more care and thought into
>>> crafting their library than they have their benchmarks.
>>>
>>> Jeff
>>>
>>>
>>> On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber <gal.dol...@gmail.com> wrote:
>>>> Slim3 is not only fast, the api is completely awesome. It has been my
>>>> choice
>>>> for a year now for all gae projects.
>>>> It includes "name safety" and and amazing querying utils.
>>>> Very recommendable!
>>>>
>>>> On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer <j...@infohazard.org> wrote:
>>>>>
>>>>> You are wrong.
>>>>>
>>>>> Try adding getProperty() calls to your LL performance test, and the
>>>>> speed advantage of the LL API goes away. I don't know what to say
>>>>> about Slim3, but here's my test case:
>>>>>
>>>>>
>>>>> http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test
>>>>>
>>>>> I created 10,000 entities in the datastore that have the same format
>>>>> as your test case - a single string property. Here's what happens
>>>>> (try it - and remember to reload the urls several times to get a
>>>>> realistic median value):
>>>>>
>>>>> ###### Low Level API with just .size()
>>>>>
>>>>> http://voodoodyne.appspot.com/fetchLLSize
>>>>>
>>>>> The code:
>>>>>
>>>>> List<Entity> things =
>>>>> DatastoreServiceFactory.getDatastoreService()
>>>>> .prepare(new
>>>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
>>>>> .asList(FetchOptions.Builder.withDefaults());
>>>>> things.size();
>>>>>
>>>>> Note that results are almost always under 2000ms. Wild guess I'd say
>>>>> the median elapsed is ~1900, just like your example.
>>>>>
>>>>> ###### Low Level API with actual fetch of the data
>>>>>
>>>>> http://voodoodyne.appspot.com/fetchLL
>>>>>
>>>>> The code:
>>>>>
>>>>> List<Entity> things =
>>>>> DatastoreServiceFactory.getDatastoreService()
>>>>> .prepare(new
>>>>> Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
>>>>> .asList(FetchOptions.Builder.withDefaults());
>>>>> for (Entity ent: things)
>>>>> {
>>>>> ent.getKey();
>>>>> ent.getProperty("value");
>>>>> }
>>>>>
>>>>> Note that the duration is now considerably longer. Eyeballing the
>>>>> median elapsed time, I'd say somewhere around 3000ms.
>>>>>
>>>>> ###### Objectify fetching from datastore
>>>>>
>>>>> http://voodoodyne.appspot.com/fetch
>>>>>
>>>>> Objectify ofy = ObjectifyService.begin();
>>>>> List<Thing> things = ofy.query(Thing.class).list();
>>>>> for (Thing thing: things)
>>>>> {
>>>>> thing.getId();
>>>>> thing.getValue();
>>>>> }
>>>>>
>>>>> Note that the timing is pretty much the same as the LL API when it
>>>>> includes actual fetches of the entity values. It is, no doubt, just a
>>>>> little higher.
>>>>>
>>>>> ###### A pure measurement of Objectify's overhead
>>>>>
>>>>> http://voodoodyne.appspot.com/fakeFetch
>>>>>
>>>>> This causes Objectify to translate 10,000 statically-created Entity
>>>>> objects to POJOs. You can see the code here:
>>>>>
>>>>>
>>>>> http://code.google.com/p/scratchmonkey/source/browse/appengine/performance-test/src/test/FakeFetchServlet.java
>>>>>
>>>>> You'll notice (after you hit the URL a couple times to warm up the
>>>>> JIT) that elapsed time converges to somewhere around 120ms.
>>>>>
>>>>> -----------
>>>>>
>>>>> Conclusion:
>>>>>
>>>>> The numbers in the original benchmark are a result of improper
>>>>> measurements. The actual wall-clock overhead for Objectify in this
>>>>> test is ~4% (120ms out of 3000ms).
>>>>>
>>>>> Further speculation on my part, but probably correct: The overhead of
>>>>> reflection is unlikely to be a significant part of that 4%.
>>>>>
>>>>> Sloppy work.
>>>>>
>>>>> Jeff
>>>>>
>>>>> On Wed, Jun 8, 2011 at 7:55 AM, Yasuo Higa <higaya...@gmail.com> wrote:
>>>>> > It is not bogus.
>>>>> > LazyList#size() fetches all data as follows:
>>>>> > public int size() {
>>>>> > resolveAllData();
>>>>> > return results.size();
>>>>> > }
>>>>> >
>>>>> > Yasuo Higa
>>>>> >
>>>>> > On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson
>>>>> > <dennisbpeter...@gmail.com> wrote:
>>>>> >> It's not my benchmark, it's Slim3's :) ...but you're right, it's bogus.
>>>>> >> I
>>>>> >> asked on the main appengine group too, and it turns out the low-level
>>>>> >> benchmark is doing lazy loading. With that fixed, their numbers come
>>>>> >> out
>>>>> >> like yours.
>>>>> >> I found this one too, which also gets results like yours:
>>>>> >> http://gaejava.appspot.com/
>>>>> >>
>>>>> >> On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur <erwin.str...@gmail.com>
>>>>> >> wrote:
>>>>> >>>
>>>>> >>> Indeed Dennis's measurements are very suspicious. First you should do
>>>>> >>> a couple of warming ups on each of the implementations to prevent
>>>>> >>> pollution like the JDO classpath scan for enhanced classes (which is
>>>>> >>> one of the reasons for the high initial run). Then do a couple of run
>>>>> >>> to determine a range of measurements to spot outlyers. your low-level
>>>>> >>> API 2millis is definately one.
>>>>> >>>
>>>>> >>> When I did the measurements I got the following results
>>>>> >>> low-level: 1150-1550
>>>>> >>> Slim3: 1150-1600
>>>>> >>> Objectify: 1950-2400
>>>>> >>> JDO: 2100-2700
>>>>> >>>
>>>>> >>> These measurements confirm that GAE designed implementations are
>>>>> >>> faster then the GAE implementation of a generic data access layer
>>>>> >>> (JDO), but not so extrem as initially posted.
>>>>> >>>
>>>>> >>> The initial response using JDO is a known issue and especially low
>>>>> >>> trafic website should not use it or use the always on feature (maybe
>>>>> >>> this will change in the new pricing model)
>>>>> >>>
>>>>> >>> Regards,
>>>>> >>>
>>>>> >>> Erwin
>>>>> >>>
>>>>> >>> On Jun 7, 11:00 am, Ian Marshall <ianmarshall...@gmail.com> wrote:
>>>>> >>> > The low-level API does indeed look very fast.
>>>>> >>> >
>>>>> >>> > Just a comment on JDO: repeat runs roughly halve the JDO run time. I
>>>>> >>> > presume that this is because for repeat runs the JDO persistence
>>>>> >>> > manager factory has already been constructed.
>>>>> >>> >
>>>>> >>> > On Jun 6, 8:44 pm, DennisP <dennisbpeter...@gmail.com> wrote:
>>>>> >>> >
>>>>> >>> > > I'm looking at this online
>>>>> >>> > > demo:http://slim3demo.appspot.com/performance/
>>>>> >>> >
>>>>> >>> > > Sample run:
>>>>> >>> > > The number of entities: 10000
>>>>> >>> > > low-level API:get: 2 millis
>>>>> >>> > > Slim3: 2490 millis
>>>>> >>> > > JDO: 6030 millis
>>>>> >>> >
>>>>> >>> > > Is the low-level API really that much faster?
>>>>> >>>
>>>>> >>> --
>>>>> >>> You received this message because you are subscribed to the Google
>>>>> >>> Groups
>>>>> >>> "Google App Engine for Java" group.
>>>>> >>> To post to this group, send email to
>>>>> >>> google-appengine-java@googlegroups.com.
>>>>> >>> To unsubscribe from this group, send email to
>>>>> >>> google-appengine-java+unsubscr...@googlegroups.com.
>>>>> >>> For more options, visit this group at
>>>>> >>> http://groups.google.com/group/google-appengine-java?hl=en.
>>>>> >>>
>>>>> >>
>>>>> >> --
>>>>> >> You received this message because you are subscribed to the Google
>>>>> >> Groups
>>>>> >> "Google App Engine for Java" group.
>>>>> >> To post to this group, send email to
>>>>> >> google-appengine-java@googlegroups.com.
>>>>> >> To unsubscribe from this group, send email to
>>>>> >> google-appengine-java+unsubscr...@googlegroups.com.
>>>>> >> For more options, visit this group at
>>>>> >> http://groups.google.com/group/google-appengine-java?hl=en.
>>>>> >>
>>>>> >
>>>>> > --
>>>>> > You received this message because you are subscribed to the Google
>>>>> > Groups "Google App Engine for Java" group.
>>>>> > To post to this group, send email to
>>>>> > google-appengine-java@googlegroups.com.
>>>>> > To unsubscribe from this group, send email to
>>>>> > google-appengine-java+unsubscr...@googlegroups.com.
>>>>> > For more options, visit this group at
>>>>> > http://groups.google.com/group/google-appengine-java?hl=en.
>>>>> >
>>>>> >
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Google App Engine for Java" group.
>>>>> To post to this group, send email to
>>>>> google-appengine-java@googlegroups.com.
>>>>> To unsubscribe from this group, send email to
>>>>> google-appengine-java+unsubscr...@googlegroups.com.
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/google-appengine-java?hl=en.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Guit: Elegant, beautiful, modular and *production ready* gwt applications.
>>>>
>>>> http://code.google.com/p/guit/
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "Google App Engine for Java" group.
>>>> To post to this group, send email to
>>>> google-appengine-java@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> google-appengine-java+unsubscr...@googlegroups.com.
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-appengine-java?hl=en.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine for Java" group.
>>> To post to this group, send email to google-appengine-java@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine-java+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine-java?hl=en.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine for Java" group.
>> To post to this group, send email to google-appengine-java@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine-java+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine-java?hl=en.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to google-appengine-java@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine-java+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
