Hmm. The one on code.google.com seems to be out of date. The version
shipping with the SDK has:
private final transient
QueryResultIteratorImplhttps://cs.corp.google.com/#google3/java/com/google/appengine/api/datastore/QueryResultIteratorImpl.javact=xref_jump_to_defl=27
Ah, so I figured out what is going on with this. This is the code that is
running in production, the SDK does not have a serializable version of
LazyList as we don't explicitly support it through the interface. 1.5.1 will
have these changes in the SDK though.
On Fri, Jun 10, 2011 at 9:44 AM,
It does uses a lazy list to do asynchronous prefetching:
http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/LazyList.java
On Tue, Jun 7, 2011 at 3:19 AM, Anders blabl...@gmail.com wrote:
I doubt that the difference can be that large.
If you are going to just iterate through a list with out doing any work,
fetching everything up front is always going to be faster. However, we
expect that you are going to be doing something with the entities you fetch.
The lazy list tries to hid the cost of fetching the entities by doing it
I modified the samples to use Iterator for a fair benchmark:
public IteratorEntity getBarsUsingLL() {
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
return
It's still producing *wild* variance when I click those buttons. By a
factor of 2.
If you want an accurate benchmark:
1) You need more iterations. 5 is not enough. At a minimum I would
say 10. If this is too long for a single button click, you can halve
the # of entities fetched at once -
Interesting - in retrospect, of course it isn't realloc overhead.
There would be at most 9 reallocs, and there's no way that could take
hundreds of milliseconds.
I'm not sure that this optimization of Python will be as effective in
Javaland. The pattern of fetching is totally different in Python
On Thu, Jun 9, 2011 at 2:56 PM, Jeff Schnitzer j...@infohazard.org wrote:
Interesting - in retrospect, of course it isn't realloc overhead.
There would be at most 9 reallocs, and there's no way that could take
hundreds of milliseconds.
I'm not sure that this optimization of Python will be as
On Thu, Jun 9, 2011 at 3:30 PM, Alfred Fuller
arfuller+appeng...@google.com wrote:
Funny because we actually don't have this feature in python (only iterators
async prefetch). Ya, I would hope coders would use asIterable() when doing a
single for loop. The real win for asList() async prefetch
H. I just whipped up a test app that uses a mock
AsyncDatastoreService to provide a set of 10,000 Entity objects to
Objectify, thus purely measuring the overhead of Objectify. It
consistently transforms 10,000 entities into POJOs (just long id,
String value) in 100ms on both my laptop and on
Ok, I have figured out what's going on. To put it gently, the
benchmark is bogus.
The benchmark code for the Low-Level API just gets the size() of the
collection and doesn't actually examine the data. GAE (apparently)
lazily populates Entity objects, so the benchmark cuts out a
significant
I doubt that the difference can be that large. The performance test code
uses the low-level PreparedQuery#asList call. The question is if the list
(ListEntity) contains entities loaded with data or if the list returned
has a lazy loading implementation so that the actual data from the the
I doubt that the difference can be that large. The performance test code
uses the low-level PreparedQuery#asList call. The question is if the list
(ListEntity) contains entities loaded with data or if the list returned
has a lazy loading implementation so that the actual data from the datastore
I don't know much about Slim3 but I just migrated from JDO to the
native API, and I can say that the performance improvement was in many
cases dramatic. I was assured many times that JDO's overhead is
insignificant, but that hasn't been my experience. I chalk it up to
JDO's reliance on
I started by looking at JDO but after founding some info about potential
performance loss I wrote a simple wrapper abstract base class around a
low-level Entity object and then extended the base class into the data
classes I needed. Super light weight. :-)
--
You received this message because
I started by looking at JDO but after finding some info about potential
performance loss I wrote a simple wrapper abstract base class around a
low-level Entity object and then extended the base class into the data
classes I needed. Super light weight. :-)
--
You received this message because
LL API uses LazyList, so PreparedQuery#asList is very fast.
I modified the test code as follows:
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
That makes more sense, thanks.
I also found this online benchmark of JDO and LL, which has similar results:
http://gaejava.appspot.com/
On Tue, Jun 7, 2011 at 8:03 PM, Anders blabl...@gmail.com wrote:
Yeah, that's what I suspected. Lazy loading. With the modification Slim3 is
almost as fast
Have you actually run a profiler against datanucleus? There are a
million reasons why a piece of software might be slow. Considering
the large number of logical operations involved in translating an
Entity into a POJO, I doubt very much the issue is that a couple calls
are via reflection. It's
I doubt the low-level API is significantly faster than JDO (have not
profiled, so can't tell you for sure). JDO just dispatches to low-level and
does serialization/deserialization. That should really be a very small part
of the entire operation.
Reasons to use the low-level API include:
- it
I added an objectify sample into the performance samples:
http://slim3demo.appspot.com/performance/
One result:
The number of entities: 1
low-level API:get: 1276 millis
Slim3: 1327 millis
Objectify: 3028 millis
JDO: 3222 millis
I have not profiled yet.
But I know java runtime reflections are
21 matches
Mail list logo
