Re: [google-appengine] Re: Upload works locally but not on server.
Hi Brian,
There is a blobstore sample using slim3:
http://slim3demo.appspot.com/blobstore/
Yasuo Higa
On Fri, Aug 26, 2011 at 9:37 PM, Brian brian...@gmail.com wrote:
Adam --
How were you able to modify the post header? I'm experiencing the
same issue and am using the slim3 mvc framework.
Thanks,
Brian
On Jul 14, 10:03 pm, adam adus...@gmail.com wrote:
I was able to solve this. The parser in production GAE is more
sensitive than the one in the development environment, and the one in
production provides no useful information to the user about why it
fails. This is a known
issue:http://code.google.com/p/googleappengine/issues/detail?id=3273
for the bug report.
The reason the upload was failing on the server was because one of the
form fields was missing a \r\n:
--jtmyobpmsbvkwefmrarmjuescosjfuxm
Content-Disposition: form-data; name=recording_time
3
is correct, whereas
--jtmyobpmsbvkwefmrarmjuescosjfuxm
Content-Disposition: form-data; name=recording_time
3
is not. The latter succeeds in the development environment, whereas it
fails in production GAE. It took a couple hours of debugging to get to
this conclusion -- it would be nice if GAE blobstore reported POST
parsing errors in some way.
Adam
On Jul 14, 12:54 pm, adam adus...@gmail.com wrote:
Just a bit more data: it's clear to me that the upload URL isn't being
caught by the WSGIApplication on the server. I know this because when
I insert a catch-all url path pattern, the upload POST request hits
that handler instead of the BlobstoreUploadHandler. My WSGIApplication
is instantiated like this:
application = webapp.WSGIApplication(
[
('/', MainPage),
('/upload', UploadHandler),
('/upload_key/([^/]+)', UploadKeyHandler)],
debug=True)
Again, POST requests to URLs generated by
blobstore.create_upload_url('/upload') are not hitting UploadHandler.
Thanks again,
Adam
On Jul 14, 12:50 pm, adam adus...@gmail.com wrote:
Hi, I have an app that allows me upload to the blobstore locally but
fails on the server. I get an upload url using
blobstore.create_upload_url('/upload'), just like the example provided
by google.
When running the app locally, this works. However, when I run it on
GAE, I get a 503; seehttp://grab.by/axFo.
In case it helps the engineers to debug, here's the URL:
http://www.recordmp3.org/_ah/upload/AMmfu6bsVT68mSvZqDM85D324uLxM39c-...
Thank you,
Adam
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
It's *NO* bogus benchmark because the sample iterates the results.
http://code.google.com/p/slim3/source/browse/trunk/slim3demo/src/slim3/demo/controller/performance/GetLLController.java
Yasuo Higa
The code linked from http://slim3demo.appspot.com/performance/ has
just gone back to calling list.size() without iterating the results,
which once again goes back to a bogus benchmark.
Jeff
On Wed, Jun 8, 2011 at 9:30 PM, Yasuo Higa higaya...@gmail.com wrote:
Slim3 uses LL API.
To resolve a strange issue that slim3 is faster than LL, I tried the
following samples:
One:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
Two:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
// VERY IMPORTANT
list.size();
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
The second one is much faster than the first one.
I fixed the samples to call list.size().
http://slim3demo.appspot.com/performance/
As a result, LL is as fast as slim3 (^^;
Yasuo Higa
On Thu, Jun 9, 2011 at 10:17 AM, Jeff Schnitzer j...@infohazard.org wrote:
Thank you for fixing the benchmark.
I am very curious. According to this new benchmark - it's hard to
tell without pushing the buttons a lot of times, but there seems to be
a trend - Slim3 is somewhat faster than the Low Level API.
Doesn't Slim3 use the Low Level API underneath? How can it possibly be
faster?
Jeff
On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa higaya...@gmail.com wrote:
What I want to provide is a fair and casual benchmark.
As jeff advised, I modified samples as follows:
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
for (Bar bar : service.getBarListUsingSlim3()) {
bar.getKey();
bar.getSortValue();
}
for (BarObjectify bar : service.getBarListUsingObjectify()) {
bar.getKey();
bar.getSortValue();
}
for (BarJDO bar : service.getBarListUsingJDO()) {
bar.getKey();
bar.getSortValue();
}
LL API is much slower than before.
http://slim3demo.appspot.com/performance/
Yasuo Higa
On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer j...@infohazard.org wrote:
Slim3 may be a nice piece of software, but it has not been
demonstrated to be faster than anything (including JDO). It might or
might not be faster - I don't know - but based on the sloppy
benchmarking, I'm pretty certain that the people making this claim
don't know either.
There's another ill-conceived performance claim on the Slim3 website:
You may worry about the overhead of global transactions. Don't worry.
It is not very expensive. There are three problems with their little
performance test:
1) It only measures wall-clock time, not cost.
2) It does not measure what happens under contention.
3) The numbers are obviously wrong - they don't even pass a smoke test.
Look at these numbers (from the Slim3 home page):
Entity Groups Local Transaction(millis) Global Transaction(millis)
1 67 70
2 450 415
3 213 539
4 1498 981
5 447 781
Just like the 1ms low-level API query performance in the benchmark
that spawned this thread, even a casual observer should be able to
recognize the obvious flaw - the numbers don't show any expected
relationship between # of entity groups or the use of global
transactions. Interpreted literally, you would assume that local
transactions are much faster for 5 entity groups, but global
transactions are much faster for 4 entity groups.
It's pretty obvious that the benchmark author just ran one pass and
took the numbers as-is. If you actually run multiple passes, you'll
find that there is enormous variance in the timing. The only way you
can realistically measure results like this on appengine is to execute
the test 100 times and take a median.
FWIW, I deliberately haven't made any performance claims for Objectify
because I haven't put the necessary amount of due diligence into
creating a proper set of benchmarks. It is not easy to measure
performance, especially in a dynamic environment like appengine. I
only hope that the Slim3 authors have put more care and thought into
crafting their library than they have their benchmarks.
Jeff
On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber gal.dol...@gmail.com wrote:
Slim3 is not only fast, the api is completely awesome. It has been my
choice
for a year now for all gae projects.
It includes name safety
Re: [google-appengine] Re: Is the native API really so much faster than JDO and slim3?
I modified the samples to use Iterator for a fair benchmark:
public IteratorEntity getBarsUsingLL() {
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
return pq.asIterator(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
}
public IteratorBar getBarsUsingSlim3() {
return Datastore.query(Bar.class).asIterator();
}
public IteratorBarObjectify getBarsUsingObjectify() {
Objectify ofy = ObjectifyService.begin();
return ofy.query(BarObjectify.class).iterator();
}
@SuppressWarnings(unchecked)
public ListBarJDO getBarsUsingJDO() {
ListBarJDO list = null;
PersistenceManager pm = PMF.get().getPersistenceManager();
try {
javax.jdo.Query q = pm.newQuery(BarJDO.class);
list = (ListBarJDO) q.execute();
list = (ListBarJDO) pm.detachCopyAll(list);
} finally {
pm.close();
}
return list;
}
private static final int COUNT = 5;
@Override
public Navigation run() throws Exception {
long start = System.currentTimeMillis();
for (int i = 0; i COUNT; i++) {
for (IteratorEntity ite = service.getBarsUsingLL();
ite.hasNext();) {
Entity e = ite.next();
e.getKey();
e.getProperty(sortValue);
}
}
sessionScope(getLL, (System.currentTimeMillis() - start) / COUNT);
return redirect(basePath);
}
@Override
public Navigation run() throws Exception {
long start = System.currentTimeMillis();
for (int i = 0; i COUNT; i++) {
for (IteratorBar ite = service.getBarsUsingSlim3();
ite.hasNext();) {
Bar bar = ite.next();
bar.getKey();
bar.getSortValue();
}
}
sessionScope(getSlim3, (System.currentTimeMillis() - start) / COUNT);
return redirect(basePath);
}
@Override
public Navigation run() throws Exception {
long start = System.currentTimeMillis();
for (int i = 0; i COUNT; i++) {
for (IteratorBarObjectify ite =
service.getBarsUsingObjectify(); ite
.hasNext();) {
BarObjectify bar = ite.next();
bar.getKey();
bar.getSortValue();
}
}
sessionScope(getObjectify, (System.currentTimeMillis() - start)
/ COUNT);
return redirect(basePath);
}
@Override
public Navigation run() throws Exception {
long start = System.currentTimeMillis();
for (int i = 0; i COUNT; i++) {
for (BarJDO bar : service.getBarsUsingJDO()) {
bar.getKey();
bar.getSortValue();
}
}
sessionScope(getJDO, (System.currentTimeMillis() - start) / COUNT);
return redirect(basePath);
}
http://slim3demo.appspot.com/performance/
Slim3 is slightly faster than frameworks that use reflection,
but the difference is small.
Yasuo Higa
On Fri, Jun 10, 2011 at 2:29 AM, Alfred Fuller
arfuller+appeng...@google.com wrote:
If you are going to just iterate through a list with out doing any work,
fetching everything up front is always going to be faster. However, we
expect that you are going to be doing something with the entities you fetch.
The lazy list tries to hid the cost of fetching the entities by doing it
asynchronously while you are 'processing' the previous batch of entities. In
my experiments (in Python) where work was actually being done (namely
thread.sleep(X)) it gave a 10-15% speed up (depending on how much work you
are doing, as the fetch time is a constant value).
I would not have expected the difference to be this significant in Java
though. We never know the # of results ahead of time so the difference
couldn't be related to growing the List organically (as it always grows
organically).
One thing that should be noted, a wrapper that proactively converts entities
(instead of doing the conversion on demand) will negate any benefit from
async prefetching. Also a realistic benchmark should tak into account that
some amount of work will be done on the entities fetched.
On Thu, Jun 9, 2011 at 9:51 AM, Alfred Fuller
arfuller+appeng...@google.com wrote:
It does uses a lazy list to do asynchronous prefetching:
http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/LazyList.java
On Tue, Jun 7, 2011 at 3:19 AM, Anders blabl...@gmail.com wrote:
I doubt that the difference can be that large. The performance test code
uses the low-level PreparedQuery#asList call. The question is if the list
(ListEntity) contains entities loaded with data or if the list returned
has a lazy loading implementation so
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
It is not bogus.
LazyList#size() fetches all data as follows:
public int size() {
resolveAllData();
return results.size();
}
Yasuo Higa
On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson
dennisbpeter...@gmail.com wrote:
It's not my benchmark, it's Slim3's :) ...but you're right, it's bogus. I
asked on the main appengine group too, and it turns out the low-level
benchmark is doing lazy loading. With that fixed, their numbers come out
like yours.
I found this one too, which also gets results like yours:
http://gaejava.appspot.com/
On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur erwin.str...@gmail.com wrote:
Indeed Dennis's measurements are very suspicious. First you should do
a couple of warming ups on each of the implementations to prevent
pollution like the JDO classpath scan for enhanced classes (which is
one of the reasons for the high initial run). Then do a couple of run
to determine a range of measurements to spot outlyers. your low-level
API 2millis is definately one.
When I did the measurements I got the following results
low-level: 1150-1550
Slim3: 1150-1600
Objectify: 1950-2400
JDO: 2100-2700
These measurements confirm that GAE designed implementations are
faster then the GAE implementation of a generic data access layer
(JDO), but not so extrem as initially posted.
The initial response using JDO is a known issue and especially low
trafic website should not use it or use the always on feature (maybe
this will change in the new pricing model)
Regards,
Erwin
On Jun 7, 11:00 am, Ian Marshall ianmarshall...@gmail.com wrote:
The low-level API does indeed look very fast.
Just a comment on JDO: repeat runs roughly halve the JDO run time. I
presume that this is because for repeat runs the JDO persistence
manager factory has already been constructed.
On Jun 6, 8:44 pm, DennisP dennisbpeter...@gmail.com wrote:
I'm looking at this online
demo:http://slim3demo.appspot.com/performance/
Sample run:
The number of entities: 1
low-level API:get: 2 millis
Slim3: 2490 millis
JDO: 6030 millis
Is the low-level API really that much faster?
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
Hi Dennis,
You can see all sources.
http://slim3demo.appspot.com/performance/
Java runtime reflections are very very slow.
If you don't think so, please try it by you.
Yasuo Higa
On Thu, Jun 9, 2011 at 12:00 AM, Dennis Peterson
dennisbpeter...@gmail.com wrote:
Apologies, no offense meant. My impression was that if you wanted to, say,
display all that data, it's going to take around 1000 ms to get it, not 1
ms.
On Wed, Jun 8, 2011 at 10:55 AM, Yasuo Higa higaya...@gmail.com wrote:
It is not bogus.
LazyList#size() fetches all data as follows:
public int size() {
resolveAllData();
return results.size();
}
Yasuo Higa
On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson
dennisbpeter...@gmail.com wrote:
It's not my benchmark, it's Slim3's :) ...but you're right, it's bogus.
I
asked on the main appengine group too, and it turns out the low-level
benchmark is doing lazy loading. With that fixed, their numbers come out
like yours.
I found this one too, which also gets results like yours:
http://gaejava.appspot.com/
On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur erwin.str...@gmail.com
wrote:
Indeed Dennis's measurements are very suspicious. First you should do
a couple of warming ups on each of the implementations to prevent
pollution like the JDO classpath scan for enhanced classes (which is
one of the reasons for the high initial run). Then do a couple of run
to determine a range of measurements to spot outlyers. your low-level
API 2millis is definately one.
When I did the measurements I got the following results
low-level: 1150-1550
Slim3: 1150-1600
Objectify: 1950-2400
JDO: 2100-2700
These measurements confirm that GAE designed implementations are
faster then the GAE implementation of a generic data access layer
(JDO), but not so extrem as initially posted.
The initial response using JDO is a known issue and especially low
trafic website should not use it or use the always on feature (maybe
this will change in the new pricing model)
Regards,
Erwin
On Jun 7, 11:00 am, Ian Marshall ianmarshall...@gmail.com wrote:
The low-level API does indeed look very fast.
Just a comment on JDO: repeat runs roughly halve the JDO run time. I
presume that this is because for repeat runs the JDO persistence
manager factory has already been constructed.
On Jun 6, 8:44 pm, DennisP dennisbpeter...@gmail.com wrote:
I'm looking at this online
demo:http://slim3demo.appspot.com/performance/
Sample run:
The number of entities: 1
low-level API:get: 2 millis
Slim3: 2490 millis
JDO: 6030 millis
Is the low-level API really that much faster?
--
You received this message because you are subscribed to the Google
Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google
Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
Hi Dennis, The following document will help you about global transactions: http://sites.google.com/site/slim3appengine/#gtx Yasuo Higa On Thu, Jun 9, 2011 at 12:33 AM, Dennis Peterson dennisbpeter...@gmail.com wrote: Those multi-entity transactions are definitely interesting to me. There's some overhead but no getting around that. A while back I was playing around with some adhoc methods to do it in a specific case, but I suspect Slim3 is more solid and maybe faster than what I was doing. Definitely easier. When I get a chance I want to dig in and find out how it works. On Wed, Jun 8, 2011 at 10:26 AM, Mike Lawrence m...@systemsplanet.com wrote: I get... The number of entities: 1 low level 1717 millis slim3 1502 millis objectify 2970 millis jdo 3485 millis probably should modify this example to do an average of several runs one important thing to note, is slim3 allows you to update multiple entity types in a single transaction. not possible with the other 3 APIs On Jun 7, 4:00 am, Ian Marshall ianmarshall...@gmail.com wrote: The low-level API does indeed look very fast. Just a comment on JDO: repeat runs roughly halve the JDO run time. I presume that this is because for repeat runs the JDO persistence manager factory has already been constructed. On Jun 6, 8:44 pm, DennisP dennisbpeter...@gmail.com wrote: I'm looking at this online demo:http://slim3demo.appspot.com/performance/ Sample run: The number of entities: 1 low-level API:get: 2 millis Slim3: 2490 millis JDO: 6030 millis Is the low-level API really that much faster? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
What I want to provide is a fair and casual benchmark.
As jeff advised, I modified samples as follows:
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
for (Bar bar : service.getBarListUsingSlim3()) {
bar.getKey();
bar.getSortValue();
}
for (BarObjectify bar : service.getBarListUsingObjectify()) {
bar.getKey();
bar.getSortValue();
}
for (BarJDO bar : service.getBarListUsingJDO()) {
bar.getKey();
bar.getSortValue();
}
LL API is much slower than before.
http://slim3demo.appspot.com/performance/
Yasuo Higa
On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer j...@infohazard.org wrote:
Slim3 may be a nice piece of software, but it has not been
demonstrated to be faster than anything (including JDO). It might or
might not be faster - I don't know - but based on the sloppy
benchmarking, I'm pretty certain that the people making this claim
don't know either.
There's another ill-conceived performance claim on the Slim3 website:
You may worry about the overhead of global transactions. Don't worry.
It is not very expensive. There are three problems with their little
performance test:
1) It only measures wall-clock time, not cost.
2) It does not measure what happens under contention.
3) The numbers are obviously wrong - they don't even pass a smoke test.
Look at these numbers (from the Slim3 home page):
Entity Groups Local Transaction(millis) Global Transaction(millis)
1 67 70
2 450 415
3 213 539
4 1498 981
5 447 781
Just like the 1ms low-level API query performance in the benchmark
that spawned this thread, even a casual observer should be able to
recognize the obvious flaw - the numbers don't show any expected
relationship between # of entity groups or the use of global
transactions. Interpreted literally, you would assume that local
transactions are much faster for 5 entity groups, but global
transactions are much faster for 4 entity groups.
It's pretty obvious that the benchmark author just ran one pass and
took the numbers as-is. If you actually run multiple passes, you'll
find that there is enormous variance in the timing. The only way you
can realistically measure results like this on appengine is to execute
the test 100 times and take a median.
FWIW, I deliberately haven't made any performance claims for Objectify
because I haven't put the necessary amount of due diligence into
creating a proper set of benchmarks. It is not easy to measure
performance, especially in a dynamic environment like appengine. I
only hope that the Slim3 authors have put more care and thought into
crafting their library than they have their benchmarks.
Jeff
On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber gal.dol...@gmail.com wrote:
Slim3 is not only fast, the api is completely awesome. It has been my choice
for a year now for all gae projects.
It includes name safety and and amazing querying utils.
Very recommendable!
On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer j...@infohazard.org wrote:
You are wrong.
Try adding getProperty() calls to your LL performance test, and the
speed advantage of the LL API goes away. I don't know what to say
about Slim3, but here's my test case:
http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test
I created 10,000 entities in the datastore that have the same format
as your test case - a single string property. Here's what happens
(try it - and remember to reload the urls several times to get a
realistic median value):
## Low Level API with just .size()
http://voodoodyne.appspot.com/fetchLLSize
The code:
ListEntity things =
DatastoreServiceFactory.getDatastoreService()
.prepare(new
Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
.asList(FetchOptions.Builder.withDefaults());
things.size();
Note that results are almost always under 2000ms. Wild guess I'd say
the median elapsed is ~1900, just like your example.
## Low Level API with actual fetch of the data
http://voodoodyne.appspot.com/fetchLL
The code:
ListEntity things =
DatastoreServiceFactory.getDatastoreService()
.prepare(new
Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
.asList(FetchOptions.Builder.withDefaults());
for (Entity ent: things)
{
ent.getKey();
ent.getProperty(value);
}
Note that the duration is now considerably longer. Eyeballing the
median elapsed time, I'd say somewhere around 3000ms.
## Objectify fetching from datastore
http://voodoodyne.appspot.com/fetch
Objectify ofy = ObjectifyService.begin();
ListThing things = ofy.query(Thing.class).list();
for (Thing thing: things)
{
thing.getId();
thing.getValue();
}
Note that the timing is pretty much the same as the LL API when it
includes
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
Slim3 uses LL API.
To resolve a strange issue that slim3 is faster than LL, I tried the
following samples:
One:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
Two:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
// VERY IMPORTANT
list.size();
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
The second one is much faster than the first one.
I fixed the samples to call list.size().
http://slim3demo.appspot.com/performance/
As a result, LL is as fast as slim3 (^^;
Yasuo Higa
On Thu, Jun 9, 2011 at 10:17 AM, Jeff Schnitzer j...@infohazard.org wrote:
Thank you for fixing the benchmark.
I am very curious. According to this new benchmark - it's hard to
tell without pushing the buttons a lot of times, but there seems to be
a trend - Slim3 is somewhat faster than the Low Level API.
Doesn't Slim3 use the Low Level API underneath? How can it possibly be
faster?
Jeff
On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa higaya...@gmail.com wrote:
What I want to provide is a fair and casual benchmark.
As jeff advised, I modified samples as follows:
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
for (Bar bar : service.getBarListUsingSlim3()) {
bar.getKey();
bar.getSortValue();
}
for (BarObjectify bar : service.getBarListUsingObjectify()) {
bar.getKey();
bar.getSortValue();
}
for (BarJDO bar : service.getBarListUsingJDO()) {
bar.getKey();
bar.getSortValue();
}
LL API is much slower than before.
http://slim3demo.appspot.com/performance/
Yasuo Higa
On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer j...@infohazard.org wrote:
Slim3 may be a nice piece of software, but it has not been
demonstrated to be faster than anything (including JDO). It might or
might not be faster - I don't know - but based on the sloppy
benchmarking, I'm pretty certain that the people making this claim
don't know either.
There's another ill-conceived performance claim on the Slim3 website:
You may worry about the overhead of global transactions. Don't worry.
It is not very expensive. There are three problems with their little
performance test:
1) It only measures wall-clock time, not cost.
2) It does not measure what happens under contention.
3) The numbers are obviously wrong - they don't even pass a smoke test.
Look at these numbers (from the Slim3 home page):
Entity Groups Local Transaction(millis) Global Transaction(millis)
1 67 70
2 450 415
3 213 539
4 1498 981
5 447 781
Just like the 1ms low-level API query performance in the benchmark
that spawned this thread, even a casual observer should be able to
recognize the obvious flaw - the numbers don't show any expected
relationship between # of entity groups or the use of global
transactions. Interpreted literally, you would assume that local
transactions are much faster for 5 entity groups, but global
transactions are much faster for 4 entity groups.
It's pretty obvious that the benchmark author just ran one pass and
took the numbers as-is. If you actually run multiple passes, you'll
find that there is enormous variance in the timing. The only way you
can realistically measure results like this on appengine is to execute
the test 100 times and take a median.
FWIW, I deliberately haven't made any performance claims for Objectify
because I haven't put the necessary amount of due diligence into
creating a proper set of benchmarks. It is not easy to measure
performance, especially in a dynamic environment like appengine. I
only hope that the Slim3 authors have put more care and thought into
crafting their library than they have their benchmarks.
Jeff
On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber gal.dol...@gmail.com wrote:
Slim3 is not only fast, the api is completely awesome. It has been my
choice
for a year now for all gae projects.
It includes name safety and and amazing querying utils.
Very recommendable!
On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer j...@infohazard.org wrote:
You are wrong.
Try adding getProperty() calls to your LL performance test, and the
speed advantage of the LL API goes away. I don't know what to say
about Slim3, but here's my test case:
http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test
I created 10,000 entities in the datastore that have the same format
Re: [google-appengine] Is the native API really so much faster than JDO and slim3?
No. Slim3 loads all data eagerly. Yasuo Higa On Wednesday, June 8, 2011, Jeff Schnitzer j...@infohazard.org wrote: Ok, I have figured out what's going on. To put it gently, the benchmark is bogus. The benchmark code for the Low-Level API just gets the size() of the collection and doesn't actually examine the data. GAE (apparently) lazily populates Entity objects, so the benchmark cuts out a significant chunk of the work. Objectify eagerly examines the Entity data to create POJOs - but your real-world app is going to fetch the data from the Entity too. Presumably Slim3 lazily passes through to the low-level size() method. This is unrealistic; if you wanted a count, you would use the query's count() method. When you iterate the result set and include an Entity.getProperty() call, Objectify's 100ms overhead fades into the natural (+/- 500ms) variance. Sloppy work. Jeff On Wed, Jun 8, 2011 at 1:44 AM, Jeff Schnitzer j...@infohazard.org wrote: H. I just whipped up a test app that uses a mock AsyncDatastoreService to provide a set of 10,000 Entity objects to Objectify, thus purely measuring the overhead of Objectify. It consistently transforms 10,000 entities into POJOs (just long id, String value) in 100ms on both my laptop and on the production servers. I'm not quite sure what's going on here, but it's not an issue with reflection. Jeff On Tue, Jun 7, 2011 at 8:51 PM, Yasuo Higa higaya...@gmail.com wrote: I added an objectify sample into the performance samples: http://slim3demo.appspot.com/performance/ One result: The number of entities: 1 low-level API:get: 1276 millis Slim3: 1327 millis Objectify: 3028 millis JDO: 3222 millis I have not profiled yet. But I know java runtime reflections are very very slow on production server, so slim3 creates the mapping logic between a model and an entity as the source code when compiling. Yasuo Higa On Wed, Jun 8, 2011 at 10:18 AM, Ikai Lan (Google) ika...@google.com wrote: I doubt the low-level API is significantly faster than JDO (have not profiled, so can't tell you for sure). JDO just dispatches to low-level and does serialization/deserialization. That should really be a very small part of the entire operation. Reasons to use the low-level API include: - it maps best to how the datastore works (schemaless) - you always get new features the fastest (async datastore API) That being said, the Slim3 and Objectify projects move pretty quickly and add features almost as quickly as we do. They're one level removed from the schemaless nature of the datastore, but this fits better with the 95% use case. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Wed, Jun 8, 2011 at 8:07 AM, Dennis Peterson dennisbpeter...@gmail.com wrote: That makes more sense, thanks. I also found this online benchmark of JDO and LL, which has similar results: http://gaejava.appspot.com/ On Tue, Jun 7, 2011 at 8:03 PM, Anders blabl...@gmail.com wrote: Yeah, that's what I suspected. Lazy loading. With the modification Slim3 is almost as fast as the native API. Strange that JDO is so slow. I thought most of the time was for accessing the datastore, not for running the Java bytecode. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/dHRvXzBkWVNkUndK. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Is the native API really so much faster than JDO and slim3?
No. Slim3 loads all data eagerly. Yasuo Higa On Wednesday, June 8, 2011, Jeff Schnitzer j...@infohazard.org wrote: Ok, I have figured out what's going on. To put it gently, the benchmark is bogus. The benchmark code for the Low-Level API just gets the size() of the collection and doesn't actually examine the data. GAE (apparently) lazily populates Entity objects, so the benchmark cuts out a significant chunk of the work. Objectify eagerly examines the Entity data to create POJOs - but your real-world app is going to fetch the data from the Entity too. Presumably Slim3 lazily passes through to the low-level size() method. This is unrealistic; if you wanted a count, you would use the query's count() method. When you iterate the result set and include an Entity.getProperty() call, Objectify's 100ms overhead fades into the natural (+/- 500ms) variance. Sloppy work. Jeff On Wed, Jun 8, 2011 at 1:44 AM, Jeff Schnitzer j...@infohazard.org wrote: H. I just whipped up a test app that uses a mock AsyncDatastoreService to provide a set of 10,000 Entity objects to Objectify, thus purely measuring the overhead of Objectify. It consistently transforms 10,000 entities into POJOs (just long id, String value) in 100ms on both my laptop and on the production servers. I'm not quite sure what's going on here, but it's not an issue with reflection. Jeff On Tue, Jun 7, 2011 at 8:51 PM, Yasuo Higa higaya...@gmail.com wrote: I added an objectify sample into the performance samples: http://slim3demo.appspot.com/performance/ One result: The number of entities: 1 low-level API:get: 1276 millis Slim3: 1327 millis Objectify: 3028 millis JDO: 3222 millis I have not profiled yet. But I know java runtime reflections are very very slow on production server, so slim3 creates the mapping logic between a model and an entity as the source code when compiling. Yasuo Higa On Wed, Jun 8, 2011 at 10:18 AM, Ikai Lan (Google) ika...@google.com wrote: I doubt the low-level API is significantly faster than JDO (have not profiled, so can't tell you for sure). JDO just dispatches to low-level and does serialization/deserialization. That should really be a very small part of the entire operation. Reasons to use the low-level API include: - it maps best to how the datastore works (schemaless) - you always get new features the fastest (async datastore API) That being said, the Slim3 and Objectify projects move pretty quickly and add features almost as quickly as we do. They're one level removed from the schemaless nature of the datastore, but this fits better with the 95% use case. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Wed, Jun 8, 2011 at 8:07 AM, Dennis Peterson dennisbpeter...@gmail.com wrote: That makes more sense, thanks. I also found this online benchmark of JDO and LL, which has similar results: http://gaejava.appspot.com/ On Tue, Jun 7, 2011 at 8:03 PM, Anders blabl...@gmail.com wrote: Yeah, that's what I suspected. Lazy loading. With the modification Slim3 is almost as fast as the native API. Strange that JDO is so slow. I thought most of the time was for accessing the datastore, not for running the Java bytecode. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/dHRvXzBkWVNkUndK. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Is the native API really so much faster than JDO and slim3?
LL API uses LazyList, so PreparedQuery#asList is very fast. I modified the test code as follows: DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Query q = new Query(Bar); PreparedQuery pq = ds.prepare(q); ListEntity list = pq.asList(FetchOptions.Builder.withOffset(0).limit( Integer.MAX_VALUE)); list.size(); return list; The result: The number of entities: 1 low-level API:get: 1199 millis Slim3: 1238 millis JDO: 3298 millis Slim3 does not need runtime reflection for mapping between a entity and a model. The mapping logic is automatically created as the source code when compiling. so the speed of Slim3 is almost equal to LL API. Yasuo Higa On Tue, Jun 7, 2011 at 8:27 PM, Anders blabl...@gmail.com wrote: I started by looking at JDO but after finding some info about potential performance loss I wrote a simple wrapper abstract base class around a low-level Entity object and then extended the base class into the data classes I needed. Super light weight. :-) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/XzF3Q3MxcFVvbGdK. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Is the native API really so much faster than JDO and slim3?
I added an objectify sample into the performance samples: http://slim3demo.appspot.com/performance/ One result: The number of entities: 1 low-level API:get: 1276 millis Slim3: 1327 millis Objectify: 3028 millis JDO: 3222 millis I have not profiled yet. But I know java runtime reflections are very very slow on production server, so slim3 creates the mapping logic between a model and an entity as the source code when compiling. Yasuo Higa On Wed, Jun 8, 2011 at 10:18 AM, Ikai Lan (Google) ika...@google.com wrote: I doubt the low-level API is significantly faster than JDO (have not profiled, so can't tell you for sure). JDO just dispatches to low-level and does serialization/deserialization. That should really be a very small part of the entire operation. Reasons to use the low-level API include: - it maps best to how the datastore works (schemaless) - you always get new features the fastest (async datastore API) That being said, the Slim3 and Objectify projects move pretty quickly and add features almost as quickly as we do. They're one level removed from the schemaless nature of the datastore, but this fits better with the 95% use case. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Wed, Jun 8, 2011 at 8:07 AM, Dennis Peterson dennisbpeter...@gmail.com wrote: That makes more sense, thanks. I also found this online benchmark of JDO and LL, which has similar results: http://gaejava.appspot.com/ On Tue, Jun 7, 2011 at 8:03 PM, Anders blabl...@gmail.com wrote: Yeah, that's what I suspected. Lazy loading. With the modification Slim3 is almost as fast as the native API. Strange that JDO is so slow. I thought most of the time was for accessing the datastore, not for running the Java bytecode. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/dHRvXzBkWVNkUndK. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] How do I run my Junit test suite on the server?
Hi Elio, See Kotori Web JUnit Runner: http://code.google.com/p/ktrwjr/ On Fri, Apr 1, 2011 at 12:28 AM, Eliot Stock 1...@eliotstock.com wrote: Is there a standard way to do this or do I have to write a servlet or something? If I have to write something, how am I going to get the log output from the tests onto the UI or the logs in the dashboard? Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Can I set GAE's dev_appserver to automatically reload context when I change .class files?
Hi Luca, Slim3 supports HOT reloading. https://sites.google.com/site/slim3appengine/ Yasuo Higa On Thu, Mar 10, 2011 at 7:58 PM, Luca Matteis lmatt...@gmail.com wrote: Hello everybody! I'm using Google AppEngine with their built in web server. My development goes about in a simple way: I make changes to my .java sources or .jsp and compile using ant and to see the changes I have to restart the development server. I'm wondering if there's a way I can avoid this last step of restarting my development server - somehow refresh the cached classes context of my web-server. The options provided by Google on this dev server are quite limited and am wondering if there's a better way. I would like to avoid using something like JRebel which I could buy, but for this simple project I'm just wondering if I can remove the burden of restarting my web-server... otherwise I'll live with it. Thanks. Luca -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] What was your Slim3 Datastore API experience like?
Hi systemsplanet, - Having to run the Annotation pre-processor as part of the IDE (will that work with maven-command line?) Yes. See http://sites.google.com/site/slim3appengine/maven - Global transaction performance. The Slim3 site shows much faster times than I'm seeing http://sites.google.com/site/slim3appengine/ The performance depends on app engine condition. - What happens when a global transaction fails? Can it be detected when it occurs so the user can be notified? If a global transaction fails before commit, it will be rolled back. If a global transaction fails after commit, it will be rolled forward by task queue. - Is it using undocumented Google APIs? No. Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [google-appengine] AppEngine entity modeling - minimizing entity groups + achieving atomic cascading
Hi Hari, I understood that a. To achieve atomicity of update/delete of several entities we need to do it in a transaction and hence all should fall under same entity group b. Having big entity groups is not scalable as it causes contention. (Q1: Correct?) Correct, but slim3, which is a java framework, supports global transactions between multiple entity groups. http://sites.google.com/site/slim3appengine/#gtx You may worry about the overhead of global transactions. Don't worry. It is not very expensive. The demonstration is as follows: http://slim3demo.appspot.com/gtx/ Yasuo Higa On Fri, Dec 3, 2010 at 7:53 PM, har_shan harsha...@gmail.com wrote: Hello, Am learning AppEngine and have started developing new app and want to clarify something. I understood that a. To achieve atomicity of update/delete of several entities we need to do it in a transaction and hence all should fall under same entity group b. Having big entity groups is not scalable as it causes contention. (Q1: Correct?) So here is an entity model of an online examination system for sake of discussion: Entities: Subject Exam Page Question Answer As you can see from top, each entity 1 - many relationship with the immediate bottom one i.e 1 Subject can have many exams, 1 exam - many pages, 1 page can have many questions... As you can see, i would like to establish cascading update/delete relationship among these entities (JPA datanucleus appengine implemention supports this (under the hood) by putting all entities under same entity group (Q2: Correct?) though AppEngine natively doesn't support this constraint) so naturally all would go under same entity group so that a. i can delete a Page (if my user does) in a transaction and be sure that all pages, questions, answers are all deleted b. or i can delete a subject altogether in a transaction all clear all stuff underneath it So when i extend this to my real app, i see that all of my (or atleast most) entities are interrelated and fit into same entity group to be able to transact them altogether - making my model inefficient. Q3: Please advice on how to rethink this design (and the best practice) and still achieve what i need. Ask me more if needed. Would be great if you could point me to relevant examples. p.s. 1 solution i could think of is having each entity in a separate entity group and a separate persistent field in each entity (say Exam) named 'IS_DELETED' defaulting to FALSE (value 0). Once a user deletes an Exam, i will set the field to 1 (TRUE) and that i don't load them anymore. I shall write a Cron job which clears all related entities in separate separate transaction in the backend which will retry upon failures if needed. But am sure this is not elegant and not sure whether this will work out.. Thanks all for your responses, Hari -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] about【PreparedQuery.countEnti ties()】 problem, how can I get the total count of a specif ic Query?
Hi liqwei, You can get the total count of a specific query as follows: Query query = new Query(Book.getKind()); PreparedQuery pq = ds.prepare(query); FetchOptions option = FetchOptions.Builder.withLimit(Integer.MAX_VALUE); out.println(pq.countEntities(option)); Hope this helps, Yasuo Higa On Sun, Nov 7, 2010 at 11:02 AM, liqwei liq...@gmail.com wrote: case 【PreparedQuery.countEntities()】 is Deprecated。 but how can I get the Total Count of a specific Query? the 【PreparedQuery.countEntities(FetchOptions fetchOptions) 】only return the count specificed by fetchOptions! ex. Query query = new Query(Book.getKind()); // there are 3 entities with Book kind in datastore zone PreparedQuery pq = ds.prepare(query); FetchOptions option = FetchOptions.Builder.withLimit(2); out.println(pq.countEntities()); // the count I really want, but the method is Deprecated currently; out.println(pq.countEntities(option)); // the count only return 2, but it should return 3 entity in this query; 3ks! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: countEntities() does not return more than 1000
I filed an issue: http://code.google.com/p/googleappengine/issues/detail?id=3671 Yasuo Higa On Thu, Aug 26, 2010 at 2:18 AM, tomkarren tkar...@gmail.com wrote: Agreed. We have an example for a fix that does not correspond to the 1.3.6 low level API spec. Could you please elaborate? On Aug 24, 5:54 pm, Yasuo Higa higaya...@gmail.com wrote: Hi Alfred, On Wed, Aug 25, 2010 at 3:46 AM, Alfred Fullerarfuller+appeng...@google.com wrote: Hi, The default limit is still 1000, you need to set you're own limit in order to exceed the default (in fact countEntities() has been deprecated to encourage you to do this). Here are some examples: query.countEntities(withLimit(5000)); // limit of 5000 query.countEntities(withDefaults()); // no limit query.countEntities(withLimit(2000).offset(1000)); // count entities in the range 1000 - 3000 query.countEntities(withLimit(5000)) looks nice, but I cannot find it. Where is PreparedQuery#countEntities(FetchOptions)? Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] countEntities() does not return more than 1000
Hi all, According to appengine SDK 1.3.6 release notes, results of datastore count() queries and offsets for all datastore queries are no longer capped at 1000. I guess datastore count() means PreparedQuery#countEntities(). But PreparedQuery#countEntities() does not seem to return more than 1000. Do I misunderstand datastore count()? Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Automatic restart of development server in Eclipse
Hi jesbox, Slim3 supports HOT reloading. HOT reloading means a new version of a class is automatically reloaded on the fly. The site of Slim3 is here: http://slim3.org Hope this helps, Yasuo Higa On Sun, Jul 4, 2010 at 1:40 PM, jesbox jesb...@gmail.com wrote: Hi! I have read that it should be sufficient to save a java source file and that the development server would reload the class. Then I could reload the page and it would be the latest version I would look at. This does not happen to me, I have to stop the development server and then run the application again. Many thanks for advice on this. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] java.lang.IllegalArgumentException: can't operate on multiple entity groups in a single transaction
Hi laurent,
You can't work with more than one entity group in a single
transaction. This is a limitation of GAE, although other people have
built distributed 2pc-style transactions on top of the datastore.
Slim3 supports Global Transactions between multiple entity groups:
http://sites.google.com/site/slim3appengine/slim3-datastore/transactions/global-transaction
You may worry about the overhead of global transactions. Don't worry.
It is not very expensive.
We prepare the demonstration:
http://slim3demo.appspot.com/gtx/
Hope this helps,
Yasuo Higa
This should help:
http://code.google.com/p/objectify-appengine/wiki/Concepts#Transactions
Jeff
On Wed, May 19, 2010 at 2:16 PM, laurent bagno_laur...@hotmail.com wrote:
i've searched for this error but don't find what it is.
my code :
ObjectifyService.register(Lock.class);
ObjectifyService.register(Connection.class);
ObjectifyService.register(Assignment.class);
if
(ObjectifyService.begin().query(Connection.class).countAll() ==
0) {
Objectify obj = ObjectifyService.beginTransaction();
obj.put(new Connection(DE PESSEMIER, Johann,
projet));
obj.put(new Connection(AUCQUIERE, Eric,
projet));
obj.put(new Lock(1l));
obj.getTxn().commit();
}
com.google.gwt.user.server.rpc.UnexpectedException: Service method
'public abstract java.util.List
miro.shared.GreetingService.getConnections()' threw an unexpected
exception: java.lang.IllegalArgumentException: can't operate on
multiple entity groups in a single transaction. found both Element {
type: Connection
id: 5
}
and Element {
type: Connection
id: 6
}
at
com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:
378)
at
com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:
581)
at
com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:
188)
at
com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:
224)
at
com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:
62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
511)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1166)
at
com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:
51)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:
43)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:
122)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:
418)
at
com.google.apphosting.utils.jetty.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:
70)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at com.google.appengine.tools.development.JettyContainerService
$ApiProxyHandler.handle(JettyContainerService.java:349)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
542)
at org.mortbay.jetty.HttpConnection
$RequestHandler.content(HttpConnection.java:938)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:
409)
at org.mortbay.thread.QueuedThreadPool
$PoolThread.run(QueuedThreadPool.java:582)
Thanks for your request
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com
[appengine-java] Testing on both production and development servers
Hi all, We are pleased to announce the release of Slim3 1.0.4. The main topic is wonderful testing feature. You can run same tests on both production and development servers using Kotori Web JUnit Runner. The demo is here: http://slim3demo.appspot.com/ktrwjr/index.html Release Notes: http://sites.google.com/site/slim3appengine/release-notes * Bundled Kotori Web JUnit Runner: http://code.google.com/p/slim3/issues/detail?id=26 * Stopped using MemcacheService#setNamespace(): http://code.google.com/p/slim3/issues/detail?id=28 Downloads: http://code.google.com/p/slim3/downloads/list If you use org.slim3.memcache.Memcache, you should upgrade to the version 1.0.4, because MemcacheService#setNamespace() causes NullPointerException on the current production server. Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
Hi James and Jeff, 1) Create query, fetch with limit of 21 items 2) Iterate through 20 items 3) Get cursor to be used for next page (if necessary) 4) Iterate to 21st item just to check for existence This is exactly what Slim3 supports. http://sites.google.com/site/slim3appengine/slim3-datastore/queries-and-indexes/query-cursors The following code may help you, see 236 lines(ModelQuery#asQueryResultList): http://code.google.com/p/slim3/source/browse/trunk/slim3/src/main/java/org/slim3/datastore/ModelQuery.java Hope this helps, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Can't decide: JDO, Twig-Persist or Objectify?
Hi jbdhl, You can choose Slim3, too. Please see the following tutorial using GWT. http://sites.google.com/site/slim3appengine/getting-started-with-gwt Slim3 supports HOT reloading when you use GWT, too. HOT reloading means new version of a class is automatically reloaded on the fly. Due to HOT reloading, when you change the source code, you can see the changed result on your browser without restarting web application. Yasuo Higa On Sun, Apr 25, 2010 at 4:04 AM, jbdhl jbirksd...@gmail.com wrote: Hi I really can't decide which datastore abstraction to use for my app. Any help deciding, would be much appreciated! The app is build using GWT and there is a lot of one-to-many relations (so I would value if all children could be automatically deleted if the parent is deleted). So... Which one of JDO, Objectify or Twig-Persist should I use? * Twig Pros: simple, low startup latency Cons: only one developer (major drawback), GAE-only * Objectify Pros: simple, low startup latency Cons: slightly more verbose syntax than in Twig, GAE-only * JDO Pros: not GAE-only Cons: complex, long startup latency -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is it possible to have collections of embedded objects?
Hi John, If you want to be able to query on the embedded instances then you will need to use one of the datastore specific persistence interfaces: ObjectDatastore from Twig and Objectify support this... not sure if others support embedded collections. Slim3 supports embedded collections as follows: @Attribute(lob = true) private ListEmbeddedChild Children; See http://sites.google.com/site/slim3appengine/slim3-datastore/defining-data-classes/serializable-objects Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is it possible to have collections of embedded objects?
Hi John, thanks for your clarification.
Current Slim3 does not support embedding instances like Twig/Objectify.
Yasuo Higa
Serializing is a bit different to embedding instances because the Blob is
opaque and not able to be queried. If you use @Embed in Twig or Objectify
you can query for all Container instances by filtering on an embedded
property.
e.g. Find all Bands with Albums that have sold over a million copies
class Band
{
�...@embed ListAlbum albums
}
class Album
{
String name;
int sold.
}
datastore.find()
.type(Band.class)
.addFilter(albums.sold, GREATER_THAN, 100)
.returnResultsNow();
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Why should app startup times be a problem.
Hi Jeff, In other words, Twig cannot perform the simple query: IterableObject foo = ofy.query().ancestor(yourobject); If you ever want to support something like this, you will need a registration process. If you ever want to support true polymorphic queries, you will need a registration process. Slim3 supports kindless ancestor queries and true polymorphic queries, and does not need a registration process. http://sites.google.com/site/slim3appengine/slim3-datastore/queries-and-indexes/introducing-queries http://sites.google.com/site/slim3appengine/slim3-datastore/polymorphic-model Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Objectify - Twig - SimpleDS articles
Hi Andreas, slim3 is more than just a datastore framework) You can use Slim3 as just a datastore framework. Please see http://sites.google.com/site/slim3appengine/slim3-datastore/using-slim3-datastore. Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Slim3 1.0.0 Released
Hi TQ,
Can we have this?
@MOdel
class Parent {
ListChild childs = new ArrayListChild();
}
@Model Child {
}
We can have bidirectional one to many relationships as follows:
@Model
public class Parent {
@Attribute(primaryKey = true);
private Key key;
@Attribute(persistent = false)
private InverseModelListRefChild, Parent childListRef =
new InverseModelListRefChild, Parent(Child.class, parentRef, this);
...
}
@Model
public class Child {
@Attribute(primaryKey = true);
private Key key;
private ModelRefParent parentRef =
new ModelRefParent(Parent.class);
...
}
If so, how to get parents based on child's property in Slim3?
Query for children:
ChildMeta cm = ChildMeta.get();
ListChild children = Datastore.query(cm).filter(...).asList();
Gathering parent keys:
SetKey keys = new HashSetKey();
for (Child child : children) {
keys.add(child.getParentRef().getKey());
}
Batch get for parents:
ListParent parents = Datastore.get(Parent.class, keys);
The document of relationships is here:
http://sites.google.com/site/slim3appengine/slim3-datastore/relationships
Hope this helps,
Yasuo Higa
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Slim3 1.0.0 Released
Hi all, We are pleased to announce the release of Slim3 1.0.0. Slim3 is a full-stack MVC framework optimized for Google App Engine/Java. The main features of Slim3 are as follows: * Global Transactions * Faster than JDO/JPA * Fast spin-up * HOT reloading * Type safe query You can find more information about Slim3 here: http://slim3.org Release Notes: http://sites.google.com/site/slim3appengine/release-notes Download: http://slim3.googlecode.com/files/slim3-blank-1.0.0.zip Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Slim3: Global Transaction support, Fast spin-up and HOT reloading
Hi all, We are pleased to announce the release of Slim3 RC1. Slim3 is a full-stack MVC framework optimized for Google App Engine/Java. Our main concept is Simple and Less Is More. Less is more means simplicity and clarity lead to good design. The main features of Slim3 are as follows: * Global Transactions * Faster than JDO/JPA * Fast spin-up * HOT reloading * Type safe query You can find more information about Slim3 here: http://slim3.org Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Slim3: Global Transaction support, Fast spin-up and HOT reloading
Hi Ikai, One question, though: do you have any unit testing examples? E.g. how would I mock out the datastore or write a Controller test? Slim3 supports Test-Driven Development. You can find more information here: http://sites.google.com/site/slim3appengine/getting-started/creating-controller-and-test It's a part of Getting Started. http://sites.google.com/site/slim3appengine/getting-started Tests of slim3demo are here: http://code.google.com/p/slim3/source/browse#svn/trunk/slim3demo/test/slim3/demo Slim3 provides the following testing framework: http://code.google.com/p/slim3/source/browse#svn/trunk/slim3/src/main/java/org/slim3/tester Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Slim3: Global Transaction support, Fast spin-up and HOT reloading
Hi Scott, It seems that in the code it writes out a task in a queue to close the commits, possibly. I'm pretty sure transactions should not be implemented using queues, for any reason. The complexity of storing temporary lock/tx objects in the datastore, time spent during that transaction, and then pushing them through with a task (maybe much later) seems like a dangerous plan. As you know, Eventually Consistent is a very important concept for scaling applications. To enable Eventually Consistent, Transactional Task Queue is a key item. See BASE: An Acid Alternative: http://queue.acm.org/detail.cfm?id=1394128 Supporting multi-entity-group transactions needs to be implemented at the datastore and not in a wrapper like that, IMO. There is no way to rollback any committed transactions from any other entity-group, and there are many other issues as well ... there is too much application specific behavior that can't be assumed. If you start committing a global transaction, you cannot cancel it as well as com.google.appengine.api.datastore.Transaction. If you do not commit a global transaction, you can cancel it freely. What happens if you have 3 transactions (1 transaction over 3 entity groups) and the first two entity group transactions commit and the third fails during commit? Is there a way to roll the first two back? Slim3 locks an entity group when getting, putting and deleting entities so that nobody updates the entity group. So it never happens that the first two entity group transactions commit and the third fails. What happens if your user expect transactions to work like a database where you have transaction isolation with gets/queries and updated the value of the same entity multiple times from repeated gets? (think of multiple get, incr. object property, put where the object only gets incremented by the last amount, not the sum of the increments) Unlike with most databases, queries and gets inside a datastore transaction do not see the results of previous writes inside that transaction. Specifically, if an entity is modified or deleted within a transaction, a query or get will return the original version of the entity as of the beginning of the transaction, or nothing if the entity did not exist then. http://code.google.com/appengine/docs/java/datastore/transactions.html#Isolation_and_Consistency Where does the user get an error if the transaction fails later when the task runs from the queue? (sure, that is application specific, but dangerous all the same as the current models don't allow for implicit non-synchronous operations). As described the above, if the task runs, the global transaction never fails. Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Slim3: Global Transaction support, Fast spin-up and HOT reloading
Hi Jeff, Slim3 locks an entity group when getting, putting and deleting entities so that nobody updates the entity group. So it never happens that the first two entity group transactions commit and the third fails. What do you do when there is a lock conflict? ThreadA locks Resource1 ThreadB locks Resource2 ThreadA tries to lock Resource2... what happens? When ThreadA tries to lock Resource2, ThreadA will encounter a ConcurrentModificationException, and the other locks that ThreadA has (the lock for Resource1) will be released automatically. What happens when a datastore error prevents the lock from being deleted? If the transaction is not committed for same reasons, the locks will be released 30 seconds later. Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Set the Timezone
Hi George,
Is there a way to set the timezone for my application?
TimeZone.setDefault(TimeZone.getTimeZone(GMT-8:00));
Have you actually *tested* this on App Engine?
Yes.
This should be inserted in the servlet initialization code?
You can insert the code when the servlet initializes as follows:
@Override
public void init() throws ServletException {
TimeZone.setDefault(TimeZone.getTimeZone(GMT-8:00));
}
When I reply something, I always try it.
Hope this helps,
Yasuo Higa
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Set the Timezone
Hi George, Is there a way to set the timezone for my application? TimeZone.setDefault(TimeZone.getTimeZone(GMT-8:00)); If you use JSTL, try the following setup in web.xml: context-param param-namejavax.servlet.jsp.jstl.fmt.timeZone/param-name param-valueGMT-8:00/param-value /context-param You can change GMT-8:00 as you want. Hope this helps, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is there a recommended way to differentiate between production and dev GAE environments?
Hi all,
App Engine officially supports a way to differentiate
between production and development as follows:
String env = System.getProperty(com.google.appengine.runtime.environment);
if (Development.equals(env)) {
...
} else if (Production.equals(env)) {
...
}
See http://code.google.com/p/googleappengine/wiki/SdkForJavaReleaseNotes
Hope this helps,
Yasuo Higa
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] An entity group that does not have a root entity supports transaction. How does it work?
Hi Ikai, On Fri, Dec 11, 2009 at 4:47 AM, Ikai L (Google) ika...@google.com wrote: The timestamp would be stored where the root entity is stored, but this is abstracted away from App Engine developers. If this is an area of interest, we can plan an article about how it works underneath the covers. Here's a pretty recent article about how the datastore works: http://code.google.com/appengine/articles/storage_breakdown.html Not a lot in there about transactions, though, but that would be something we would add to the series. I am looking forward to the article. Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] An entity group that does not have a root entity supports transaction. How does it work?
Hi Ikai, On Thu, Dec 10, 2009 at 2:49 AM, Ikai L (Google) ika...@google.com wrote: Which timestamp are you referring to? I mean timestamp as committed timestamp explained by the following document(50p): http://snarfed.org/space/datastore_talk.html Thanks, Yasuo Higa On Tue, Dec 8, 2009 at 6:35 PM, Yasuo Higa higaya...@gmail.com wrote: Thanks for your reply, Ikai. Entity groups are distributed based on the key and not necessarily the root entity. Thus, it is possible to place entities in the same entity group without an entity parent. Something similar to what you have done is also possible by creating a root entity, adding descendants, then removing the root entity. Transactions are possible for entities in the same entity group. Do you mean that a root key has time-stamp for transaction or an entity group itself is an entity? I would like to know where time-stamp for transaction is stored. Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] An entity group that does not have a root entity supports transaction. How does it work?
Thanks for your reply, Ikai. Entity groups are distributed based on the key and not necessarily the root entity. Thus, it is possible to place entities in the same entity group without an entity parent. Something similar to what you have done is also possible by creating a root entity, adding descendants, then removing the root entity. Transactions are possible for entities in the same entity group. Do you mean that a root key has time-stamp for transaction or an entity group itself is an entity? I would like to know where time-stamp for transaction is stored. Thanks, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Query by grandparent
Hi Lucian, Is there a way to query for entities based on their grandparent? For example, suppose we have a root A entity, a B entity having A as parent, and a C entity having B as parent (A - B - C). Is is possible to query for all C entities in the database that have A entity as grand parent? Unfortunately, AppEngine JDO does not support such a query, but Low Level API supports an ancestor query, as follows: DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); ds.prepare(new Query(C, grandparentKey)).asList(FetchOptions.Builder.withOffset(0)); Hope this helps, Yasuo Higa --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: PreparedQuery.countEntities() clarifications
Hi Pion, On my Development Server, it returns the total number of the entities which is over 40,000 entities. But when deploying it on GAE, it always returns 1,000 entities. Is this because of this limitation http://code.google.com/appengine/docs/java/datastore/overview.html#Quotas_and_Limits? If so, what is the best way to find total number of entities I have? If your query has no filter condition, I recommend the following query: DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Query query = new Query(__Stat_Kind__); query.addFilter(kind_name, FilterOperator.EQUAL, kind); Entity stat = ds.prepare(query).asSingleEntity(); Long count = (Long) stat.getProperty(count); If your query has some filter conditions, I recommend the following query: DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Query query = new Query(kind); query.setKeysOnly(); query.addFilter(...); int count = ds.prepare(query).asList(FetchOptions.Builder.withOffset(0)).size(); The first query is faster than the second one. Hope this helps, Yasuo Higa --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: PreparedQuery.countEntities() clarifications
Hi Pion, // Version 3 - This code works on both (deployed) App Engine and Development server // But The Admin Console - Datastore - Statistic shows that it has about 5,000 entities // This code returns about 6, entities Query query = new Query(kind); query.setKeysOnly(); FetchOptions fetchOptions = FetchOptions.Builder.withOffset(0); How about: FetchOptions fetchOptions = FetchOptions.Builder.withOffset(0).limit(Integer.MAX_VALUE); Instead of: FetchOptions fetchOptions = FetchOptions.Builder.withOffset(0); PreparedQuery preparedQuery = datastore.prepare(query); int result = preparedQuery.asList(fetchOptions).size(); logger.info(Integer.toString(result)); return result; } Hope this helps, Yasuo Higa --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: PreparedQuery.countEntities() clarifications
Hi Pion, It still returns 6218 entities while The Admin Console - Datastore - Statistic shows 5029. It is unusual. I think the admin console uses Statistics API. The Statistic data may be wrong or ... Could you try an another sample? Thanks, Yasuo Higa --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: Best way to update 25 entities
Hi WSouza, On Wed, Oct 28, 2009 at 1:50 AM, WSouza willmso...@gmail.com wrote: Is there any better way to update 25 entities (different entity groups) than doing everything in a single put? It is taking about 500ms plus about 2-3s of CPU time and I have to do that a lot. A batch put is more efficient than a single put. JDO: PersistenceManager#makePersistenceAll(Collection); Low level API: DatastoreService#put(IterableEntity) Hope this helps, Yasuo Higa --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: Bulk delete behind RPC
Hi Eric,
I'm trying to delete some unwanted data from my datastore and found
some code using the DatastoreService that I modified to delete as many
entries as possible in 10 seconds:
DatastoreService datastore =
DatastoreServiceFactory.getDatastoreService();
Query query = new Query(PostalCodes);
long starttime = (new Date()).getTime();
for (Entity entity : datastore.prepare(query).asIterable()) {
datastore.delete(entity.getKey());
if ((new Date().getTime()) (starttime + 1))
break;
}
Your code looks like no problem, I do not know the reason
the whole thing has been restored. It is unusual.
Regarding the performance, the following code is faster:
DatastoreService ds =
DatastoreServiceFactory.getDatastoreService();
Query query = new Query(PostalCodes);
query.setKeysOnly();
long starttime = (new Date()).getTime();
while (new Date().getTime() - starttime = 1) {
ListKey keys = new ArrayListKey();
for (Entity entity :
ds.prepare(query).asList(FetchOptions.Builder.withLimit(500))) {
keys.add(entity.getKey());
}
ds.delete(keys);
}
The points are setKeysOnly() and batch delete.
Hope this helps,
Yasuo Higa
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: using contains(Key) in JDO query
Hi Andy,
query.setFilter(key == :keyList);
ListMaster list = (ListMaster) query.execute(keyList);
I'd like to know if that works with GAE/J, because it is illegal JDOQL
syntax and would be a bug. JDOQL is supposed to follow Java syntax,
and you simply cannot do
Key == ListKey
and get success in Java.
You are right, but unfortunately the above query works on GAE/J.
I should have recommended the following query:
ListKey keys = ...;
ListObject ids = new ArrayListObject();
for (Key key : keys) {
ids.add(pm.newObjectIdInstance(Master.class, key));
}
ListMaster list = (ListMaster) pm.getObjectsById(ids);
Thanks,
Yasuo Higa
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: Tutorial for datastore with low-level api?
Hi Richard, On Thu, Oct 22, 2009 at 3:11 PM, Richard richard.wat...@gmail.com wrote: Hi Yasuo, The maximum number of entities in a batch put or batch delete is 500. The maximum number of entities in a batch get is 1000. That's true, but in this case I'm assuming not all keys have entries so I'll only get back the entities that exist. They keys that have no associated entities will return nothing. So keys entities, and hopefully keys could 1000. Put in e.g. 1500 keys, get out 50 entities, providing me with a fast(er) IN search. You cannot specify more than 1000 keys in a batch get. If you specify more than 1000 keys in a batch get, you will encounter the following exception: java.lang.IllegalArgumentException: cannot get more than 1000 keys in a single call Hope this helps, Yasuo Higa --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: Casting Text to String exception
Hi seungjin, On Thu, Oct 22, 2009 at 2:05 PM, seungjin seung...@seungjin.net wrote: I was trying to cast from Text (com.google.appengine.api.datastore.Text) to String with getValue() and toString() method but keep getting java.lang.ClassCastException: java.lang.String. You can convert Text to String using Text#getValue() as follows: Text text = new Text(aaa); assertEquals(aaa, text.getValue()); Hope this helps, Yasuo Higa --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: using contains(Key) in JDO query
Hi king, On Fri, Oct 23, 2009 at 4:21 AM, king kingalpha...@gmail.com wrote: Thanks Yasuo, it works! Do you know how I can set filter for multiple keys? that is, how I can do something like: query.setFilter(detailKeySet.contains(:keyList)); In case of a primary key field, you can use a filter of multiple keys as follows: PersistenceManager pm = PMF.get().getPersistenceManager(); Query query = pm.newQuery(Master.class); // key is a name of a primary key field. query.setFilter(key == :keyList); ListMaster list = (ListMaster) query.execute(keyList); Hope this helps, Yasuo Higa --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: using contains(Key) in JDO query
Hi king,
On Thu, Oct 22, 2009 at 6:19 AM, king kingalpha...@gmail.com wrote:
Can anyone help me to figure out how to write a query to do contains
(Key) in JDO? Here is my scriplet:
select from + master.class.getName()
+ where detailKeySet.contains(' + detail.getKey().toString() + ')
PersistenceManager pm = PMF.get().getPersistenceManager();
Query query = pm.newQuery(Master.class);
query.setFilter(detailKeySet.contains(:key));
ListMaster list = (ListMaster) query.execute(detail.getKey());
Hope this helps,
Yasuo Higa
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: JDO startsWith for Japanese does not work on production server
Thanks for your suggestion, Jason.
I filed it.
http://code.google.com/p/googleappengine/issues/detail?id=2135
Thanks,
Yasuo Higa
On Thu, Sep 17, 2009 at 6:56 AM, Jason (Google) apija...@google.com wrote:
Thanks for the extra digging, Yasuo. Feel free to file a bug in the issue
tracker regarding the default character encoding issue:
http://code.google.com/p/googleappengine/issues/list
- Jason
On Mon, Sep 14, 2009 at 7:13 PM, Yasuo Higa higaya...@gmail.com wrote:
Hi Jason,
Thanks for your reply.
On Tue, Sep 15, 2009 at 8:21 AM, Jason (Google) apija...@google.com
wrote:
You should add a filter that calls request.setCharacterEncoding() and
sets
the appropriate encoding before handling the request. If you query for
the
data without startsWith and display it on-screen, do you see what you
expect
or is it garbled?
I call request.setCharacterEncoding(UTF-8),
and my JSP is not garbled.
I think the cause is DatastoreQuery#getUpperLimitForStartsWithStr().
In the method, val.getBytes() is called.
String#getBytes() depends on default charset encoding.
Shift_JIS is used on my local server,
but an another charset encoding may be used on production server.
The following code may work fine:
private Literal getUpperLimitForStartsWithStr(String val) {
return new Literal(val + \ufffd);
}
Also, make sure you add this line after declaring the Query:
query.declareParameters(String content);
I use implicit parameter as :content.
Thanks,
Yasuo Higa
- Jason
On Sat, Sep 12, 2009 at 10:32 PM, Yasuo Higa higaya...@gmail.com
wrote:
Hi all,
JDO startsWith for Japanese works fine on development server,
but it does not work on production server.
The following url is my test application.
http://2.latest.higayasuo.appspot.com/blog/
Test code:
Query query = pm.newQuery(Blog.class, content.startsWith(:content));
ListBlog blogList = (ListBlog)
query.execute(request.getParameter(content));
request.setAttribute(blogList, blogList);
Blog.java:
@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class Blog {
�...@primarykey
�...@persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
�...@extension(vendorName = datanucleus, key = gae.encoded-pk,
value = true)
private String key;
�...@persistent
private String title;
�...@persistent
private String content;
...
}
My jsp's encoding is UTF-8.
Is there a workaround?
Thanks,
Yasuo Higa
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: JDO startsWith for Japanese does not work on production server
Hi Jason,
Thanks for your reply.
On Tue, Sep 15, 2009 at 8:21 AM, Jason (Google) apija...@google.com wrote:
You should add a filter that calls request.setCharacterEncoding() and sets
the appropriate encoding before handling the request. If you query for the
data without startsWith and display it on-screen, do you see what you expect
or is it garbled?
I call request.setCharacterEncoding(UTF-8),
and my JSP is not garbled.
I think the cause is DatastoreQuery#getUpperLimitForStartsWithStr().
In the method, val.getBytes() is called.
String#getBytes() depends on default charset encoding.
Shift_JIS is used on my local server,
but an another charset encoding may be used on production server.
The following code may work fine:
private Literal getUpperLimitForStartsWithStr(String val) {
return new Literal(val + \ufffd);
}
Also, make sure you add this line after declaring the Query:
query.declareParameters(String content);
I use implicit parameter as :content.
Thanks,
Yasuo Higa
- Jason
On Sat, Sep 12, 2009 at 10:32 PM, Yasuo Higa higaya...@gmail.com wrote:
Hi all,
JDO startsWith for Japanese works fine on development server,
but it does not work on production server.
The following url is my test application.
http://2.latest.higayasuo.appspot.com/blog/
Test code:
Query query = pm.newQuery(Blog.class, content.startsWith(:content));
ListBlog blogList = (ListBlog)
query.execute(request.getParameter(content));
request.setAttribute(blogList, blogList);
Blog.java:
@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class Blog {
�...@primarykey
�...@persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
�...@extension(vendorName = datanucleus, key = gae.encoded-pk,
value = true)
private String key;
�...@persistent
private String title;
�...@persistent
private String content;
...
}
My jsp's encoding is UTF-8.
Is there a workaround?
Thanks,
Yasuo Higa
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] JDO startsWith for Japanese does not work on production server
Hi all,
JDO startsWith for Japanese works fine on development server,
but it does not work on production server.
The following url is my test application.
http://2.latest.higayasuo.appspot.com/blog/
Test code:
Query query = pm.newQuery(Blog.class, content.startsWith(:content));
ListBlog blogList = (ListBlog)
query.execute(request.getParameter(content));
request.setAttribute(blogList, blogList);
Blog.java:
@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class Blog {
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
@Extension(vendorName = datanucleus, key = gae.encoded-pk,
value = true)
private String key;
@Persistent
private String title;
@Persistent
private String content;
...
}
My jsp's encoding is UTF-8.
Is there a workaround?
Thanks,
Yasuo Higa
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
