Re: [appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
This depends on what you mean by lazily load the entity. There are
at least three possible interpretations of lazy loading:
1) Lazily fetching data across the wire
2) Lazily converting from protocol buffers to Entity objects
3) Lazily translating Entity objects to Java POJOs
The first two steps are encapsulated within the Low-Level API, the
third step is part of JDO/Objectify/etc.
I know that when you issue a query, the results are batched in chunks
of a size that you can control. #1 and #2 are done together; the data
is pulled across the wire into materialized Entity objects in
increments of the batch size.
Objectify does #3 at the last possible moment; if you do not call
next() and retrieve a POJO entity, the POJO will not be instantiated.
You can call hasNext() to determine the existence of another page and
this will not create a POJO - however, the Low-Level API will create
the Entity.
In practice, it's hard to imagine that this matters unless your page
size is something like 2. If you have 20 elements on a page, the
worst-case scenario is that fetching 21 will add 5% to the cost of the
query. That's not much.
You can do what you want with Objectify and I suspect you can do what
you want with JDO, just don't use getResultList(). Get an actual
QueryResultIterator so the getCursor() call will return the correct
location.
Jeff
On Thu, May 6, 2010 at 8:48 PM, Chau Huynh cmhu...@gmail.com wrote:
Hi Jeff,
With JPA I will need to excute getResultList() to get a collection
http://java.sun.com/javaee/5/docs/api/javax/persistence/Query.html#getResultList()
Somehow org.datanucleus.store.appengine.query.getCursor(Iterator? iter)
can not get the cursor in the middle of my iterator, a null value is
returned in sample code below.
I guess getCursor() has something to do with the List returned.
EntityManager em =
Query query = em.createQuery(queryString);
query.setMaxResults(pageSize + 1);
em.getTransaction().begin();
IteratorTestEntity iterator = query.getResultList().iterator();
ListTestEntity list = new ArrayListTestEntity();
for (int i = 0; (i pageSize) iterator.hasNext(); i++) {
list.add(iterator.next());
}
Cursor cursor = JPACursorHelper.getCursor(iterator);
// assert iterator.hasNext() == true and cursor == null
em.getTransaction().rollback();
Just my curious:
if I'm able to iterate through the result (using
com.google.appengine.api.datastore.QueryResultIteratorT)
is it true that each iterator.hasNext() will lazily load entity from the
data store?
My concern is it might cost me more time that way than using
Query.getResultList(). Please advise.
Thank you.
On Fri, May 7, 2010 at 2:24 AM, Jeff Schnitzer j...@infohazard.org wrote:
I'm a little confused by this. Why doesn't the OP's approach work?
1) Create query, fetch with limit of 21 items
2) Iterate through 20 items
3) Get cursor to be used for next page (if necessary)
4) Iterate to 21st item just to check for existence
Why won't this work? The documentation seems to imply that this is
how cursors work...
Jeff
On Thu, May 6, 2010 at 1:03 AM, Ikai L (Google) ika...@google.com wrote:
This is a datastore limitation and not related to JDO/JPA, so you won't
be
able to do this in Objectify either (you can look at the low-level API,
it
doesn't provide this functionality). I believe I saw an issue in the
issues
tracker about creating cursors from arbitrary locations.
--
You received this message because you are subscribed to the Google
Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
Re: [appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
Thank you very much, Jeff. I'm so happy to learn those.
I will definitely try the suggested library.
Thanks.
On Sat, May 8, 2010 at 4:15 AM, Jeff Schnitzer j...@infohazard.org wrote:
This depends on what you mean by lazily load the entity. There are
at least three possible interpretations of lazy loading:
1) Lazily fetching data across the wire
2) Lazily converting from protocol buffers to Entity objects
3) Lazily translating Entity objects to Java POJOs
The first two steps are encapsulated within the Low-Level API, the
third step is part of JDO/Objectify/etc.
I know that when you issue a query, the results are batched in chunks
of a size that you can control. #1 and #2 are done together; the data
is pulled across the wire into materialized Entity objects in
increments of the batch size.
Objectify does #3 at the last possible moment; if you do not call
next() and retrieve a POJO entity, the POJO will not be instantiated.
You can call hasNext() to determine the existence of another page and
this will not create a POJO - however, the Low-Level API will create
the Entity.
In practice, it's hard to imagine that this matters unless your page
size is something like 2. If you have 20 elements on a page, the
worst-case scenario is that fetching 21 will add 5% to the cost of the
query. That's not much.
You can do what you want with Objectify and I suspect you can do what
you want with JDO, just don't use getResultList(). Get an actual
QueryResultIterator so the getCursor() call will return the correct
location.
Jeff
On Thu, May 6, 2010 at 8:48 PM, Chau Huynh cmhu...@gmail.com wrote:
Hi Jeff,
With JPA I will need to excute getResultList() to get a collection
http://java.sun.com/javaee/5/docs/api/javax/persistence/Query.html#getResultList()http://java.sun.com/javaee/5/docs/api/javax/persistence/Query.html#getResultList%28%29
Somehow org.datanucleus.store.appengine.query.getCursor(Iterator? iter)
can not get the cursor in the middle of my iterator, a null value is
returned in sample code below.
I guess getCursor() has something to do with the List returned.
EntityManager em =
Query query = em.createQuery(queryString);
query.setMaxResults(pageSize + 1);
em.getTransaction().begin();
IteratorTestEntity iterator = query.getResultList().iterator();
ListTestEntity list = new ArrayListTestEntity();
for (int i = 0; (i pageSize) iterator.hasNext(); i++) {
list.add(iterator.next());
}
Cursor cursor = JPACursorHelper.getCursor(iterator);
// assert iterator.hasNext() == true and cursor == null
em.getTransaction().rollback();
Just my curious:
if I'm able to iterate through the result (using
com.google.appengine.api.datastore.QueryResultIteratorT)
is it true that each iterator.hasNext() will lazily load entity from the
data store?
My concern is it might cost me more time that way than using
Query.getResultList(). Please advise.
Thank you.
On Fri, May 7, 2010 at 2:24 AM, Jeff Schnitzer j...@infohazard.org
wrote:
I'm a little confused by this. Why doesn't the OP's approach work?
1) Create query, fetch with limit of 21 items
2) Iterate through 20 items
3) Get cursor to be used for next page (if necessary)
4) Iterate to 21st item just to check for existence
Why won't this work? The documentation seems to imply that this is
how cursors work...
Jeff
On Thu, May 6, 2010 at 1:03 AM, Ikai L (Google) ika...@google.com
wrote:
This is a datastore limitation and not related to JDO/JPA, so you
won't
be
able to do this in Objectify either (you can look at the low-level
API,
it
doesn't provide this functionality). I believe I saw an issue in the
issues
tracker about creating cursors from arbitrary locations.
--
You received this message because you are subscribed to the Google
Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google
Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
Re: [appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
This is a datastore limitation and not related to JDO/JPA, so you won't be able to do this in Objectify either (you can look at the low-level API, it doesn't provide this functionality). I believe I saw an issue in the issues tracker about creating cursors from arbitrary locations. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
I'm a little confused by this. Why doesn't the OP's approach work? 1) Create query, fetch with limit of 21 items 2) Iterate through 20 items 3) Get cursor to be used for next page (if necessary) 4) Iterate to 21st item just to check for existence Why won't this work? The documentation seems to imply that this is how cursors work... Jeff On Thu, May 6, 2010 at 1:03 AM, Ikai L (Google) ika...@google.com wrote: This is a datastore limitation and not related to JDO/JPA, so you won't be able to do this in Objectify either (you can look at the low-level API, it doesn't provide this functionality). I believe I saw an issue in the issues tracker about creating cursors from arbitrary locations. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
Hi Jeff,
With JPA I will need to excute getResultList() to get a collection
http://java.sun.com/javaee/5/docs/api/javax/persistence/Query.html#getResultList()
Somehow org.datanucleus.store.appengine.query.getCursor(Iterator? iter)
can not get the cursor in the middle of my iterator, a null value is
returned in sample code below.
I guess getCursor() has something to do with the List returned.
EntityManager em =
Query query = em.createQuery(queryString);
query.setMaxResults(pageSize + 1);
em.getTransaction().begin();
IteratorTestEntity iterator = query.getResultList().iterator();
ListTestEntity list = new ArrayListTestEntity();
for (int i = 0; (i pageSize) iterator.hasNext(); i++) {
list.add(iterator.next());
}
Cursor cursor = JPACursorHelper.getCursor(iterator);
// assert iterator.hasNext() == true and cursor == null
em.getTransaction().rollback();
Just my curious:
if I'm able to iterate through the result (using
com.google.appengine.api.datastore.QueryResultIteratorT)
is it true that each iterator.hasNext() will lazily load entity from the
data store?
My concern is it might cost me more time that way than using
Query.getResultList(). Please advise.
Thank you.
On Fri, May 7, 2010 at 2:24 AM, Jeff Schnitzer j...@infohazard.org wrote:
I'm a little confused by this. Why doesn't the OP's approach work?
1) Create query, fetch with limit of 21 items
2) Iterate through 20 items
3) Get cursor to be used for next page (if necessary)
4) Iterate to 21st item just to check for existence
Why won't this work? The documentation seems to imply that this is
how cursors work...
Jeff
On Thu, May 6, 2010 at 1:03 AM, Ikai L (Google) ika...@google.com wrote:
This is a datastore limitation and not related to JDO/JPA, so you won't
be
able to do this in Objectify either (you can look at the low-level API,
it
doesn't provide this functionality). I believe I saw an issue in the
issues
tracker about creating cursors from arbitrary locations.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
There's 1 typo in my message, please read
org.datanucleus.store.appengine.query.CursorHelper.getCursor(Iterator?
iter)
Thanks
On Fri, May 7, 2010 at 10:48 AM, Chau Huynh cmhu...@gmail.com wrote:
Hi Jeff,
With JPA I will need to excute getResultList() to get a collection
http://java.sun.com/javaee/5/docs/api/javax/persistence/Query.html#getResultList()http://java.sun.com/javaee/5/docs/api/javax/persistence/Query.html#getResultList%28%29
Somehow org.datanucleus.store.appengine.query.getCursor(Iterator? iter)
can not get the cursor in the middle of my iterator, a null value is
returned in sample code below.
I guess getCursor() has something to do with the List returned.
EntityManager em =
Query query = em.createQuery(queryString);
query.setMaxResults(pageSize + 1);
em.getTransaction().begin();
IteratorTestEntity iterator = query.getResultList().iterator();
ListTestEntity list = new ArrayListTestEntity();
for (int i = 0; (i pageSize) iterator.hasNext(); i++) {
list.add(iterator.next());
}
Cursor cursor = JPACursorHelper.getCursor(iterator);
// assert iterator.hasNext() == true and cursor == null
em.getTransaction().rollback();
Just my curious:
if I'm able to iterate through the result (using
com.google.appengine.api.datastore.QueryResultIteratorT)
is it true that each iterator.hasNext() will lazily load entity from the
data store?
My concern is it might cost me more time that way than using
Query.getResultList(). Please advise.
Thank you.
On Fri, May 7, 2010 at 2:24 AM, Jeff Schnitzer j...@infohazard.orgwrote:
I'm a little confused by this. Why doesn't the OP's approach work?
1) Create query, fetch with limit of 21 items
2) Iterate through 20 items
3) Get cursor to be used for next page (if necessary)
4) Iterate to 21st item just to check for existence
Why won't this work? The documentation seems to imply that this is
how cursors work...
Jeff
On Thu, May 6, 2010 at 1:03 AM, Ikai L (Google) ika...@google.com
wrote:
This is a datastore limitation and not related to JDO/JPA, so you won't
be
able to do this in Objectify either (you can look at the low-level API,
it
doesn't provide this functionality). I believe I saw an issue in the
issues
tracker about creating cursors from arbitrary locations.
--
You received this message because you are subscribed to the Google
Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
Hi James and Jeff, 1) Create query, fetch with limit of 21 items 2) Iterate through 20 items 3) Get cursor to be used for next page (if necessary) 4) Iterate to 21st item just to check for existence This is exactly what Slim3 supports. http://sites.google.com/site/slim3appengine/slim3-datastore/queries-and-indexes/query-cursors The following code may help you, see 236 lines(ModelQuery#asQueryResultList): http://code.google.com/p/slim3/source/browse/trunk/slim3/src/main/java/org/slim3/datastore/ModelQuery.java Hope this helps, Yasuo Higa -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
I'm implementing some UI paging and trying to detect whether a record exists beyond the current page (in order to show a more button). I hoped to iterate over a result set of pageSize + 1, grabbing the cursor as I pass the item @ pageSize, and then using the existence of the last item to know there's more to be shown. Unfortunately I can't find a way to accomplish this using JDO. JDOCursorHelper.getCursor(ListT) is documented to get the cursor for the last item in the list, so that works as documented but doesn't help me. JDOCursorHelper.getCursor(IteratorT) seems to be what I need, but every Iterator that I can find to pass to it causes it to return null. I've tried passing ((ListT)query.execute).iterator() as well as ((QueryResult)query.execute).iterator() and both have the same result. Anyone know if there's a not-too-kludgy way to do this w/o requerying the datastore for the additional item. Or is it easier in Objectify (I'm considering switching)? Thanks much, James -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] [JDO Paging] Getting a cursor BEFORE the end of results
About JDO/JPA, I tried the same thing, and Google advised to make 2 calls: the first one is your query, the second one, which is a key-only single item query so that it can be cheap, to check if more data is available. http://groups.google.com/group/google-appengine-java/browse_thread/thread/80f1a0314987f26b?pli=1 On Thu, May 6, 2010 at 12:18 AM, James jamesk...@gmail.com wrote: I'm implementing some UI paging and trying to detect whether a record exists beyond the current page (in order to show a more button). I hoped to iterate over a result set of pageSize + 1, grabbing the cursor as I pass the item @ pageSize, and then using the existence of the last item to know there's more to be shown. Unfortunately I can't find a way to accomplish this using JDO. JDOCursorHelper.getCursor(ListT) is documented to get the cursor for the last item in the list, so that works as documented but doesn't help me. JDOCursorHelper.getCursor(IteratorT) seems to be what I need, but every Iterator that I can find to pass to it causes it to return null. I've tried passing ((ListT)query.execute).iterator() as well as ((QueryResult)query.execute).iterator() and both have the same result. Anyone know if there's a not-too-kludgy way to do this w/o requerying the datastore for the additional item. Or is it easier in Objectify (I'm considering switching)? Thanks much, James -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
