[appengine-java] Re: Storing query results in memcache
Hi Cristian, If I am correct (and I could be wrong), you want to update more than one entity group in the datastore by performing a set of sequential writes to the datastore. You may then want to read from the datastore. The writes and subsequent read are to be delayed so that the datastore write/read will take into account the already-written data. If I am right in my interpretation of what you ask, memcache is not a guaranteed data store, in that data can be deleted from it by the GAE framework at any time. I personally create queued tasks to perform sequential writes to the datastore to the same entity group. I set the interval between successive writes to be long enough for the HRD to cope correctly. Queued tasks do not get deleted, and act as my delayed datastore write queue. Does this help you? Ian On Sep 28, 5:27 pm, "nicanor.babula" wrote: > Hi, > > I just moved my app to HRD and I am trying to use the memcache to overcome > the eventual consistency problem.. When it comes to get-by-id queries the > solution is very simple. The problem appears when I insert an entity and I > want to appear in the results of more complex queries. > Because using entity groups is not the answer in certain situations I had to > use some sort of foreign keys.. > So far I thought of a system that gets an unique id from the query based on > its FilterPredicates (eg: "MEMCACHE_PHONE_NUMBERS_contactId_12345") and then > uses that to update or read the entities from there. I would build this > system by doing my own implementation of the DatastoreService interface and > attach the memcache management on the various put/delete/get methods. > I don't want to reinvent the wheel. Is there anything already done that I > could use? > > If anybody has any suggestions feel free to enlighten me ;) > > Thanks, > Cristian. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Storing query results in memcache
Hi Ian,
Thanks for sharing your strategy.. I am not sure if I understand exactly how
could I use your system to solve my problem. Actually, the memcache concept
is working nice solving similar, but simpler situations. I think I explained
poorly my situation. Here it is:
An application manages contacts. It actually has thousands of them. On the
GUI side we have a listview, an editview and a createview. The list view has
4 columns, and is paged (30 records per page) and by default is sorted
ascending by the first column. I can search the listview by issuing queries
serverside (I want the results always ordered and paged accordingly).
Now, a user clicks on "new contact". The application goes into createView.
Data input then save. Application goes to list view. The contact should have
been in the first page, but it isn't. After a few seconds, it appears.
The previous example with phone numbers is kind of the same issue, but with
simpler queries:
User modifies a contact by adding some phone numbers. Hits save and goes to
list view, then he remembers that one number was wrong and goes back to
editing the same contact. The phone numbers inserted seconds ago aren't
there. In this case, (since for other reasons I the phone numbers entities
cannot be children of the contact entity) every phone number entity has a
property called "contactId" that points to the father contact. The solution
employed in this simple case is storing in the memcache the phone numbers by
using contact-based keys so that for example
[code]
memcache.get("MEMCACHE_phones_contactId_1234")
[/code]
would give me an all the phone numbers for contact having id=1234. Somehow
(too long and boring to explain how) I manage to keep this array always up
to date so when the user returns lightning-fast to the editview the newly
inserted phone number are there, only that they are read from the memcache.
Analogically, how could this be done for the fore-mentioned listview
situation, where the query is much more complex?
Hope I didn't confused you more..
Thanks,
Cristian
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine-java/-/3kOA7GAMTQUJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Storing query results in memcache
use the level 2 cache?
"Objects are placed in the L2 cache when you commit() the transaction
of a PersistenceManager. This means that you only have datastore-
persisted objects in that cache. Also, if an object is deleted during
a transaction then at commit it will be removed from the L2 cache if
it is present."
http://www.datanucleus.org/products/accessplatform/jdo/cache.html#level2
On Sep 29, 3:59 am, "nicanor.babula" wrote:
> Hi Ian,
>
> Thanks for sharing your strategy.. I am not sure if I understand exactly how
> could I use your system to solve my problem. Actually, the memcache concept
> is working nice solving similar, but simpler situations. I think I explained
> poorly my situation. Here it is:
>
> An application manages contacts. It actually has thousands of them. On the
> GUI side we have a listview, an editview and a createview. The list view has
> 4 columns, and is paged (30 records per page) and by default is sorted
> ascending by the first column. I can search the listview by issuing queries
> serverside (I want the results always ordered and paged accordingly).
> Now, a user clicks on "new contact". The application goes into createView.
> Data input then save. Application goes to list view. The contact should have
> been in the first page, but it isn't. After a few seconds, it appears.
>
> The previous example with phone numbers is kind of the same issue, but with
> simpler queries:
> User modifies a contact by adding some phone numbers. Hits save and goes to
> list view, then he remembers that one number was wrong and goes back to
> editing the same contact. The phone numbers inserted seconds ago aren't
> there. In this case, (since for other reasons I the phone numbers entities
> cannot be children of the contact entity) every phone number entity has a
> property called "contactId" that points to the father contact. The solution
> employed in this simple case is storing in the memcache the phone numbers by
> using contact-based keys so that for example
> [code]
> memcache.get("MEMCACHE_phones_contactId_1234")
> [/code]
> would give me an all the phone numbers for contact having id=1234. Somehow
> (too long and boring to explain how) I manage to keep this array always up
> to date so when the user returns lightning-fast to the editview the newly
> inserted phone number are there, only that they are read from the memcache.
> Analogically, how could this be done for the fore-mentioned listview
> situation, where the query is much more complex?
>
> Hope I didn't confused you more..
>
> Thanks,
> Cristian
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Storing query results in memcache
Hi Christian,
If Nichole's tip doesn't help you, then for your list view, you could
back it up with an array (in memcache?). Upon any create, edit, delete
operation, you will have to test the relevant entity (using logic
equivalent to your query) to see whether its entry in the array should
be added, edited or removed.
You would have to think about whether you want subsequent updates by
other users or your system should be reflected at once or if a
snapshot at first list view would suffice.
(Note that for postings to this group, there is a delay of a few tens
of seconds before a new post is visible in the list. That is OK for a
technical forum, but I see that this is unacceptable for your app.)
Good luck!
Ian
On Sep 30, 7:49 am, Nichole wrote:
> use the level 2 cache?
>
>
>
> "Objects are placed in the L2 cache when you commit() the transaction
> of a PersistenceManager. This means that you only have datastore-
> persisted objects in that cache. Also, if an object is deleted during
> a transaction then at commit it will be removed from the L2 cache if
> it is present."
>
> http://www.datanucleus.org/products/accessplatform/jdo/cache.html#level2
>
> On Sep 29, 3:59 am, "nicanor.babula" wrote:
>
>
>
>
>
>
>
> > Hi Ian,
>
> > Thanks for sharing your strategy.. I am not sure if I understand exactly how
> > could I use your system to solve my problem. Actually, the memcache concept
> > is working nice solving similar, but simpler situations. I think I explained
> > poorly my situation. Here it is:
>
> > An application manages contacts. It actually has thousands of them. On the
> > GUI side we have a listview, an editview and a createview. The list view has
> > 4 columns, and is paged (30 records per page) and by default is sorted
> > ascending by the first column. I can search the listview by issuing queries
> > serverside (I want the results always ordered and paged accordingly).
> > Now, a user clicks on "new contact". The application goes into createView.
> > Data input then save. Application goes to list view. The contact should have
> > been in the first page, but it isn't. After a few seconds, it appears.
>
> > The previous example with phone numbers is kind of the same issue, but with
> > simpler queries:
> > User modifies a contact by adding some phone numbers. Hits save and goes to
> > list view, then he remembers that one number was wrong and goes back to
> > editing the same contact. The phone numbers inserted seconds ago aren't
> > there. In this case, (since for other reasons I the phone numbers entities
> > cannot be children of the contact entity) every phone number entity has a
> > property called "contactId" that points to the father contact. The solution
> > employed in this simple case is storing in the memcache the phone numbers by
> > using contact-based keys so that for example
> > [code]
> > memcache.get("MEMCACHE_phones_contactId_1234")
> > [/code]
> > would give me an all the phone numbers for contact having id=1234. Somehow
> > (too long and boring to explain how) I manage to keep this array always up
> > to date so when the user returns lightning-fast to the editview the newly
> > inserted phone number are there, only that they are read from the memcache.
> > Analogically, how could this be done for the fore-mentioned listview
> > situation, where the query is much more complex?
>
> > Hope I didn't confused you more..
>
> > Thanks,
> > Cristian
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Storing query results in memcache
Thanks for your help. I'll go with Ian's suggestion, storing arrays in the memcache. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/8R5rRrRrb00J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Storing query results in memcache
Level2 cache might be what you want, but anyway, just to add to the
dialog, I tinkered
with what's available in datanucleus-core v1.1.5.
Choices in datanucleus are: none, soft, and weak
(jcache is not available in any version, but see below for
customization)
In addition to those, one can make a class that implements
Level2Cache
by adopting the connectors that you see in the src code for
WeakLevel2Cache.java.
If implementing your own Level2Cache, you can, for example, use
Google's MapMaker in
Guava for finer control of expiration:
http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/collect/MapMaker.html
If you pursue a custom implementation, you'll have to edit
datanucleus-core-1.1.5.jar's plugin.xml to add the new class and name
for it.
http://www.datanucleus.org/extensions/level2_cache.html
http://grepcode.com/file_/repo1.maven.org/maven2/org.datanucleus/datanucleus-core/1.1.5/org/datanucleus/cache/WeakLevel2Cache.java/?v=source
Here's a quick check I used in development and production to assert
that the
level2 cache works in appengine (presumably by container, and not yet
distributed, not sure...)
/*
* Level 1 Cache - caching of instances within a
PersistenceManager
* Level 2 Cache - caching of instances within all
PersistenceManagers of a PersistenceManagerFactory,
*/
/*
* Add this to jdoconfig.xml or change yours for the level2
entries:
*
*
*/
PersistenceManagerFactory pmf = null;
PersistenceManager pm = null;
Transaction tx = null;
Event event1 = null;
try {
pmf = JDOHelper.getPersistenceManagerFactory("l2cache");
pm = pmf.getPersistenceManager();
tx = pm.currentTransaction();
DataStoreCache l2Cache = pmf.getDataStoreCache();
l2Cache.pinAll(true, Event.class);
tx.begin();
event1 = createEvent1();
pm.makePersistent(event1);
tx.commit();
} catch (Exception e) {
e.printStackTrace();
e.printStackTrace(out);
} finally {
if (tx.isActive()) {
tx.rollback();
}
if (pm != null) {
pm.close();
}
log.info("wrote 1 event to PM");
}
Level2Cache l2Cache =
((JDODataStoreCache)pmf.getDataStoreCache()).getLevel2Cache();
int nObjectsInCache = l2Cache.getSize();
int nPinnedObjects = l2Cache.getNumberOfPinnedObjects();
int nUnpinnedObjects = l2Cache.getNumberOfUnpinnedObjects();
log.info("tx is closed:");
log.info("number of objects in Level2Cache is " +
nObjectsInCache);
log.info("number of pinned objects in Level2Cache is " +
nPinnedObjects);
log.info("number of unpinned objects in Level2Cache is " +
nUnpinnedObjects);
l2Cache.evictAll();
nObjectsInCache = l2Cache.getSize();
nPinnedObjects = l2Cache.getNumberOfPinnedObjects();
nUnpinnedObjects = l2Cache.getNumberOfUnpinnedObjects();
log.info("number of objects in Level2Cache is " +
nObjectsInCache);
log.info("number of pinned objects in Level2Cache is " +
nPinnedObjects);
log.info("number of unpinned objects in Level2Cache is " +
nUnpinnedObjects);
On Oct 3, 5:14 am, "nicanor.babula" wrote:
> Thanks for your help. I'll go with Ian's suggestion, storing arrays in the
> memcache.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
