[appengine-java] Re: How to ensure object synchronization when using the datastore
Hi John, you mentioned about persisting the memcache counts periodically into datastore. What's the best practice to do these periodical updates let's say on a situation when the traffic is very volatile? I've implemented Twig-persist into my project, so would you like to provide some additional code for the example based on the Cookbook strategy, http://code.google.com/p/twig-persist/wiki/Cookbook? On Apr 8, 12:33 pm, John Patterson jdpatter...@gmail.com wrote: You will need to read and update the object in a transaction which will throw an exception on commit if another thread updates the same entity. You can then refresh the object and try again. Also look into memcache which has an atomic counter increment method that is much faster than reading and writing to the datastore. The downside is that the counts could be reset so you may need to periodically store them in the datastore. On 8 Apr 2010, at 16:17, mscwd01 wrote: Hey, How do you ensure, once you have queried an object from the datastore, that another call to that object cannot be made? I basically have a bunch of objects with counters which get incremented during each call to that object. What I need to ensure is the object doesn't get called by two threads, each of which increment the counter by 1 and persist the object back to the datastore, only for the counter to be incremented by 1 and not by 2. Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com . For more options, visit this group athttp://groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: How to ensure object synchronization when using the datastore
If you have low traffic, you're best off writing to a sharded counter. Here are a few strategies I've considered or used in the past and their tradeoffs: - Periodic flushing. Has it passed some time since I've last flushed the value in this key? Yes? Persist this. You can also set this to be generational - that is, if it's rare this is flushed, almost always flush it. If it's a number that grows frequently, flush less. This strategy works best if you have a few objects with a low of growing counts. - Maintain an index of Memcache values to flush. You may want to do this in Memcache. Note that there are race conditions here, but if you're doing flushing, you're likely willing to absorb a few lost counts. The nice thing is that this can be done by a cron job, and you're less likely to miss values that are infrequently updated. This doesn't scale too well if you have a LOT of values getting updated frequently, but I suppose you could shard the indexes. This strategy is better if you have many objects with low counts. For anything in between, Memcache flushing is a premature optimization. If you have a moderate number of objects that are updated occasionally, you're just giving yourself unnecessary work. Even if they are updated a bit more frequently, you will likely be best with a sharded counter strategy. On Thu, Apr 8, 2010 at 11:42 PM, Tero Nurminen tero.nurmi...@gmail.comwrote: Hi John, you mentioned about persisting the memcache counts periodically into datastore. What's the best practice to do these periodical updates let's say on a situation when the traffic is very volatile? I've implemented Twig-persist into my project, so would you like to provide some additional code for the example based on the Cookbook strategy, http://code.google.com/p/twig-persist/wiki/Cookbook? On Apr 8, 12:33 pm, John Patterson jdpatter...@gmail.com wrote: You will need to read and update the object in a transaction which will throw an exception on commit if another thread updates the same entity. You can then refresh the object and try again. Also look into memcache which has an atomic counter increment method that is much faster than reading and writing to the datastore. The downside is that the counts could be reset so you may need to periodically store them in the datastore. On 8 Apr 2010, at 16:17, mscwd01 wrote: Hey, How do you ensure, once you have queried an object from the datastore, that another call to that object cannot be made? I basically have a bunch of objects with counters which get incremented during each call to that object. What I need to ensure is the object doesn't get called by two threads, each of which increment the counter by 1 and persist the object back to the datastore, only for the counter to be incremented by 1 and not by 2. Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group athttp:// groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: How to ensure object synchronization when using the datastore
Thanks John. So basically the process should be: - Query for the object inside a transaction - Update the object counter - Commit - If commit fails - re-query the datastore for the same object? Is it possible to just catch the exception and try committing the object again or will I have to query the datastore for the object again? Also will it work when you query for a parent object and then update the counter of a child object? Thanks again. On Apr 8, 10:33 am, John Patterson jdpatter...@gmail.com wrote: You will need to read and update the object in a transaction which will throw an exception on commit if another thread updates the same entity. You can then refresh the object and try again. Also look into memcache which has an atomic counter increment method that is much faster than reading and writing to the datastore. The downside is that the counts could be reset so you may need to periodically store them in the datastore. On 8 Apr 2010, at 16:17, mscwd01 wrote: Hey, How do you ensure, once you have queried an object from the datastore, that another call to that object cannot be made? I basically have a bunch of objects with counters which get incremented during each call to that object. What I need to ensure is the object doesn't get called by two threads, each of which increment the counter by 1 and persist the object back to the datastore, only for the counter to be incremented by 1 and not by 2. Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com . For more options, visit this group athttp://groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: How to ensure object synchronization when using the datastore
Here's a blog post Max Ross wrote about optimistic locking using built in features of JDO/JPA On Thu, Apr 8, 2010 at 2:42 AM, mscwd01 mscw...@gmail.com wrote: Thanks John. So basically the process should be: - Query for the object inside a transaction - Update the object counter - Commit - If commit fails - re-query the datastore for the same object? Is it possible to just catch the exception and try committing the object again or will I have to query the datastore for the object again? Also will it work when you query for a parent object and then update the counter of a child object? Thanks again. On Apr 8, 10:33 am, John Patterson jdpatter...@gmail.com wrote: You will need to read and update the object in a transaction which will throw an exception on commit if another thread updates the same entity. You can then refresh the object and try again. Also look into memcache which has an atomic counter increment method that is much faster than reading and writing to the datastore. The downside is that the counts could be reset so you may need to periodically store them in the datastore. On 8 Apr 2010, at 16:17, mscwd01 wrote: Hey, How do you ensure, once you have queried an object from the datastore, that another call to that object cannot be made? I basically have a bunch of objects with counters which get incremented during each call to that object. What I need to ensure is the object doesn't get called by two threads, each of which increment the counter by 1 and persist the object back to the datastore, only for the counter to be incremented by 1 and not by 2. Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group athttp:// groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: How to ensure object synchronization when using the datastore
Oops. Link is here: http://gae-java-persistence.blogspot.com/2009/10/optimistic-locking-with-version.html On Thu, Apr 8, 2010 at 10:00 AM, Ikai L (Google) ika...@google.com wrote: Here's a blog post Max Ross wrote about optimistic locking using built in features of JDO/JPA On Thu, Apr 8, 2010 at 2:42 AM, mscwd01 mscw...@gmail.com wrote: Thanks John. So basically the process should be: - Query for the object inside a transaction - Update the object counter - Commit - If commit fails - re-query the datastore for the same object? Is it possible to just catch the exception and try committing the object again or will I have to query the datastore for the object again? Also will it work when you query for a parent object and then update the counter of a child object? Thanks again. On Apr 8, 10:33 am, John Patterson jdpatter...@gmail.com wrote: You will need to read and update the object in a transaction which will throw an exception on commit if another thread updates the same entity. You can then refresh the object and try again. Also look into memcache which has an atomic counter increment method that is much faster than reading and writing to the datastore. The downside is that the counts could be reset so you may need to periodically store them in the datastore. On 8 Apr 2010, at 16:17, mscwd01 wrote: Hey, How do you ensure, once you have queried an object from the datastore, that another call to that object cannot be made? I basically have a bunch of objects with counters which get incremented during each call to that object. What I need to ensure is the object doesn't get called by two threads, each of which increment the counter by 1 and persist the object back to the datastore, only for the counter to be incremented by 1 and not by 2. Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group athttp:// groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: How to ensure object synchronization when using the datastore
Thanks Ikai, that link was very helpful. On Apr 8, 6:01 pm, Ikai L (Google) ika...@google.com wrote: Oops. Link is here: http://gae-java-persistence.blogspot.com/2009/10/optimistic-locking-w... On Thu, Apr 8, 2010 at 10:00 AM, Ikai L (Google) ika...@google.com wrote: Here's a blog post Max Ross wrote about optimistic locking using built in features of JDO/JPA On Thu, Apr 8, 2010 at 2:42 AM, mscwd01 mscw...@gmail.com wrote: Thanks John. So basically the process should be: - Query for the object inside a transaction - Update the object counter - Commit - If commit fails - re-query the datastore for the same object? Is it possible to just catch the exception and try committing the object again or will I have to query the datastore for the object again? Also will it work when you query for a parent object and then update the counter of a child object? Thanks again. On Apr 8, 10:33 am, John Patterson jdpatter...@gmail.com wrote: You will need to read and update the object in a transaction which will throw an exception on commit if another thread updates the same entity. You can then refresh the object and try again. Also look into memcache which has an atomic counter increment method that is much faster than reading and writing to the datastore. The downside is that the counts could be reset so you may need to periodically store them in the datastore. On 8 Apr 2010, at 16:17, mscwd01 wrote: Hey, How do you ensure, once you have queried an object from the datastore, that another call to that object cannot be made? I basically have a bunch of objects with counters which get incremented during each call to that object. What I need to ensure is the object doesn't get called by two threads, each of which increment the counter by 1 and persist the object back to the datastore, only for the counter to be incremented by 1 and not by 2. Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2B unsubscr...@googlegroups.com . For more options, visit this group athttp:// groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2B unsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com|http://twitter.com/app_engine -- Ikai Lan Developer Programs Engineer, Google App Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
