[appengine-java] Best practice for multithreaded access/update to entity
I have an application that accesses a user entity. This entity also contains a number of user specific settings that the user can update. Some of those settings are updated by different servlets in parallel (for instance uploading a profile picture to the blobstore) I'm having some serious issues keeping everything in sync. If I'm not careful, I can easily overwrite updates that have been done in another thread. I have the feeling this is a recurring issue for many GAE application, so I'm looking for some best practices of how to deal with it. In the relational world, often a "version" column is added. Before updating a row, the version number in the database is compared to the one in memory. I suppose I can implement something like that on GAE too, but I was wondering whether there are better/other options available in the NoSQL world. Regards, Peter -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Best practice for multithreaded access/update to entity
If the updates are that important, just use a transaction since it effectively does what Didier describes. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Sun, Apr 3, 2011 at 5:10 AM, Peter Backx wrote: > I have an application that accesses a user entity. This entity also > contains a number of user specific settings that the user can update. Some > of those settings are updated by different servlets in parallel (for > instance uploading a profile picture to the blobstore) > > I'm having some serious issues keeping everything in sync. If I'm not > careful, I can easily overwrite updates that have been done in another > thread. > > I have the feeling this is a recurring issue for many GAE application, so > I'm looking for some best practices of how to deal with it. > > In the relational world, often a "version" column is added. Before updating > a row, the version number in the database is compared to the one in memory. > I suppose I can implement something like that on GAE too, but I was > wondering whether there are better/other options available in the NoSQL > world. > > > Regards, > Peter > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Best practice for multithreaded access/update to entity
When I first started using appengine, my initial impression of transactions is "what good is this?" because of the entity group limitation. But I've since found that single-entity transactions are really useful for preventing exactly this issue. So, if you are worried about conflicts, use a transaction. One thing that helps in javaland is to adapt Python's programming model of a transaction. Implement your transactional behavior as an inner class that you hand off to a method that will retry it as many times as necessary. Jeff On Mon, Apr 4, 2011 at 1:34 AM, Ikai Lan (Google) wrote: > If the updates are that important, just use a transaction since it > effectively does what Didier describes. > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog: http://googleappengine.blogspot.com > Twitter: http://twitter.com/app_engine > Reddit: http://www.reddit.com/r/appengine > > > On Sun, Apr 3, 2011 at 5:10 AM, Peter Backx wrote: >> >> I have an application that accesses a user entity. This entity also >> contains a number of user specific settings that the user can update. Some >> of those settings are updated by different servlets in parallel (for >> instance uploading a profile picture to the blobstore) >> >> I'm having some serious issues keeping everything in sync. If I'm not >> careful, I can easily overwrite updates that have been done in another >> thread. >> >> I have the feeling this is a recurring issue for many GAE application, so >> I'm looking for some best practices of how to deal with it. >> >> In the relational world, often a "version" column is added. Before >> updating a row, the version number in the database is compared to the one in >> memory. I suppose I can implement something like that on GAE too, but I was >> wondering whether there are better/other options available in the NoSQL >> world. >> >> >> Regards, >> Peter >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-java@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Best practice for multithreaded access/update to entity
Hi Ikai, Could you expand your answer a little? Do you mean that I don't need to manage a timestap/version if I use transactions? Or do you mean I should use transactions to combine a read (to check for a changed timestamp/version) with a write? If I don't manage the timestamp myself, how does AppEngine know another process change the data since the last read? part of the key? Sorry for those beginner questions, I started using the datastore via the JDO API with a relational background, but it looks it isn't always the best fit for typical datastore issues. Objectify looks much better as it is specifically designed for it. Thanks every one for your answers Peter -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Best practice for multithreaded access/update to entity
This video will help you immensely: http://sites.google.com/site/io/under-the-covers-of-the-google-app-engine-datastore You can skip to the end if you just want to grok the transactions, but the rest of it is extraordinarily helpful as well. You are correct, if you use transactions, you do not need to manage a timestamp. This is because GAE has its own timestamp on each entity group. When you load an entity in a transaction, GAE checks this timestamp. When you commit, it checks the timestamp again - and if something changed, you get a ConcurrentModificationException. By running your transaction in a retry loop and checking for ConcurrentModificationException, you can safely ensure that non-stale data is written. Note that this isn't too far off from a description of what happens when you do a non-transactional put(). It's just that the transaction and retry loop are performed within the datastore itself. Creating an explicit transaction just provides the opportunity for your business logic to get "in the loop" so to speak. Jeff On Thu, Apr 7, 2011 at 7:09 AM, Peter Backx wrote: > Hi Ikai, > > Could you expand your answer a little? Do you mean that I don't need to > manage a timestap/version if I use transactions? Or do you mean I should use > transactions to combine a read (to check for a changed timestamp/version) > with a write? > > If I don't manage the timestamp myself, how does AppEngine know another > process change the data since the last read? part of the key? > > Sorry for those beginner questions, I started using the datastore via the > JDO API with a relational background, but it looks it isn't always the best > fit for typical datastore issues. Objectify looks much better as it is > specifically designed for it. > > > Thanks every one for your answers > > Peter > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Best practice for multithreaded access/update to entity
This inspired me to add a little snippet to Objectify's BestPractices document:
http://code.google.com/p/objectify-appengine/wiki/BestPractices#Use_Pythonic_Transactions
Basically, to run an automatically repeated transaction that updates a
(non-sharded to make the example simple) counter, I would do this:
DAOT.repeatInTransaction(new Transactable() {
@Override
public void run(DAOT daot)
{
Counter count = daot.ofy().find(Counter.class, COUNTER_ID);
count.increment();
daot.ofy().put(count);
}
});
This will guarantee that updates to the counter are not lost.
Jeff
On Thu, Apr 7, 2011 at 10:08 AM, Jeff Schnitzer wrote:
> This video will help you immensely:
>
> http://sites.google.com/site/io/under-the-covers-of-the-google-app-engine-datastore
>
> You can skip to the end if you just want to grok the transactions, but
> the rest of it is extraordinarily helpful as well.
>
> You are correct, if you use transactions, you do not need to manage a
> timestamp. This is because GAE has its own timestamp on each entity
> group. When you load an entity in a transaction, GAE checks this
> timestamp. When you commit, it checks the timestamp again - and if
> something changed, you get a ConcurrentModificationException. By
> running your transaction in a retry loop and checking for
> ConcurrentModificationException, you can safely ensure that non-stale
> data is written.
>
> Note that this isn't too far off from a description of what happens
> when you do a non-transactional put(). It's just that the transaction
> and retry loop are performed within the datastore itself. Creating an
> explicit transaction just provides the opportunity for your business
> logic to get "in the loop" so to speak.
>
> Jeff
>
> On Thu, Apr 7, 2011 at 7:09 AM, Peter Backx wrote:
>> Hi Ikai,
>>
>> Could you expand your answer a little? Do you mean that I don't need to
>> manage a timestap/version if I use transactions? Or do you mean I should use
>> transactions to combine a read (to check for a changed timestamp/version)
>> with a write?
>>
>> If I don't manage the timestamp myself, how does AppEngine know another
>> process change the data since the last read? part of the key?
>>
>> Sorry for those beginner questions, I started using the datastore via the
>> JDO API with a relational background, but it looks it isn't always the best
>> fit for typical datastore issues. Objectify looks much better as it is
>> specifically designed for it.
>>
>>
>> Thanks every one for your answers
>>
>> Peter
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine for Java" group.
>> To post to this group, send email to google-appengine-java@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine-java+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine-java?hl=en.
>>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Best practice for multithreaded access/update to entity
Perfect, exactly the info I needed! Thanks a lot Jeff. Peter -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
