[google-appengine] Re: handling simple data structure upgrades
Thanks much! Apparently something being a one-liner with RDBMS is no
indication it'll be simple with GAE's datastore :)
2009/7/27 Alkis Evlogimenos ('Αλκης Ευλογημένος) evlogime...@gmail.com:
What I used to do was to use the mapper framework found in GAE articles.
After the task queue was released I found it a lot easier to code and write
tests for a simple handler that updates one at a time. Basically the handler
has a get that is accessible only from admin and get enqueues a task for the
first item in the kind you want to upgrade. The post does the upgrade of the
passed key and finds the next in line to upgrade and queues a task for that.
When there is no entities left to process I send an email to myself that
this is done.
It looks like this:
class Upgrade(webapp.RequestHandler):
def get(self):
self.response.out.write('Started rewriting stuff')
next = MyModel.all(keys_only=True).get()
upgrader_queue.add(
taskqueue.Task(url='/admin/upgrade_mymodel/', params={'next':
str(next)}))
def post(self):
key = db.Key(self.request.get('next'))
next = MyModel.all(keys_only=True).filter('__key__ ', key).get()
def txn():
entity = db.get(key)
if entity:
// do stuff
if next:
upgrader_queue.add(
taskqueue.Task(url='/admin/upgrade_mymodel/', params={'next':
next}))
else:
// mail myself that upgrade is done!!!
db.run_in_transaction(txn)
This has a race condition: if an entity is added during the time you perform
the upgrade you will not process it. This is ok if all new entities do not
need to be upgraded which is often the case.
On Mon, Jul 27, 2009 at 3:56 PM, Alexander Tereshkin ateresh...@y-node.com
wrote:
Hello All,
Could you please share your wisdom on the following issue: I have
added a new property to a model and would like to filter the data on
this property. However, I already have quite a bit of data stored and
all the entities that were created before the last upgrade don't have
the new property and hence won't be returned by the queries once the
filter is added.
How are you dealing with such situations? Is there some sort of best
practice here or maybe I just missed something in the docs? Since
there are no batch updates, it looks like the only option is to cycle
through all the entities and update them one by one. But then again
there's too much data to do it in a single request, so I should
process only this much entities at a time and enqueue the rest of
processing.
It just sounds like too much code for something this simple. I'm
considering writing a generic routine in python to handle this type of
upgrades, but wanted to check first if I'm missing anything or if such
routine already exists.
Thanks,
Alex
--
Alkis
--
Best Regards,
Alexander Tereshkin,
Y-NODE Software
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: handling simple data structure upgrades
What I used to do was to use the mapper framework found in GAE articles.
After the task queue was released I found it a lot easier to code and write
tests for a simple handler that updates one at a time. Basically the handler
has a get that is accessible only from admin and get enqueues a task for the
first item in the kind you want to upgrade. The post does the upgrade of the
passed key and finds the next in line to upgrade and queues a task for that.
When there is no entities left to process I send an email to myself that
this is done.
It looks like this:
class Upgrade(webapp.RequestHandler):
def get(self):
self.response.out.write('Started rewriting stuff')
next = MyModel.all(keys_only=True).get()
upgrader_queue.add(
taskqueue.Task(url='/admin/upgrade_mymodel/', params={'next':
str(next)}))
def post(self):
key = db.Key(self.request.get('next'))
next = MyModel.all(keys_only=True).filter('__key__ ', key).get()
def txn():
entity = db.get(key)
if entity:
// do stuff
if next:
upgrader_queue.add(
taskqueue.Task(url='/admin/upgrade_mymodel/', params={'next':
next}))
else:
// mail myself that upgrade is done!!!
db.run_in_transaction(txn)
This has a race condition: if an entity is added during the time you perform
the upgrade you will not process it. This is ok if all new entities do not
need to be upgraded which is often the case.
On Mon, Jul 27, 2009 at 3:56 PM, Alexander Tereshkin
ateresh...@y-node.comwrote:
Hello All,
Could you please share your wisdom on the following issue: I have
added a new property to a model and would like to filter the data on
this property. However, I already have quite a bit of data stored and
all the entities that were created before the last upgrade don't have
the new property and hence won't be returned by the queries once the
filter is added.
How are you dealing with such situations? Is there some sort of best
practice here or maybe I just missed something in the docs? Since
there are no batch updates, it looks like the only option is to cycle
through all the entities and update them one by one. But then again
there's too much data to do it in a single request, so I should
process only this much entities at a time and enqueue the rest of
processing.
It just sounds like too much code for something this simple. I'm
considering writing a generic routine in python to handle this type of
upgrades, but wanted to check first if I'm missing anything or if such
routine already exists.
Thanks,
Alex
--
Alkis
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
