Yes, when you did the official switch to put the new version of your app live for your customers.. you would need to take the app down and re-convert all the datastore data.
Now, you could put a lot of time and energy figuring out a synchronization process.. but it seems like that would take a lot of extra coding. You'd have to have some process to compare the old datastore entities to the new datastore entities (either some sort of "datemodified" property or entity checksum.. or some sort of prop by prop comparison) and really.. seems like that would take more resources than just doing a straight re-conversion. I suppose if the re-conversion took hours and hours.. and cost a lot of money.. it might make sense. But then.. you'd have to figure.. would the re-conversion cost more money (and customer time) than the price of your time (and saved customer time) to code up a iterative sync process? Trying to straddle two databases (one for an old version of code.. and one for a new version of code) that stays synchronized can get pretty mind bending. But, that all depends on the complexity of your datastore.. So, synchronization is sort of a catch 22. If your datastore is so big and complex that synchronization takes less time than total re-conversion, the time to code out a sync process might be prohibitive. If your datastore is so simple that coding a synchronization process is easy, then re-conversion probably wouldn't take much time to run.. But, this is just a guess on my part.. I don't know what your datastore changes are going to look like. On Mon, Sep 27, 2010 at 9:37 PM, Kyle Baley <k...@baley.org> wrote: > Eli, > > I can see a benefit to this approach but isn't there a chance some > data would be missed? If customers are using the app while you're > testing the new version, wouldn't you need to re-import the data from > version 1 and convert it again while the app is down? > > On Sep 23, 11:27 am, Eli Jones <eli.jo...@gmail.com> wrote: > > Yes, you'll need to duplicate your data. > > > > This is the cleanest way to do this.. in my mind. You would have > > two separate versions of your app.. and you could easily switch back to > your > > old version of you realized some horrible mistake was made after > switching > > to the new version. > > > > The main work you would need to do for the changeover from Old Version to > > New Version would be: > > > > 1. Write code that converts old datastore data to the New Version > namespace > > data. > > 2. Test New Version datastore with converted data on New App version. > > 3. Schedule a downtime late at night where the app is brought offline > (just > > upload a dummy app.yaml for the Old Version that points /.* to some > message > > page.. or you could be official and have a "Maintenance Period" version > of > > your app that you make live.. so that you make no changes whatsoever to > the > > "Old Version".), and the datastore conversion code is run after you are > > comfortable that no users are using the site and are all viewing the > > "maintenance period" messages. > > 4. Once complete, make New Version app the live version. > > > > To me, it seems like a massive headache to try to convert the data in > > place.. in the same datastore.. you have to have new model names or at > least > > some new property for "version" on your models.. and then your key_names > > have to be different.. etc.. If you screw something up, that's a > potential > > headache for your live app.. with the New Version namespace.. you can > > freely move ahead with experimenting with your coding.. and even make > plenty > > of changes to your models (if you think they are beneficial) without > > thinking "I don't know.. it's going to be a pain in the ass to figure out > > how to keep track of the old version and new version entities and now I > have > > to add a new model or property definition to my schema code.." > > > > With a namespace, you just have to add the correct mapping, conversion > code > > to your datastore converter code, and make schema changes to your New > > Version app code.. and just leave the Old Version datastore and code > alone. > > > > You can also get fancy with your converter.. (depending on how the > > conversion actually works) and you can have your converter code get a > cursor > > with keys_only for each model, serialize that cursor and pass it off to a > > task that then iterates through the cursor and breaks up the keys into > > batches to be converted and fires off tasks to do the batches in > parallel. > > And, to make the process simpler, you could just used the deferred api.. > > (so you don't have to deal with setting up task handler urls or anything > > like that). > > > > However big your datastore is.. and however much money it might cost to > have > > your data duplicated (it costs $0.01 a day for 2 GB of data above the > free > > limit).. I think it would be worth it.. it would be valuable to learn to > do > > it this way, since a large scale site would (or should) do something more > > like this. > > > > > > > > On Wed, Sep 22, 2010 at 9:39 PM, Kyle Baley <k...@baley.org> wrote: > > > That's an interesting idea. But if you do that, wouldn't you need to > > > first copy all the data from the v1 namespace to the v2 namespace? > > > > > On Sep 22, 7:07 pm, Eli Jones <eli.jo...@gmail.com> wrote: > > > > It might be useful for you to use a namespace for the new version of > the > > > > datastore. > > > > > > Thus, you could have the "new version" of the app deployed as a > non-live > > > > version of the app.. and code that "new version" to use the "new > version > > > > datastore" namespace. > > > > > > Then, when you are ready.. just change the live version of your app > to > > > the > > > > "new version". > > > > > > Here's a link: > > > > > > > http://code.google.com/appengine/docs/python/multitenancy/multitenanc... > > > > > > < > http://code.google.com/appengine/docs/python/multitenancy/multitenanc.. > > > .>So.. > > > > you could just think of your version 1 datastore as "that old > customer > > > who > > > > we're going to dump just as soon as our new version 2 datastore > customer > > > is > > > > ready"... or something like that. > > > > > > It's better (I think) than adding a "version" property to all of your > > > models > > > > or trying to maintain model consistency between app versions. > > > > > > On Mon, Sep 20, 2010 at 2:29 PM, Kyle Baley <k...@baley.org> wrote: > > > > > We've just released the first version of our application and are > now > > > > > looking at a problem we've been avoiding until now. Namely, what is > > > > > the best way to upgrade the application to a new version that > requires > > > > > changes to the datastore. We're looking at two options: > > > > > > > 1) Big Bang Upgrade > > > > > We take the application down and run an upgrade process to update > all > > > > > entities from version 1 to version 2. > > > > > > > Pros: Easy to maintain; intuitive > > > > > Cons: App has to be taken down for a period of time, which will > > > > > increase as time passes and more data is added to the datastore > > > > > (potentially hitting the limit for long-running processes > eventually) > > > > > Question: What's a good way to take the app offline? > > > > > > > 2) Version Entities Individually > > > > > Each entity has a version number and we have a series of commands, > > > > > each one responsible for upgrading an entity from one version to > the > > > > > next. As we request entities, we check to see if it's the latest > > > > > version. If not, we run each necessary upgrade command in sequence > > > > > until it is the latest version. > > > > > > > Pros: No need to take the app offline; provides flexibility on > whether > > > > > to upgrade everything at once or piecemeal > > > > > Cons: Not as intuitive; entities with different versions in the > > > > > datastore (if that matters) > > > > > > > What do other people do to upgrade their datastore for a live > > > > > application? > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Google App Engine" group. > > > > > To post to this group, send email to > google-appengine@googlegroups.com > > > . > > > > > To unsubscribe from this group, send email to > > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com><google-appengine%2Bunsubscrib > > > e...@googlegroups.com> > > > > > . > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to google-appengine@googlegroups.com > . > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib > e...@googlegroups.com> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
