Yes, I usually use the built-in index (like any field that does not
get indexed=False). The field is never used in any other (composite)
indexes. I leave it indeed so that I can very easily run a query such
as: TheModel().all().filter('version <', 7).fetch(50). That allows
me to have background processing to slowly (1 serial task), or quickly
(many parallel tasks), update my entities in a reliable way.
Performance is fine, just like any other property.
You could also forgo the index and instead loop over all entities of
that kind to do a bulk update. Or, you could simply leave code that
knows how to handle the old schema in place, and update entities as
you encounter them.
Robert
On Wed, Sep 29, 2010 at 04:11, Tonny <12br...@gmail.com> wrote:
> Is that version number indexed?
>
> And if, how is performance on puts (and searches on that index)?
>
> On Sep 28, 5:16 am, Robert Kluin <robert.kl...@gmail.com> wrote:
>> I have several apps with complex models and relationships. Personally
>> I have found using a version property on my models has made most of my
>> upgrades and schema changes fairly straight forward. Although it does
>> sometimes take an interim version of the code capable of handling both
>> versions; or, possibly that sets the "version" explicitly on a put()
>> so that it can be upgraded later.
>>
>> In some cases I simply leave a version of the 'old' and 'new' logic in
>> place. When dealing with new data I use the new logic; when dealing
>> with existing data I use the version of the logic corresponding to the
>> entity or set of entities I am dealing with. Of course the structure
>> of my code itself facilitates this method, but it has proven very easy
>> to make even significant schema changes. Basically I push the interim
>> version then kick off my upgrade handlers. Works really well if you
>> do not want much, if any, downtime. YMMV.
>>
>> I like the idea of using namespaces to version the data too. But
>> figuring out how to either keep stuff in sync or going offline for a
>> full conversion could be tricky. I suppose you could add some type of
>> change indicator that gets set on _all_ of your models, then increment
>> it each time you run a conversion. That would let you identify what
>> has changed since your last run... possibly minimizing downtime?
>>
>> Robert
>>
>>
>>
>> On Mon, Sep 27, 2010 at 22:56, Eli Jones <eli.jo...@gmail.com> wrote:
>> > Yes, when you did the official switch to put the new version of your app
>> > live for your customers.. you would need to take the app down and
>> > re-convert
>> > all the datastore data.
>> > Now, you could put a lot of time and energy figuring out a synchronization
>> > process.. but it seems like that would take a lot of extra coding.
>> > You'd have to have some process to compare the old datastore entities to
>> > the
>> > new datastore entities (either some sort of "datemodified" property or
>> > entity checksum.. or some sort of prop by prop comparison) and really..
>> > seems like that would take more resources than just doing a straight
>> > re-conversion.
>> > I suppose if the re-conversion took hours and hours.. and cost a lot of
>> > money.. it might make sense. But then.. you'd have to figure.. would the
>> > re-conversion cost more money (and customer time) than the price of your
>> > time (and saved customer time) to code up a iterative sync process?
>> > Trying to straddle two databases (one for an old version of code.. and one
>> > for a new version of code) that stays synchronized can get pretty mind
>> > bending. But, that all depends on the complexity of your datastore..
>> > So, synchronization is sort of a catch 22.
>> > If your datastore is so big and complex that synchronization takes less
>> > time
>> > than total re-conversion, the time to code out a sync process might be
>> > prohibitive.
>> > If your datastore is so simple that coding a synchronization process is
>> > easy, then re-conversion probably wouldn't take much time to run..
>> > But, this is just a guess on my part.. I don't know what your datastore
>> > changes are going to look like.
>>
>> > On Mon, Sep 27, 2010 at 9:37 PM, Kyle Baley <k...@baley.org> wrote:
>>
>> >> Eli,
>>
>> >> I can see a benefit to this approach but isn't there a chance some
>> >> data would be missed? If customers are using the app while you're
>> >> testing the new version, wouldn't you need to re-import the data from
>> >> version 1 and convert it again while the app is down?
>>
>> >> On Sep 23, 11:27 am, Eli Jones <eli.jo...@gmail.com> wrote:
>> >> > Yes, you'll need to duplicate your data.
>>
>> >> > This is the cleanest way to do this.. in my mind. You would have
>> >> > two separate versions of your app.. and you could easily switch back to
>> >> > your
>> >> > old version of you realized some horrible mistake was made after
>> >> > switching
>> >> > to the new version.
>>
>> >> > The main work you would need to do for the changeover from Old Version
>> >> > to
>> >> > New Version would be:
>>
>> >> > 1. Write code that converts old datastore data to the New Version
>> >> > namespace
>> >> > data.
>> >> > 2. Test New Version datastore with converted data on New App version.
>> >> > 3. Schedule a downtime late at night where the app is brought offline
>> >> > (just
>> >> > upload a dummy app.yaml for the Old Version that points /.* to some
>> >> > message
>> >> > page.. or you could be official and have a "Maintenance Period" version
>> >> > of
>> >> > your app that you make live.. so that you make no changes whatsoever to
>> >> > the
>> >> > "Old Version".), and the datastore conversion code is run after you are
>> >> > comfortable that no users are using the site and are all viewing the
>> >> > "maintenance period" messages.
>> >> > 4. Once complete, make New Version app the live version.
>>
>> >> > To me, it seems like a massive headache to try to convert the data in
>> >> > place.. in the same datastore.. you have to have new model names or at
>> >> > least
>> >> > some new property for "version" on your models.. and then your key_names
>> >> > have to be different.. etc.. If you screw something up, that's a
>> >> > potential
>> >> > headache for your live app.. with the New Version namespace.. you can
>> >> > freely move ahead with experimenting with your coding.. and even make
>> >> > plenty
>> >> > of changes to your models (if you think they are beneficial) without
>> >> > thinking "I don't know.. it's going to be a pain in the ass to figure
>> >> > out
>> >> > how to keep track of the old version and new version entities and now I
>> >> > have
>> >> > to add a new model or property definition to my schema code.."
>>
>> >> > With a namespace, you just have to add the correct mapping, conversion
>> >> > code
>> >> > to your datastore converter code, and make schema changes to your New
>> >> > Version app code.. and just leave the Old Version datastore and code
>> >> > alone.
>>
>> >> > You can also get fancy with your converter.. (depending on how the
>> >> > conversion actually works) and you can have your converter code get a
>> >> > cursor
>> >> > with keys_only for each model, serialize that cursor and pass it off to
>> >> > a
>> >> > task that then iterates through the cursor and breaks up the keys into
>> >> > batches to be converted and fires off tasks to do the batches in
>> >> > parallel.
>> >> > And, to make the process simpler, you could just used the deferred
>> >> > api..
>> >> > (so you don't have to deal with setting up task handler urls or anything
>> >> > like that).
>>
>> >> > However big your datastore is.. and however much money it might cost to
>> >> > have
>> >> > your data duplicated (it costs $0.01 a day for 2 GB of data above the
>> >> > free
>> >> > limit).. I think it would be worth it.. it would be valuable to learn to
>> >> > do
>> >> > it this way, since a large scale site would (or should) do something
>> >> > more
>> >> > like this.
>>
>> >> > On Wed, Sep 22, 2010 at 9:39 PM, Kyle Baley <k...@baley.org> wrote:
>> >> > > That's an interesting idea. But if you do that, wouldn't you need to
>> >> > > first copy all the data from the v1 namespace to the v2 namespace?
>>
>> >> > > On Sep 22, 7:07 pm, Eli Jones <eli.jo...@gmail.com> wrote:
>> >> > > > It might be useful for you to use a namespace for the new version of
>> >> > > > the
>> >> > > > datastore.
>>
>> >> > > > Thus, you could have the "new version" of the app deployed as a
>> >> > > > non-live
>> >> > > > version of the app.. and code that "new version" to use the "new
>> >> > > > version
>> >> > > > datastore" namespace.
>>
>> >> > > > Then, when you are ready.. just change the live version of your app
>> >> > > > to
>> >> > > the
>> >> > > > "new version".
>>
>> >> > > > Here's a link:
>>
>> >> > > > >http://code.google.com/appengine/docs/python/multitenancy/multitenanc...
>>
>> >> > > > <http://code.google.com/appengine/docs/python/multitenancy/multitenanc..
>> >> > > .>So..
>> >> > > > you could just think of your version 1 datastore as "that old
>> >> > > > customer
>> >> > > who
>> >> > > > we're going to dump just as soon as our new version 2 datastore
>> >> > > > customer
>> >> > > is
>> >> > > > ready"... or something like that.
>>
>> >> > > > It's better (I think) than adding a "version" property to all of
>> >> > > > your
>> >> > > models
>> >> > > > or trying to maintain model consistency between app versions.
>>
>> >> > > > On Mon, Sep 20, 2010 at 2:29 PM, Kyle Baley <k...@baley.org> wrote:
>> >> > > > > We've just released the first version of our application and are
>> >> > > > > now
>> >> > > > > looking at a problem we've been avoiding until now. Namely, what
>> >> > > > > is
>> >> > > > > the best way to upgrade the application to a new version that
>> >> > > > > requires
>> >> > > > > changes to the datastore. We're looking at two options:
>>
>> >> > > > > 1) Big Bang Upgrade
>> >> > > > > We take the application down and run an upgrade process to update
>> >> > > > > all
>> >> > > > > entities from version 1 to version 2.
>>
>> >> > > > > Pros: Easy to maintain; intuitive
>> >> > > > > Cons: App has to be taken down for a period of time, which will
>> >> > > > > increase as time passes and more data is added to the datastore
>> >> > > > > (potentially hitting the limit for long-running processes
>> >> > > > > eventually)
>> >> > > > > Question: What's a good way to take the app offline?
>>
>> >> > > > > 2) Version Entities Individually
>> >> > > > > Each entity has a version number and we have a series of commands,
>> >> > > > > each one responsible for upgrading an entity from one version to
>> >> > > > > the
>> >> > > > > next. As we request entities, we check to see if it's the latest
>> >> > > > > version. If not, we run each necessary upgrade command in sequence
>> >> > > > > until it is the latest version.
>>
>> >> > > > > Pros: No need to take the app offline; provides flexibility on
>> >> > > > > whether
>> >> > > > > to upgrade everything at once or piecemeal
>> >> > > > > Cons: Not as intuitive; entities with different versions in the
>> >> > > > > datastore (if that matters)
>>
>> >> > > > > What do other people do to upgrade their datastore for a live
>> >> > > > > application?
>>
>> >> > > > > --
>> >> > > > > You received this message because you are subscribed to the Google
>> >> > > Groups
>> >> > > > > "Google App Engine" group.
>>
>> ...
>>
>> read more »
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
