First: I think the namespace feature is a horrible idea and should never be used (and I say this as someone that runs a whitelabeled, multitenant system).
That said, doing migrations with namespaces shouldn't be all that different from doing migrations without namespaces, you just have to add one more layer of mapping (enqueue one rewrite task per namespace). Unless you're doing something really exotic, data migration is really just load/save entities and figuring out how to 'map' a task onto the task queue for every entity in the system. Rather than the dataflow system (which seems far more complex than necessary for this simple process), here's some code that may help: https://gist.github.com/stickfigure/1a345f9a8a56897da6a0 It just uses the scatter property to enqueue tasks per shard that themselves enqueue tasks per entity. Just like the map/reduce framework. The only twist is that you need to start with one task per namespace and run the scatter query on each namespace separately. Jeff On Fri, Nov 27, 2015 at 1:02 AM, Rajesh Gupta < rajesh.gu...@veersoftsolutions.com> wrote: > Hello Nick, > Our data is organized by a tenant/company. Each tenant/company's data is > in a separate namespace. > So, we have many namespaces. > > The web app is continuously evolving, and hence we have to do many times > schema migration of the datastore entities. > > Following Objectify link, tells more details on the kind of migrations we > do. > https://github.com/objectify/objectify/wiki/SchemaMigration > > Dataflow pipelines are created by specifying one namespace for a > pipeline. The namespace issue is here > https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/83 > > > > On Fri, Nov 27, 2015 at 12:19 AM, Nick (Cloud Platform Support) < > pay...@google.com> wrote: > >> Hey Rajesh, >> >> It seems that your question is a bit vague here. What kind of migration >> or data transformation are you looking to perform, how are you using >> namespaces, how would you like to use Dataflow, etc.? >> >> >> On Wednesday, November 25, 2015 at 1:35:51 AM UTC-5, Rajesh Gupta wrote: >>> >>> Anyone using namespaces for real projects? >>> >>> Cloud Dataflow was my latest attempt to do the data changes for entities >>> in large number of namespaces. The dataflow pipelines also does not help >>> here >>> >>> Anyone has any framework developed to do migration/process data for >>> large number of namespaces. >>> -- >>> Rajesh >>> *www.VeersoftSolutions.com <http://www.VeersoftSolutions.com>* >>> *www.AccountingGuru.net <http://www.AccountingGuru.net>* >>> *Accounting/Inventory/Orders on Google Cloud Platform and Mobile* >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to google-appengine+unsubscr...@googlegroups.com. >> To post to this group, send email to google-appengine@googlegroups.com. >> Visit this group at http://groups.google.com/group/google-appengine. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/078f3c3d-b119-4f92-aa8b-15733afc8093%40googlegroups.com >> <https://groups.google.com/d/msgid/google-appengine/078f3c3d-b119-4f92-aa8b-15733afc8093%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Rajesh > *www.VeersoftSolutions.com <http://www.VeersoftSolutions.com>* > *www.AccountingGuru.net <http://www.AccountingGuru.net>* > *Accounting/Inventory/Orders on Google Cloud Platform and Mobile* > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-appengine/CA%2BS7ijY0jg7SAHkVea%3DDazTe8_4cSGBH8Z68QCr_LSSfp0dTbg%40mail.gmail.com > <https://groups.google.com/d/msgid/google-appengine/CA%2BS7ijY0jg7SAHkVea%3DDazTe8_4cSGBH8Z68QCr_LSSfp0dTbg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0ujjGo19arcpNgZQTm8A_xh%3Dxnzt1f9PGNyy1AsWJf_G2A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
