Hey Greg,
You might also check out the Datastore Admin. Some of the Googlers
have already done more-or-less what you describe:
http://code.google.com/appengine/docs/adminconsole/datastoreadmin.html
Otherwise, you could do something like what you describe.
Personally I would probably 'fix' the keys on the export side, and
encode the data being transferred as something like JSON, but if
you're into protobufs go for it. You know your data and how stuff
could be batched, so you might be able to get pretty good throughput.
Robert
On Tue, Apr 5, 2011 at 01:27, Greg <g.fawc...@gmail.com> wrote:
> I'm working on the best way to transfer data from one app to another
> for a MS-HR migration. My tests with the bulkloader have indicated I
> would need to disable my app for several hours, to download from the
> old app and then upload to the new one.
>
> Downsides to this approach are that there is no obvious way to do the
> download and upload at the same time, and all the data has to be
> downloaded to and from a staging server. In my case that server is in
> New Zealand, which increases latency a lot.
>
> So my current thinking is for my old app to use urlfetch to send data
> directly to the new app - all over Google's network, so it should have
> the minimum latency. After dissecting the bulkloader, I'm planning the
> following process:
>
> - The old app converts a batch of entities to protocol buffers
> - The old app urlencodings and POSTs to the new app
> - The new app rewrites the app ID in the protocol buffers
> - The new app converts the protocol buffers back to entities and saves
> them
>
> Can anyone see any potential problems here? I use entity ids heavily,
> and I think this process preserves them, as the bulkloader does.
>
> Assuming all goes well, I will be documenting the process to aid
> others wanting to make the big MS-HR switch.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
