On Aug 18, 2011, at 2:10 PM, Fred Sauer wrote: > On Wednesday, August 17, 2011 5:24:48 AM UTC-7, Joshua Smith wrote: > I certainly hope they don't deprecate M/S, unless/until the following issues > are addressed: > 1) Ability to migrate without changing app ID. This is fundamental, and > we've never seen any *reason* why this is impossible. I suspect that this is > very hard, but not impossible. > > We are able to create an alias for migrated apps, so that traffic sent to > oldappid.appspot.com will be transparently routed to your new application. So > while you won't be able to keep the same app id, you can keep using the same > appspot.com domain, which is probably what you care about most.
Perhaps that what some people care about most, but not me. What I care about is what I spelled out in the next two bullet points :) > > Changing the app ID causes problems because: > > - External references to entity keys (not IDs) are pervasive in many apps > (such as google search links to deep pages, user bookmarks, or RSS feed URLs) > because the documentation encourages the use of these keys in URLs > > This remains an issues, but can be easily addressed in a couple of lines of > application code in the appropriate request handlers. The string encoded keys > that are referenced externally have encoded within them the old app id. If > you take the decoded key and extract the parent, kind and id/name to create a > new key (again, this is 1-2 lines of code), then your app can use that new > key to locate the migrated entity within the new datastore. > > As far as reference (list) properties: those reference keys are automatically > updated as part of the the datastore migration and should not cause you any > issues. > > - Many users have keys wrapped up in properties that the migration tool > cannot recognize [this doesn't impact my app, but it's a common issue on this > list] > > This does affect some apps. As I mentioned above, a couple of lines of code > in the right place should take care of this. 1-2 lines of code, in dozens of handlers is the crux of the problem. Dealing with this at the application layer is problematic because there are so many code paths that need to be examined. > As an alternative, the infrastructure could be fixed to recognize the case of > using the old app ID instead of the new app ID in a key, and quietly make the > substitution. > > That's a neat idea. Neat enough that you guys might implement it? I've suggested this before, but I'm glad that I've got your attention this time! This would be a pretty huge step toward making the migration practical for me. > > 2) Migration of large data sets takes an extremely long time. Given that we > regularly see M/S downtimes in the couple of hours range, a couple of hours > is probably a reasonable amount of time to migrate. A couple of days is much > too much. [Note that I have not personally gotten far enough into a > migration to see this duration; I'm basing this on many user complaints I've > read on this list.] > > Applications with small amounts of data can easily self migrate in a short > period of time using datastore_admin. Applications with more data will > benefit from the new migration tool Ikai mentioned. We created the tool > specifically to simplify and speed up the migration for larger apps, and to > reduce the read-only period required to make the final switch. Yes, I saw that in his note, and if it works as advertised, I suspect it will resolve this issue. > > 3) The problem of transactional consistency needs to be addressed. It is not > acceptable for transactions to end in a "maybe this worked, maybe it didn't" > state. They should either succeed or fail. (Although the docs not not say > whether this problem is exclusive to HR, it seems to be, and certainly it > matters much more in HR since every put is implicitly transactional.) > > There are subtle differences between the consistency guarantees offered by > M/S and HRD. In some cases, apps need to make a few tweaks in order to work > correctly in the new environment. The benefit is much more > predictable/consistency latency and increased reliability. A good article > which reviews the consistency guarantees can be found here: > http://code.google.com/appengine/articles/transaction_isolation.html > I'm not talking about consistency guarantees. I get that things change under HR, and it's pretty easy to understand what the implications of that are. What I'm talking about is the problem where you have to write idempotent transactions because you can never tell whether a transaction that threw certain exceptions succeeded or failed. That's really messed up, and you really need to fix that. But, to be fair, I'm not sure this problem will cause me actual problems if I switch to HR. It just makes me really, really uncomfortable. > > 4) As Ikai said, there are some unresolved issues in HR that need to be > fixed, such as the current mystery in the 'Serious Problem: Rollback of data > on HRD' thread (which is eerily familiar: I think somebody else reported a > similar issue a while back). > > > I haven't looked at that thread yet, but I suspect that reviewing the above > article > (http://code.google.com/appengine/articles/transaction_isolation.html) would > probably be beneficial in the context of that thread. Check out the thread. It's spooky weird behavior that he is reporting, well beyond the stuff in that article. > > > Even if all these are addressed, it would certainly be nice if M/S remained > an option. It makes different trade-offs than HR, which may be better > trade-offs for some applications. I will be very disappointed if they decide > to make HR a requirement for multi-threading Python, since the two are > completely orthogonal choices. From my reading, M/S seems to have better > throughput than HR, at the expense of more frequent downtime, and spurious > datastore timeouts. I'd like to have that option in a multi-threaded app. > > > Thanks for the input. I hope the above response addresses some of your > concerns. Ikai pointed out that MS doesn't really have higher throughput, so I guess keeping the option is not needed if the other issues are resolved. So, to boil this down to the bits that are critical: - Implement the "neat idea" of fixing wrong app IDs within the infrastructure; - Get the new migration tool out of trusted tester status; - Resolve the outstanding HR issues, such as the one in the Rollback thread Solve those, and I can migrate. Thanks for getting back to me on this!!! -Joshua -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
