On Thu, Aug 18, 2011 at 11:48 AM, Joshua Smith <joshuaesm...@charter.net>wrote:
> > On Aug 18, 2011, at 2:10 PM, Fred Sauer wrote: > > On Wednesday, August 17, 2011 5:24:48 AM UTC-7, Joshua Smith wrote: >> >> I certainly hope they don't deprecate M/S, unless/until the following >> issues are addressed: >> >> 1) Ability to migrate without changing app ID. This is fundamental, and >> we've never seen any *reason* why this is impossible. I suspect that this >> is very hard, but not impossible. >> > We are able to create an alias for migrated apps, so that traffic sent to > oldappid.appspot.com will be transparently routed to your new application. > So while you won't be able to keep the same app id, you can keep using the > same appspot.com domain, which is probably what you care about most. > > > Perhaps that what some people care about most, but not me. What I care > about is what I spelled out in the next two bullet points :) > > >> Changing the app ID causes problems because: >> > - External references to entity keys (not IDs) are pervasive in many apps >> (such as google search links to deep pages, user bookmarks, or RSS feed >> URLs) because the documentation encourages the use of these keys in URLs >> > This remains an issues, but can be easily addressed in a couple of lines of > application code in the appropriate request handlers. The string encoded > keys that are referenced externally have encoded within them the old app id. > If you take the decoded key and extract the parent, kind and id/name to > create a new key (again, this is 1-2 lines of code), then your app can use > that new key to locate the migrated entity within the new datastore. > > As far as reference (list) properties: those reference keys are > automatically updated as part of the the datastore migration and should not > cause you any issues. > > - Many users have keys wrapped up in properties that the migration tool >> cannot recognize [this doesn't impact my app, but it's a common issue on >> this list] >> > This does affect some apps. As I mentioned above, a couple of lines of code > in the right place should take care of this. > > > 1-2 lines of code, in dozens of handlers is the crux of the problem. > Dealing with this at the application layer is problematic because there are > so many code paths that need to be examined. > > > As an alternative, the infrastructure could be fixed to recognize the case >> of using the old app ID instead of the new app ID in a key, and quietly make >> the substitution. >> > That's a neat idea. > > > Neat enough that you guys might implement it? > > I've suggested this before, but I'm glad that I've got your attention this > time! > I'll certainly bring up with the team. > > This would be a pretty huge step toward making the migration practical for > me. > > >> 2) Migration of large data sets takes an extremely long time. Given that >> we regularly see M/S downtimes in the couple of hours range, a couple of >> hours is probably a reasonable amount of time to migrate. A couple of days >> is much too much. [Note that I have not personally gotten far enough into a >> migration to see this duration; I'm basing this on many user complaints I've >> read on this list.] >> > Applications with small amounts of data can easily self migrate in a short > period of time using datastore_admin. Applications with more data will > benefit from the new migration tool Ikai mentioned. We created the tool > specifically to simplify and speed up the migration for larger apps, and to > reduce the read-only period required to make the final switch. > > > Yes, I saw that in his note, and if it works as advertised, I suspect it > will resolve this issue. > > >> 3) The problem of transactional consistency needs to be addressed. It is >> not acceptable for transactions to end in a "maybe this worked, maybe it >> didn't" state. They should either succeed or fail. (Although the docs not >> not say whether this problem is exclusive to HR, it seems to be, and >> certainly it matters much more in HR since every put is implicitly >> transactional.) >> > There are subtle differences between the consistency guarantees offered by > M/S and HRD. In some cases, apps need to make a few tweaks in order to work > correctly in the new environment. The benefit is much more > predictable/consistency latency and increased reliability. A good article > which reviews the consistency guarantees can be found here: > http://code.google.com/appengine/articles/transaction_isolation.html > > > I'm not talking about consistency guarantees. I get that things change > under HR, and it's pretty easy to understand what the implications of that > are. > > What I'm talking about is the problem where you have to write idempotent > transactions because you can never tell whether a transaction that threw > certain exceptions succeeded or failed. That's really messed up, and you > really need to fix that. > > But, to be fair, I'm not sure this problem will cause me actual problems if > I switch to HR. It just makes me really, really uncomfortable. > > > 4) As Ikai said, there are some unresolved issues in HR that need to be >> fixed, such as the current mystery in the 'Serious Problem: Rollback of data >> on HRD' thread (which is eerily familiar: I think somebody else reported a >> similar issue a while back). >> > > I haven't looked at that thread yet, but I suspect that reviewing the above > article ( > http://code.google.com/appengine/articles/transaction_isolation.html) > would probably be beneficial in the context of that thread. > > > Check out the thread. It's spooky weird behavior that he is reporting, > well beyond the stuff in that article. > > > > Even if all these are addressed, it would certainly be nice if M/S remained >> an option. It makes different trade-offs than HR, which may be better >> trade-offs for some applications. I will be very disappointed if they >> decide to make HR a requirement for multi-threading Python, since the two >> are completely orthogonal choices. From my reading, M/S seems to have >> better throughput than HR, at the expense of more frequent downtime, and >> spurious datastore timeouts. I'd like to have that option in a >> multi-threaded app. >> > > Thanks for the input. I hope the above response addresses some of your > concerns. > > > Ikai pointed out that MS doesn't really have higher throughput, so I guess > keeping the option is not needed if the other issues are resolved. > > So, to boil this down to the bits that are critical: > > - Implement the "neat idea" of fixing wrong app IDs within the > infrastructure; > - Get the new migration tool out of trusted tester status; > - Resolve the outstanding HR issues, such as the one in the Rollback thread > > Solve those, and I can migrate. > > Thanks for getting back to me on this!!! > > -Joshua > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Fred Sauer Developer Advocate Google Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043 fre...@google.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
