And you and your users are happy with this, Tom? I've been thinking more about entity groups, since they make this consistency mess go away. In fact, while Entity Groups are not a great solution for all my data, they do seem to fit for most of my models. Meetings can be parented by Boards, Meeting Changes can be parented by Meetings, etc.. So it's really only the top-level (Town) view where the consistency will be all over the place, and doing what you suggested for that seems like it could work.
However, I'm now realizing that adding entity groups to an existing application (even during the transition to HR) is a real problem. - I won't be able to use the migration tools, since they are going to create the new entities matching the old ones, and you can't change a parent once the model is created. - The trick of making a new key from the old key (to compensate for the app ID changing) won't work, because the parent paths will be wrong. So if I wanted to use entity groups, the best I can come up with is: - Do the migration from M/S to HR completely manually, with lots of receiving-side logic to build up entity groups where none existed before - Create a table mapping old app keys to new app keys, so that when I get a request containing a key with the old app ID, I can look it up in the table to find out the new key (which includes a parent path) Also, there are a few places where I use IDs in URLs instead of keys (like in an URL that might be sent by email, to keep it short). By my reading, that won't work, because I can't call Model.get_by_id() unless I know the parent already. So once you go to entity groups, id's are pretty much useless. (Is that right?) I guess I could create id-like things that combine the IDs of all the things up the chain of parenting (1234_5678_9012). The long strings of digits might still be nicer than full keys. I'm feeling like I have a choice between a non-entity-group solution with a ton of problems cause by consistency, or an entity-group solution with a ton of problems cause by migration (for those models where entity groups are feasible at all). Lesser of two evils design choices. Lovely. I'm really starting to hate HR. -Joshua On Sep 5, 2011, at 4:13 PM, Tom Phillips wrote: > What I try to do in these cases is a combination of: > > - when a new xxxxx is added, say "Your xxxxx has been added > successfully. It may take a few minutes to appear in the xxxxx list" > - In the xxxxx listing page, only show the most basic info for each > (Ideally a name that doesn't change, but see next bullet). So a > vanilla query accross entity groups is fine here. Some users may see > the new xxxxx sooner than others (no biggie, matter of seconds > usually) > - If the name _can_ get changed, say something to the changer like > "The new name for your xxxxx may take a few minutes to appear in the > xxxxx list" > - Clicking on the "details" of a xxxxx does a get-by-key, ensuring the > latest > > And If you're going to cache the list page, have the cache maintained > by a cron job, to ensure there are no collisions. Cron job needs to be > a bit smart, since in theory it could also see a new xxxxx one run, > then not see it on a query a minute later. But at least there is a > single place to deal with that with no contention on the cache. > > /Tom > > On Sep 5, 1:58 pm, Joshua Smith <joshuaesm...@charter.net> wrote: >> Yeah, that'll work, too. Unless there are two changes to the list of boards >> at about the same time. Then it will give a bizarre result. >> >> The general idea of augmenting queries with extra results that you suspect >> it might not get seems to come up a lot as a "solution" to the eventual >> consistency problem. Other than entity groups (which doesn't seem to fit >> any of my real-world data models), this is the only suggestion in the docs. >> >> However, it really kinda sucks. Since query results are usually sorted in >> some way, you need to re-sort to get the fixed record into the right place. >> But, of course, you can't do that if you are using the query as an iterator. >> And, of course, the change may have made that record no longer fit the >> query criteria, or it may have caused it to fit where it didn't before, so >> you need to deal with all three cases: remove the record from the list of >> results, add the record to the list of results, and replace the record in a >> list of results. And then there is the question of where you put the >> record. Putting it into a cookie or query parameter is weird, and fragile. >> Putting it into memcache is fragile. >> >> I suppose it's possible that there just is no good design pattern to use >> here. In particular, I'm having trouble coming up with a way that doesn't >> require me to wrap 3 old lines of code with 20 lines of crap, then repeating >> that in 50 different places. I don't mind having a bunch of crap if there >> is some way to reuse it. But I'm struggling with that. >> >> Is ANYBODY out there happy with the way they solved this problem? >> >> -Joshua >> >> On Sep 4, 2011, at 5:51 PM, Tom Phillips wrote: >> >> >> >>> "I clear the cache whenever the list of boards changes in some way" >> >>> How about update the cache at that point instead of clearing it? >> >>> Need be you could even generate the HTML for the cache update with a >>> URLFetch to your UI handler where you include the added/changed board >>> key(s) as parameters, so they can be gotten with strong consistency on >>> that request and merge into the query result. >> >>> /Tom >> >>> On Sep 4, 11:16 am, Joshua Smith <joshuaesm...@charter.net> wrote: >>>> My monkeypatching solution (see my recent post in the -python group), >>>> which Guido says I shouldn't use, but which is just so darned pretty I >>>> can't help it, has gotten me through the first challenge of switching to >>>> HR, which is dealing with google search results containing keys into my >>>> old app's data store. >> >>>> So now I'm looking at the big Kahuna problem of consistency. Here's my >>>> first messy challenge there: >> >>>> My app puts a list of boards on the home page for a town, along with the >>>> list of meetings. Generating that list of boards was taking a lot of CPU, >>>> but they hardly ever change, so I put in a memcache system that built the >>>> HTML when it wasn't in the cache, and then cached it before serving. I >>>> clear the cache whenever the list of boards changes in some way. >> >>>> Well that ain't gonna work in HR. It's quite possible that I update a >>>> board, clear the cache, and someone comes and hits that page before >>>> "eventually consistent" comes to pass. So now I've got a cached copy of >>>> the stale data. >> >>>> (Note that I cannot use entity groups to solve this because some boards >>>> are municipal agencies, and therefore cannot be parented to the town that >>>> is building its list. I could parent all boards to some global parent, >>>> but, well, yuck.) >> >>>> I have some different ideas about how to fix this, but I'm wondering if >>>> anyone else who's done the port to HR has come up with a solution they >>>> find particularly elegant? I assume this is a pretty common problem, so >>>> there must be a design pattern out there… somewhere. >> >>>> Here are my ideas: >> >>>> - Clear the cache with a periodic task that re-clears it several times. >>>> I'm thinking a recurring geometric retry would be prudent (1, 2, 4, 8, 16, >>>> 32, 64, 128, 256, 512 seconds, and then pray that we have consistency) >> >>>> - Checksum the modified or new board, and put that sum into memcache. >>>> When generating the new board, confirm that any checksums are good. This >>>> seems more deterministic, except I don't trust memache not to squelch the >>>> checksum record. So perhaps I should do something in the datastore. This >>>> feels like it's be about 10x as much code as the stupid geometric flush. >> >>>> Any suggestions? >> >>>> -Joshua >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group >>> athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
