Hi Jeff, thanks for the feedback.
> From previous discussions here, HRD eventuality is usually seconds, sometimes minutes, and in catastrophic scenarios could be hours. That's interesting. So there is no guarantee for a maximum delay of the "eventually consistency". I understand you right, that in practice it could take hours for a new blog post to show up in a query? Cheers Mos On Mon, Oct 24, 2011 at 9:39 PM, Jeff Schnitzer <j...@infohazard.org> wrote: > I used to advise people to use shared entity groups as little as possible. > That was on M/S. I have found that the HRD changes everything, and now I > use hierarchical entity groups quite a lot. > > From previous discussions here, HRD eventuality is usually seconds, > sometimes minutes, and in catastrophic scenarios could be hours. > > For a lot of situations, this isn't a big deal. For some (say, anything > commerce-related) this could result in serious real-world problems. It's > not ok to have someone's shopping cart go blank at a memcache flush, and > under no circumstances should a customer's purchase be "lost", even for a > few seconds. > > I've found that it's possible to work around this in most cases - it's not > necessary to "accept" eventuality. The oft-repeated advice that an entity > group should roughly correspond to a single user's data is good - a shopping > cart will always be consistent that way. Another trick is to use lookup > entities instead of indexed fields; for example, create an EmailLookup keyed > by the normalized email address and which points at the relevant user > record. Get by primary key is strongly consistent and you can use a > transaction to ensure that any email address points at one and only one user > record. > > Almost all of us came from RDBMS-land stuffed full of academic theories > about the "correct" way to model data. I don't think there's an equivalent > set of theories for the datastore. I've found that every application ends > up with a pretty radically different schema depending on query profile and > the specific shape of data. There's no way to say "entity groups are good" > or "entity groups are bad" except for specific use cases, and any mature app > on the HRD will probably have examples of each. > > Jeff > > > On Mon, Oct 24, 2011 at 2:02 AM, Mos <mosa...@googlemail.com> wrote: > >> Thanks for the feedback. >> Regarding eventually consistency one advice of this talk was: >> "Accept it." >> >> I think for many use-cases in typical web-application this is ok. >> For showing a use his own post immediately, you can remember the key of >> the entity and add it manually to the query-list (if not yet available). >> >> But one question remains: >> What does "eventually consistency" mean in practice? >> >> Does it only concern new written data? >> For how long would this data not be shown up in queries? >> Is it just a matter of milliseconds or a few seconds (or minutes)? >> >> >> >> >> On Sat, Oct 22, 2011 at 10:43 PM, Gabriel Charette >> <gcharet...@gmail.com>wrote: >> >>> If you design it right you can benefit from strong consistency in the >>> situations you need it while still not hitting the 1 write/sec/group >>> cap, see the Google IO talk about it (in particular the user centric >>> design): >>> http://www.youtube.com/watch?v=xO015C3R6dw >>> >>> And the slides of the talk: >>> >>> http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en/us/events/io/2011/static/presofiles/more_9s_under_the_covers_of_the_high_replication_datastore.pdf >>> >>> As described in that talk, you want strong consistency in particular >>> for a user's own posts (i.e. it's annoying as a user if you post and >>> don't see your own post right away..) >>> >>> Cheers, >>> Gabriel >>> >>> On Oct 22, 3:16 pm, Mos <mosa...@googlemail.com> wrote: >>> > I'm still in a design phase of a new App Engine application and I'm >>> > wondering >>> > if it makes sense to go *without* using any Entity Groups. >>> > >>> > My application's data-model is similar to a discussion-forum. >>> > It has Postings, Comments, Users, Ratings and so on. If the application >>> > becomes successful, there will be significant write operations on >>> > Postings, Comments and Ratings. >>> > >>> > If I would model hierarchical *Entity-Groups* (e.g. starting from a >>> > User-Root), I would >>> > have the following *advantages*: >>> > +1: Transactional behavior during writes >>> > +2: Queries are strongly consistent >>> > +3: For hierachical data-structures, fast ancestor queries are >>> available >>> > >>> > But *Entity-Groups* introduce also some *disadvantages*: >>> > -1: Limitation in write throughput (in worst case just 1 write per >>> > second on an entity group) >>> > -2: Inflexible and a more complex data-model, which makes it harder >>> to add >>> > or re-factor relationships (forces "owned-relations" all over your >>> > application) >>> > >>> > *I think all together the disadvantages outweigh the advantages for a >>> > typical web application like mine.* >>> > The limitation for writes seems to be a significant disadvantage. Ok, >>> you >>> > can work around this with sharding but this makes >>> > you data-model even more complex. >>> > And an hierarchical data-model which is technically motivated and not >>> > related to your domain seems to be a bad idea. >>> > >>> > On the other hand transactions during write operations are often not >>> > necessary ("the last one wins"). >>> > Hierarchical searches are not needed most of the time. >>> > >>> > The only issue I'm not sure about is the advantage of "Queries are >>> strongly >>> > consistent "? >>> > What does it mean if they are not strongly consistent? >>> > Is this just an issue for a small time frame when new data are written? >>> Once >>> > the data is >>> > stored and replicated this shouldn't be an issue at all, or? >>> > >>> > I'm happy about feedback! >>> > >>> > Thanks >>> > Mos >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
