As far as managing complex data relationships, I don't think such a set of practices exists. What I can and should do (once I get some time) is publish some case studies about how we have persisted data in some cases. True, denormalizing data often requires you to think a little bit, but that's also why I like it: the school of normalization can often lead to people going down a checklist approach for figuring out persistence schemas which can often produce substandard structures.
Here's an example of a recent internal app I built/am building: a trip planner. For each user, I store every trip that user takes in a serialized structure on that user. For each region, I store a serialized list of trips to that region. Whenever someone updates a trip, I have to update several structures, but the assumption is that it will be read-heavy instead of update heavy. The application is heavily denormalized and uses get-by-key as much as possible. This app could also very easily have been built using a normalized approach: trip table joins to regions joins to cities. users join to user_trips join to trips. The way to approach denormalization is to think about what you are trying to achieve, and starting from there, moving backwards to figuring out how to save data. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Aug 5, 2011 at 12:13 PM, Ikai Lan (Google) <[email protected]>wrote: > I didn't mean to suggest that. > > Yes, a fanout is potentially bad, but the problem with the normalized > approach is that you equally optimize for reads and writes. In the address > book example, I update my address book about 1 time every 3 years. I read my > address book 20 times a day. I think it's fair to pay the cost of a fan out > on change because a change is so infrequent. > > Normalization has its benefits: it's harder for things to get out of sync, > which is ALWAYS a risk with denormalization. A denormalized solution tends > to favor eventually consistency approaches over strongly consistent > approaches. > > My point is that every app can be built in a denormalized approach, and in > the majority of cases, you actually *want* to build your app in this > approach, not the other way around. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > > On Fri, Aug 5, 2011 at 12:00 PM, JT <[email protected]> wrote: > >> William, >> >> >> You might want to go over this >> http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/mapreduce-osdi04.pdf, >> and come back again with any questions. Ikai and possibly others were >> trying to convey to you that bigtable approach is more scalable than >> relational approach. >> >> >> >> If it works for Google, why woudn't it work for you? Do you have a larger >> data than Google? >> >> >> On Fri, Aug 5, 2011 at 2:45 PM, William Levesque >> <[email protected]>wrote: >> >>> I was trying to explain that with... >>> >>> >>> If someones address is denormalized into 1000 contact records, then when >>> the user updates their address the system has to go out to all of the >>> contact records and update them as well. And this gets multiplied by every >>> complex relationship that exists in the data. And redundant fields can >>> increase data size exponentially. >>> >>> But is Google's position that all data should be denormalized? >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine for Java" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-appengine-java/-/W4INlksPkb0J. >>> >>> To post to this group, send email to >>> [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine-java?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
