I don't think the original poster had a requirement for synchronous writes; he just didn't want to do the writes asynchronously because it involved a lot more code.
I'm also perfectly fine with asynchronous writes and a very lax interpretation of consistency. I don't even mind writing extra code. The thing I worry about is the feasibility of a heavy write load and the total cost of it. Unfortunately I really can't describe in detail what I want to do (I normally laugh at this kind of secrecy, but in this case it's warranted). For the game mechanic I'm thinking about, the average-case scenario is not very far from the worst-case scenario. Just a little detail: * There is no requirement that all of a user's friends must be playing the game or even have installed the app to receive points. Welcome to the world of social gaming, you can play without even without knowing it! * There are *lots* of FB users that have > 1k friends. Probably millions. More active FB users are likely to have more friends... and more likely to use my app. * Points can be assigned to multiple layers, so the # of updates is (layers * friends). * Tens of thousands of people play this "game". It could become hundreds of thousands very soon. If I'm lucky, millions. I would love to implement this game mechanic, but I just can't. Asynchronous or not, it's *way* too expensive on appengine. When it comes time to implement this feature (and it's going to come, I can see the winds blowing), I'm probably going to have to move my scoring system out of appengine. Which is a bit ironic, because one of the main advantages of appengine is scalability. I would *love* to see some sort of super-super-lax and super-super-cheap consistency option for BigTable. Or even an alternative key/value datastore that simply works like a persistent version of memcached. Something that would let me sustain 10k writes/sec without bankrupting me. Jeff On Thu, Feb 25, 2010 at 11:16 AM, Ikai L (Google) <ika...@google.com> wrote: > Jeff, point taken, but the original poster has been asking for three > different requirements: > - requirement to do all writes synchronously > - sub-some-couple-hundred-millisecond writes > - 12k entities being written > > This just won't scale well if it's common. Messaging users can be done > asynchronously, as can the portion crediting friends. I understand the > argument that you may want to do this during the lifecycle of the request so > the original user gets some kind of feedback backed by a strongly consistent > datastore, but this just isn't done. Feedback is usually faked out > optimistically, assuming that the writes will all be successful with some > cached layer being the only part of the stack being updated inside the > request. Thinking of worse case scenarios is a good thought exercise, but > it's also a bit too optimistic to design a product assuming all of a Users' > friends will play the game and engineer to meet that unrealistic > expectation. What are the standard and slightly non-standard use cases? I'd > probably look at a solution where I can store the data somewhere associated > with the original user for any users not already in the datastore, then > retrieve and generate a score for any of that User's friends on first > access. Facebook's developer admin tool has some pretty good statistics such > as bounce rate, block rate and invitation accept rate that can be used to > tune this design. > Slightly off topic, but we've been asked before if it was possible to > provide different levels of datastore consistency. In some cases I can see > the tradeoffs making sense. > > On Wed, Feb 24, 2010 at 5:52 PM, Jeff Schnitzer <j...@infohazard.org> wrote: >> >> On Wed, Feb 24, 2010 at 1:06 PM, Ikai L (Google) <ika...@google.com> >> wrote: >> > My point wasn't necessarily that it wasn't possible. makePersistentAll >> > does >> > use a batch write, and there are definitely sites that can do 12,000+ >> > writes >> > a second (and well above that), but I don't know of any that will >> > attempt to >> > do that in a single request. While it's an interesting thought exercise >> > to >> > see if BigTable can do it through App Engine's interface (hint: it can, >> > globally, easily), I can't think of a single use case for a site to need >> > to >> > do this all the time and with the sub-second requirement. I think it's >> > reasonable to ask why this design exists and why the requirements exist >> > and >> > rethink one or the other. >> >> It does seem to be a pretty extreme case, but it's not all that far >> fetched. It's possible for a Facebook user to have 5,000 friends. >> Perhaps a user wants to message all 5k of them. >> >> I could actually use this right ability now. I would like to add a >> game mechanic which, when you score some points, you also credit a >> portion of that to all of a user's friends. Worst case scenario is a >> 5,000 element read followed by a 5,000 element write. I'm probably >> going to skip this mechanic for now because I can't afford it - even >> with the average 200 or so friends. If I want it badly enough, I may >> ultimately need to move my scoring system offsite. >> >> Jeff >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-j...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> > > > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > http://googleappengine.blogspot.com | http://twitter.com/app_engine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
