Just have some of those famous google PhDs noodle on it a while. I'm sure they can come up with some insanely clever way to make a scalable counter. Perhaps simplify the problem domain a little. For example, a direct-it-to-a-single-machine solution can scale just fine if you make it asynchronous. The requestor asks for the next ID and provides a URL handler to finish the transaction when one becomes available.
On Mar 18, 2011, at 8:22 PM, Nick Johnson (Google) wrote: > Hi folks, > > The problem - and part of the reason App Engine doesn't provide it - is that > there's no scalable way to provide an arbitrarily scalable monotonic counter. > Pretty much by definition, it requires a single coordinating machine that > issues all the IDs, which limits how far it can scale. App Engine's approach > to issuing IDs for entities is a way of avoiding that, at the expense of > monotonicity. > > On Fri, Mar 18, 2011 at 7:03 AM, Joshua Smith <joshuaesm...@charter.net> > wrote: > I second this. I think the "there are lots and lots of use cases" is a lazy > excuse for not providing a good implementation of the most typical one. > > "Generate the next sequential ID" is a function universally provided by SQL > databases. Also, SQL databases always provide an efficient count(rows) > function, whereas GAE does not. So I don't think it's unreasonable to expect > google to back-fill this deficiency as David asks. > > Actually, I'm not aware of any relational databases that provide an efficient > count() implementation other than on complete tables (which is rarely very > useful) - counting the results in an arbitrary result set requires iterating > over all the rows, which is what App Engine does. > > -Nick Johnson > > > -Joshua > > On Mar 17, 2011, at 10:14 PM, Thomas Wiradikusuma wrote: > > > But by saying that, you agree that it's indeed difficult and it can be > > implemented in various ways, which actually only lead to one thing: a > > counter. > > > > If GAE has it built-in, lazy people like me don't have to scratch head > > to implement one that works efficiently (and keep wondering whether > > there's better implementation elsewhere), because the wheel is already > > invented. > > > > But I agree that covering all use cases are hard. > > > > On Mar 17, 10:49 am, Robert Kluin <robert.kl...@gmail.com> wrote: > >> Implementing a scalable counter is pretty hard, and different uses > >> have different requirements on the behavior of counters. Some will > >> apps might demand sequential numbers, others monotonically increasing > >> (but not necessarily sequential) numbers, some might also need to > >> support decrementing the counter. There are several techniques you > >> can use to create a counters, depending on your needs and the traffic > >> your app gets. I suspect that's why Google does not provide it as > >> service -- it won't satisfy everyone's needs. > >> > >> Robert > >> > >> On Wed, Mar 16, 2011 at 22:38, Thomas Wiradikusuma > >> > >> > >> > >> > >> > >> > >> > >> <wiradikus...@gmail.com> wrote: > >>> I agree with David. > >> > >>> It's a bit "higher level" compared to other GAE services, but from > >>> application perspective, it's just another plumbing. > >>> Just like MapReduce in GAE and AppStats, they sit on top of existing > >>> services, yet they are considered low level for the rest of us. > >> > >>> Just my 2 cents :) > >> > >>> On Mar 17, 2:52 am, "Ikai Lan (Google)" <ika...@google.com> wrote: > >>>> So there are different schools of thoughts on this issue. There are > >>>> those of > >>>> us who think that App Engine should be as bare metal as possible,there > >>>> are > >>>> those of us who think App Engine should provide every single possible > >>>> API, > >>>> and there are people who argue for various positions in between. > >> > >>>> I personally think this makes sense as a configurable option or a > >>>> library, > >>>> but then that requires that we have a better place to surface App Engine > >>>> libraries. There are a lot of great libraries today, but even now, it's > >>>> not > >>>> obvious to developers how to find Googler maintained projects like App > >>>> Engine Mapper (http://code.google.com/p/appengine-mapreduce/) or the > >>>> Pipeline API (http://code.google.com/p/appengine-pipeline/). Don't even > >>>> get > >>>> me started on non-Googler maintained projects like Fantasm > >>>> (http://code.google.com/appengine/articles/fantasm.html). > >> > >>>> Ikai Lan > >>>> Developer Programs Engineer, Google App Engine > >>>> Blog:http://googleappengine.blogspot.com > >>>> Twitter:http://twitter.com/app_engine > >>>> Reddit:http://www.reddit.com/r/appengine > >> > >>>> On Tue, Mar 15, 2011 at 10:25 PM, David Keyes <keyes...@gmail.com> wrote: > >>>>> Been using gae/j for a while now. One big missing piece IMO is a > >>>>> service > >>>>> that would allow me to manage global distributed counters. Yes I know > >>>>> that > >>>>> there are patterns for doing this (e.g. sharding), but it is such a > >>>>> common > >>>>> use case that I think a highly optimized service would be ideal. > >> > >>>>> The perfect solution would allow me to have read-only access to the > >>>>> counters via the datastore. > >> > >>>>> Thoughts? > >> > >>>>> -- > >>>>> You received this message because you are subscribed to the Google > >>>>> Groups > >>>>> "Google App Engine" group. > >>>>> To post to this group, send email to google-appengine@googlegroups.com. > >>>>> To unsubscribe from this group, send email to > >>>>> google-appengine+unsubscr...@googlegroups.com. > >>>>> For more options, visit this group at > >>>>> http://groups.google.com/group/google-appengine?hl=en. > >> > >>> -- > >>> You received this message because you are subscribed to the Google Groups > >>> "Google App Engine" group. > >>> To post to this group, send email to google-appengine@googlegroups.com. > >>> To unsubscribe from this group, send email to > >>> google-appengine+unsubscr...@googlegroups.com. > >>> For more options, visit this group > >>> athttp://groups.google.com/group/google-appengine?hl=en. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > Nick Johnson, Developer Programs Engineer, App Engine > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
