i started to build somwthing similar, but rather than an in-app queue, it was external. basically, write your new entries into a fifo or a circular buffer on some external box(es) and have as many boxes as you need/can afford watching this and writing all entries to cache.
this frees the app from having to do multiple writes and gives you some scalability in number of listener/writer instances. awl On Mar 9, 2011 3:30 PM, "Nelz" <nelz9...@gmail.com> wrote: > Hey all, > > Here's a long-ish story for you, and at the end I'll ask for some suggestions. > > We have been able to 'swimlane' our traffic into two realms, > transactional (TX) and read-only (RO). This enables us to have lower > load on our TX/master db, and have all the RO servers talk to > replicated slave dbs. > > We, of course, have been able to greatly increase our efficiency by > putting a memcached layer between the dbs and the webheads, which we > also split into two logical (TX and RO) groups. > > All well and good so far... > > The interactions in our app mean that a user can make a modification > in the TX realm, and expect to see the results from the RO realm in > real-time. > Because we know there's a likelihood that there's a (now-stale) entry > in the RO cache, we had the TX realm send invalidate (DELETE) to the > RO cache. This would have worked, except that we now have a race > condition between the master->slave replication ('eventual > consistency', anyone?) and the request coming into the RO realm which > basically puts back a stale version in the cache. > To combat this, we decided that the TX realm should, upon update, > eagerly do puts (SET) on the RO logical cache. This solved our issues > (for the time being)... > > As we scale up, we now want to put additional copies of our RO realm > in N other (cloud) datacenters. Of course, this only makes the db > replication lag worse, so now more than ever we need to keep doing the > eager puts (SET). > > But this raises several other issues for us: > 1) the TX app has to manage not just connections to the RO and TX > logical caches, it also needs to manage N number of remote caches as > well. > 2) cache connectivity between datacenters (via stunnels) is fitful at best. > > #2 lead to a lot of talk of how to handle 'network fault tolerance'. > I've realized that *most other* people's fault-tolerance is achieved > by being on an intranet and the clients' ability to handle going from > N -> (N-1) physical nodes fairly gracefully. This doesn't really apply > for us, because due to going across the wire via stunnels, our caches > go from N -> 0 nodes when there are network hiccups, and in these > cases the clients (spymemcached for us) doesn't handle that nearly as > gracefully. [NTS: Hrm... Maybe that is a concrete suggestion I can > make on the spymemcached project page...?] > > As a short-term fix, I've made our (now broadcast) SET code be > asynchronous, using an in-app queue, but this doesn't mitigate problem > #1, and it increases resource use on our TX webservers. > > These problems have led us to talking with the CouchBase folks (Hi > Chris! Matt, Dustin, I know you watch here too!). The discussion seems > to have been about using the TAP interface > ( http://blog.membase.com/northscale-blog/2010/03/want-to-know-what-your-memcached-servers-are-doing-tap-them.html ), > but that feels a bit after-the-fact to me, as we'd have to listen to > each of physical memcached nodes in the 'main' DC's RO cache for > broadcast to multiple logical and remote RO caches. > > (There was also talk that using CouchBase in the future. I would love > it to solve our problems, but I'm not convinced it could guarantee > beating an eventual-consistency race condition.) > > So, I feel like I want something between the web clients and the > memcached servers. (A proxy server you say? Why, that sounds loverly!) > > I looked at Moxi, and while I didn't get too hands-on, it seems like > isn't a fire-and-forget interaction pattern, meaning we'd still have > to do async processing while it goes through the cloud if we want to > keep the TX realm running fast. > > Today I found http://code.google.com/p/memagent ... It seems to have > (at least a 2-logical) broadcast for non-GET operations, which is > cool. But the docs are shy on details, and it looks like kinda barren. > > TL:DR > > I guess I'm asking: > Has anyone used Memagent, and what do they think of it? > If you took the time to read my long-winded explanation, do you have > other suggestions for addressing the issues? > > Thanks for indulging me. > > - Nelz