i started to build somwthing similar, but rather than an in-app queue, it
was external. basically, write your new entries into a fifo or a circular
buffer on some external box(es) and have as many boxes as you need/can
afford watching this and writing all entries to cache.
this frees the app from having to do multiple writes and gives you some
scalability in number of listener/writer instances.
awl
On Mar 9, 2011 3:30 PM, "Nelz" <nelz9...@gmail.com> wrote:
> Hey all,
>
> Here's a long-ish story for you, and at the end I'll ask for some
suggestions.
>
> We have been able to 'swimlane' our traffic into two realms,
> transactional (TX) and read-only (RO). This enables us to have lower
> load on our TX/master db, and have all the RO servers talk to
> replicated slave dbs.
>
> We, of course, have been able to greatly increase our efficiency by
> putting a memcached layer between the dbs and the webheads, which we
> also split into two logical (TX and RO) groups.
>
> All well and good so far...
>
> The interactions in our app mean that a user can make a modification
> in the TX realm, and expect to see the results from the RO realm in
> real-time.
> Because we know there's a likelihood that there's a (now-stale) entry
> in the RO cache, we had the TX realm send invalidate (DELETE) to the
> RO cache. This would have worked, except that we now have a race
> condition between the master->slave replication ('eventual
> consistency', anyone?) and the request coming into the RO realm which
> basically puts back a stale version in the cache.
> To combat this, we decided that the TX realm should, upon update,
> eagerly do puts (SET) on the RO logical cache. This solved our issues
> (for the time being)...
>
> As we scale up, we now want to put additional copies of our RO realm
> in N other (cloud) datacenters. Of course, this only makes the db
> replication lag worse, so now more than ever we need to keep doing the
> eager puts (SET).
>
> But this raises several other issues for us:
> 1) the TX app has to manage not just connections to the RO and TX
> logical caches, it also needs to manage N number of remote caches as
> well.
> 2) cache connectivity between datacenters (via stunnels) is fitful at
best.
>
> #2 lead to a lot of talk of how to handle 'network fault tolerance'.
> I've realized that *most other* people's fault-tolerance is achieved
> by being on an intranet and the clients' ability to handle going from
> N -> (N-1) physical nodes fairly gracefully. This doesn't really apply
> for us, because due to going across the wire via stunnels, our caches
> go from N -> 0 nodes when there are network hiccups, and in these
> cases the clients (spymemcached for us) doesn't handle that nearly as
> gracefully. [NTS: Hrm... Maybe that is a concrete suggestion I can
> make on the spymemcached project page...?]
>
> As a short-term fix, I've made our (now broadcast) SET code be
> asynchronous, using an in-app queue, but this doesn't mitigate problem
> #1, and it increases resource use on our TX webservers.
>
> These problems have led us to talking with the CouchBase folks (Hi
> Chris! Matt, Dustin, I know you watch here too!). The discussion seems
> to have been about using the TAP interface
> (
http://blog.membase.com/northscale-blog/2010/03/want-to-know-what-your-memcached-servers-are-doing-tap-them.html
),
> but that feels a bit after-the-fact to me, as we'd have to listen to
> each of physical memcached nodes in the 'main' DC's RO cache for
> broadcast to multiple logical and remote RO caches.
>
> (There was also talk that using CouchBase in the future. I would love
> it to solve our problems, but I'm not convinced it could guarantee
> beating an eventual-consistency race condition.)
>
> So, I feel like I want something between the web clients and the
> memcached servers. (A proxy server you say? Why, that sounds loverly!)
>
> I looked at Moxi, and while I didn't get too hands-on, it seems like
> isn't a fire-and-forget interaction pattern, meaning we'd still have
> to do async processing while it goes through the cloud if we want to
> keep the TX realm running fast.
>
> Today I found http://code.google.com/p/memagent ... It seems to have
> (at least a 2-logical) broadcast for non-GET operations, which is
> cool. But the docs are shy on details, and it looks like kinda barren.
>
> TL:DR
>
> I guess I'm asking:
> Has anyone used Memagent, and what do they think of it?
> If you took the time to read my long-winded explanation, do you have
> other suggestions for addressing the issues?
>
> Thanks for indulging me.
>
> - Nelz
