> Hi all, > Suppose I have, say, 5 memcached servers and 3 clients. Let's say all the > processes are fine, but for whatever reason, a connection between one > client and one server is lost, e.g. there's a timeout and the socket closes. > > Now what? The one client now only see 4 servers, but the other 2 clients see > 3 servers. Whether you're using hash mod number of machines or > consistent hashing, different clients will look on different machines for the > same key. > > This must be common enough that there's a standard solution, no? Do clients > typically try to reconnect to servers periodically? Do you just live > with the duplicates? Or does everyone tend to have just a single client? > The vBuckets blog post didn't describe how different clients keep their > vBuckets in sync, making me think that people typically only have a single > client per set of memecached servers.
Well behaving clients will have a backoff/retry and treat a dead connection as a cache miss. So if one client loses connectivity it'll just see the errors as misses, then it'll attempt to reconnect one second later, then 3 seconds, etc. It works fine. vBuckets are reversed, so a server can tell you if it doesn't own that vbucket anymore. Then it's up to the clients on how to resync their state (I guess most people use proxies or something).
