libmemcache: memcached_behavior_number_of_replicas working?
http://docs.libmemcached.org/memcached_behavior_set.html#memcached_behavior_number_of_replicas http://docs.libmemcached.org/memcached_behavior_set.html#memcached_behavior_number_of_replicas or http://docs.libmemcached.org/memcached_behavior_set.html#memcached_behavior_number_of_replicas http://search.cpan.org/~timb/Memcached-libmemcached-0.4406/lib/Memcached/libmemcached/memcached_behavior.pm#MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS Is that feature working in libmemcached?I tried with Perl's Memcached::libmemcached two different versions: - Version 0.3102 (with libmemcached-0.31 embedded) - Version 0.4406 (with libmemcached-0.44 embedded) http://docs.libmemcached.org/memcached_behavior_set.html#memcached_behavior_number_of_replicas and watching with Wireshark (and also reading from individual servers) I only saw a writes to a single server. use Memcached::libmemcached qw(:memcached_behavior_t ); use strict; use warnings; my $memc = Memcached::libmemcached-new; $memc-memcached_server_add( 'localhost', 11211 ); $memc-memcached_server_add( 'localhost', 11212 ); # Also tried specifying 2 for number of replicas $memc-memcached_behavior_set( MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS() = 1 ); $memc-memcached_set( 'test', 1, 10 ) || die 'set failed ' . $memc-errstr; Also, what does MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT do? Does that mean after a server reports that many failures any gets or sets to that server will fail without the library attempting to send to the memcached server? I assume that doesn't mean the server will be pulled out of the pool and keys would get rehashed to new servers. (Although, maybe if MEMCACHED_BEHAVIOR_AUTO_EJECT_HOSTS is set, but if that means keys would hash to different servers that seems like a bad idea.) Yes, I'm looking at poor-man's HA with these two features. Looking at how pulling the plug on one server won't result in get or set failures, and not an increased amount of time (say waiting for a failed server to respond). -- Bill Moseley mose...@hank.org
Memcached-1.4.5 Solaris 10 Error
Hi All, After some problems, i have success in compiling memcached on solaris sparc. libmemcached was very hard to get working too. I have a version here with some patches to detect and implement unsuported features on solaris like getopt_long and etc... Now, its compiled successfuly and working well on solaris sparc 9. On solaris sparc 10, I'm getting the follow error: -bash-3.00$ uname -a SunOS server 5.10 Generic_118833-33 sun4u sparc SUNW,Sun-Fire-V490 -bash-3.00$ ./memcached [warn] ioctl: DP_POLL: Invalid argument [warn] ioctl: DP_POLL: Invalid argument [warn] ioctl: DP_POLL: Invalid argument [warn] ioctl: DP_POLL: Invalid argument [warn] ioctl: DP_POLL: Invalid argument -bash-3.00$ -bash-3.00$ ulimit -a core file size(blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files(-n) 256 pipe size (512 bytes, -p) 10 stack size(kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes(-u) 29995 virtual memory(kbytes, -v) unlimited -bash-3.00$ -bash-3.00$ truss ./memcached ... /1: setsockopt(39, SOL_SOCKET, SO_SNDBUF, 0xFFBFF4FC, 4, SOV_DEFAULT) = 0 /1: setsockopt(39, SOL_SOCKET, SO_SNDBUF, 0xFFBFF4FC, 4, SOV_DEFAULT) Err#132 ENOBUFS /1: bind(39, 0x0003C0A0, 16, SOV_SOCKBSD) = 0 /1: write(9, \0, 1) = 1 /1: write(16, \0, 1) = 1 /1: write(23, \0, 1) = 1 /1: write(30, \0, 1) = 1 /1: pwrite(3, \0\0\004\001\0\0\0\0\0 $.., 24, 0) = 24 /1: ioctl(3, DP_POLL, 0xFFBFF490) Err#22 EINVAL /1: mmap(0x0001, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFEFA [/1:write(2, [, 1) = 1 warn/1: write(2, w a r n, 4) = 4 ] /1: write(2, ] , 2) = 2 ioctl: DP_POLL: Invalid argument/1: write(2, i o c t l : D P _ P O.., 32) = 32 /1: write(2, \n, 1) = 1 /1: lwp_unpark(6) = 0 /6: lwp_park(0x, 0) = 0 /6: lwp_sigmask(SIG_SETMASK, 0xFFBFFEFF, 0xFFF7) = 0xFFBFFEFF [0x] /6: lwp_exit() /1: lwp_wait(6, 0xFFBFF584) = 0 _exit(0) I have look around for some problems on libevent on solaris 10 but cant find any solution. Thanks in advance
Re: libmemcache: memcached_behavior_number_of_replicas working?
Hi, Its works for me on version 0.45 with C. I didnt like of this behavior but its works. The concept of master and slave dont exist. So, if its get a error trying to send data to the hash(key)-server, its dont send data to the others server(s). On Mar 3, 12:09 pm, Bill Moseley mose...@hank.org wrote: http://docs.libmemcached.org/memcached_behavior_set.html#memcached_be... http://docs.libmemcached.org/memcached_behavior_set.html#memcached_be... or http://docs.libmemcached.org/memcached_behavior_set.html#memcached_be...http://search.cpan.org/~timb/Memcached-libmemcached-0.4406/lib/Memcac... Is that feature working in libmemcached? I tried with Perl's Memcached::libmemcached two different versions: - Version 0.3102 (with libmemcached-0.31 embedded) - Version 0.4406 (with libmemcached-0.44 embedded) http://docs.libmemcached.org/memcached_behavior_set.html#memcached_be... and watching with Wireshark (and also reading from individual servers) I only saw a writes to a single server. use Memcached::libmemcached qw(:memcached_behavior_t ); use strict; use warnings; my $memc = Memcached::libmemcached-new; $memc-memcached_server_add( 'localhost', 11211 ); $memc-memcached_server_add( 'localhost', 11212 ); # Also tried specifying 2 for number of replicas $memc-memcached_behavior_set( MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS() = 1 ); $memc-memcached_set( 'test', 1, 10 ) || die 'set failed ' . $memc-errstr; Also, what does MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT do? Does that mean after a server reports that many failures any gets or sets to that server will fail without the library attempting to send to the memcached server? I assume that doesn't mean the server will be pulled out of the pool and keys would get rehashed to new servers. (Although, maybe if MEMCACHED_BEHAVIOR_AUTO_EJECT_HOSTS is set, but if that means keys would hash to different servers that seems like a bad idea.) Yes, I'm looking at poor-man's HA with these two features. Looking at how pulling the plug on one server won't result in get or set failures, and not an increased amount of time (say waiting for a failed server to respond). -- Bill Moseley mose...@hank.org
Re: Memcached-1.4.5 Solaris 10 Error
Hi Felipe, Do note that the memcached build on Solaris works best when using Sun's cc, not gcc. Last I checked, it was still freely available (but required registration). Which version of libevent did you use? Solaris 10 has libevent built in over in the SFW repository, and that one has been well tested. It should be compatible with 1.4.5. I'd really recommend using it if at all possible. Also, note that just to get you going, you can set different envvars to have libevent use different event mechanisms with the OS. I believe both event ports and /dev/poll are supported with Solaris. Hope that helps, Matt On 3/3/11 9:33 AM, Felipe Cerqueira wrote: Hi All, After some problems, i have success in compiling memcached on solaris sparc. libmemcached was very hard to get working too. I have a version here with some patches to detect and implement unsuported features on solaris like getopt_long and etc... Now, its compiled successfuly and working well on solaris sparc 9. On solaris sparc 10, I'm getting the follow error: -bash-3.00$ uname -a SunOS server 5.10 Generic_118833-33 sun4u sparc SUNW,Sun-Fire-V490 -bash-3.00$ ./memcached [warn] ioctl: DP_POLL: Invalid argument [warn] ioctl: DP_POLL: Invalid argument [warn] ioctl: DP_POLL: Invalid argument [warn] ioctl: DP_POLL: Invalid argument [warn] ioctl: DP_POLL: Invalid argument -bash-3.00$ -bash-3.00$ ulimit -a core file size(blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files(-n) 256 pipe size (512 bytes, -p) 10 stack size(kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes(-u) 29995 virtual memory(kbytes, -v) unlimited -bash-3.00$ -bash-3.00$ truss ./memcached ... /1: setsockopt(39, SOL_SOCKET, SO_SNDBUF, 0xFFBFF4FC, 4, SOV_DEFAULT) = 0 /1: setsockopt(39, SOL_SOCKET, SO_SNDBUF, 0xFFBFF4FC, 4, SOV_DEFAULT) Err#132 ENOBUFS /1: bind(39, 0x0003C0A0, 16, SOV_SOCKBSD) = 0 /1: write(9, \0, 1) = 1 /1: write(16, \0, 1) = 1 /1: write(23, \0, 1) = 1 /1: write(30, \0, 1) = 1 /1: pwrite(3, \0\0\004\001\0\0\0\0\0 $.., 24, 0) = 24 /1: ioctl(3, DP_POLL, 0xFFBFF490) Err#22 EINVAL /1: mmap(0x0001, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFEFA [/1: write(2, [, 1) = 1 warn/1: write(2, w a r n, 4) = 4 ] /1: write(2, ] , 2) = 2 ioctl: DP_POLL: Invalid argument/1: write(2, i o c t l : D P _ P O.., 32) = 32 /1: write(2, \n, 1) = 1 /1: lwp_unpark(6) = 0 /6: lwp_park(0x, 0) = 0 /6: lwp_sigmask(SIG_SETMASK, 0xFFBFFEFF, 0xFFF7) = 0xFFBFFEFF [0x] /6: lwp_exit() /1: lwp_wait(6, 0xFFBFF584) = 0 _exit(0) I have look around for some problems on libevent on solaris 10 but cant find any solution. Thanks in advance
Re: libmemcache: memcached_behavior_number_of_replicas working?
On Thu, Mar 3, 2011 at 9:54 AM, Felipe Cerqueira skylaz...@gmail.comwrote: Hi, Its works for me on version 0.45 with C. Well, perhaps I'm not configuring the client correctly. See below. I didnt like of this behavior but its works. The concept of master and slave dont exist. So, if its get a error trying to send data to the hash(key)-server, its dont send data to the others server(s). Are you saying that even with MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS set to 1 if the first server write fails then it won't write to the second? That is, once a server fails only reads work? Here's a simple test script using the most recent Perl version with output below. Am I not setting this up correctly? use Memcached::libmemcached qw( :memcached_behavior_t ); use strict; use warnings; my $memc = Memcached::libmemcached-new; $memc-memcached_behavior_set( MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS() = 1 ); $memc-memcached_server_add( 'localhost', 11211 ); $memc-memcached_server_add( 'localhost', 11212 ); my $mem1 = Memcached::libmemcached-new; $mem1-memcached_server_add( 'localhost', 11211 ); my $mem2 = Memcached::libmemcached-new; $mem2-memcached_server_add( 'localhost', 11212 ); printf Running libmemcached version %s\n, Memcached::libmemcached::memcached_lib_version(); for ( 1 .. 8 ) { my $id = int rand 1000; print \n= key $id =\n; $memc-memcached_set( $id, 1, 10 ) || die 'set failed ' . $memc-errstr . '\n; my ( $flag, $rc ); $mem1-memcached_get( $id, $flag, $rc ) print found in mem1\n; $mem2-memcached_get( $id, $flag, $rc ) print found in mem2\n; } Running this I see -- not it's only finding the key in one server. Running libmemcached version 0.44 = key 869 = found in mem2 = key 321 = found in mem1 = key 618 = found in mem1 = key 234 = found in mem2 = key 667 = found in mem1 = key 54 = found in mem2 = key 441 = found in mem1 = key 178 = found in mem1 One thing I always find odd is if I don't have Memcached running I get: set failed SYSTEM ERROR Success' Does that mean a successful system error? ;) -- Bill Moseley mose...@hank.org
Re: libmemcache: memcached_behavior_number_of_replicas working?
Hi Bill, On Mar 3, 3:54 pm, Bill Moseley mose...@hank.org wrote: On Thu, Mar 3, 2011 at 9:54 AM, Felipe Cerqueira skylaz...@gmail.comwrote: Hi, Its works for me on version 0.45 with C. Well, perhaps I'm not configuring the client correctly. See below. I didnt like of this behavior but its works. The concept of master and slave dont exist. So, if its get a error trying to send data to the hash(key)-server, its dont send data to the others server(s). Are you saying that even with MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS set to 1 if the first server write fails then it won't write to the second? That is, once a server fails only reads work? First of all, replication is only for write. Internally, its wont ask for a second server. About your question, its really wont replicate the data if fails on first server. You can see it on the source code of libmemcached: See: http://goo.gl/JSSro I dont like this behaivor too. Here's a simple test script using the most recent Perl version with output below. Am I not setting this up correctly? use Memcached::libmemcached qw( :memcached_behavior_t ); use strict; use warnings; my $memc = Memcached::libmemcached-new; $memc-memcached_behavior_set( MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS() = 1 ); $memc-memcached_server_add( 'localhost', 11211 ); $memc-memcached_server_add( 'localhost', 11212 ); my $mem1 = Memcached::libmemcached-new; $mem1-memcached_server_add( 'localhost', 11211 ); my $mem2 = Memcached::libmemcached-new; $mem2-memcached_server_add( 'localhost', 11212 ); printf Running libmemcached version %s\n, Memcached::libmemcached::memcached_lib_version(); The replication only works with binary mode. Did you set it up? And, remember, this feature is only for replication data, not consulting (get). I have tested on libmemcached with C. I dont known about perl. for ( 1 .. 8 ) { my $id = int rand 1000; print \n= key $id =\n; $memc-memcached_set( $id, 1, 10 ) || die 'set failed ' . $memc-errstr . '\n; my ( $flag, $rc ); $mem1-memcached_get( $id, $flag, $rc ) print found in mem1\n; $mem2-memcached_get( $id, $flag, $rc ) print found in mem2\n; } Running this I see -- not it's only finding the key in one server. Running libmemcached version 0.44 = key 869 = found in mem2 = key 321 = found in mem1 = key 618 = found in mem1 = key 234 = found in mem2 = key 667 = found in mem1 = key 54 = found in mem2 = key 441 = found in mem1 = key 178 = found in mem1 One thing I always find odd is if I don't have Memcached running I get: set failed SYSTEM ERROR Success' Does that mean a successful system error? ;) -- Bill Moseley mose...@hank.org
Re: Memcached-1.4.5 Solaris 10 Error
Hi Felipe, Am 03.03.2011 um 20:01 schrieb Felipe Cerqueira: thanks for your reply. I'm using libevent-2.0.10-stable. I have compiled everything on solaris 9 with gcc version 3.4.0. Now, trying to resolv the problem, I have download from SFW memcached-1.2.6-sol10-sparc-local.gz. Its too old and depends of libevent 1.2a (older than memcached). rs But, installing from this packages, its works fine. So, I think its a problem with cross compilation of libevent between solaris 9 and 10. I'll compile on solaris 10 and return here the result. I have prepared a full libevent 2.0.10 in both 32 and 64 bit for Solaris 9 and 10, both x86 and Sparc for OpenCSW: http://www.opencsw.org/packages/CSWlibevent2-0-5/ The only critical thing to note is that the tests must be run with a bash instead of /bin/sh. Memcached had some issue which you can see from my build recipe: http://sourceforge.net/apps/trac/gar/browser/csw/mgar/pkg/memcached/trunk Experimental packages for Solaris 9 and additional ones for Solaris 10 x86 with DTrace support both 32/64 bit are available from http://buildfarm.opencsw.org/experimental.html#memcached However, as Sparc on Solaris 10 does not pass the testsuite in 64 bit completely I don't know if there are other issue, so tests are definitely welcome. You can do so easiest by pkgadd -d http://mirror.opencsw.org/opencsw/pkgutil-`uname -p`.pkg /opt/csw/bin/pkgutil -t http://buildfarm.opencsw.org/opencsw/experimental/memcached -i memcached The current version officially distributed from OpenCSW is 1.4.3, also 32/64 bit, which should run fine. It was the last version btw. to compile OOTB on Solaris. It would be nice if you would give them a try and let me know if you encounter any issues. Best regards -- Dago
Replication ?
Hi all, I know I'll get blasted for not googling enough, but I have a quick question. I was under the impression memcached servers replicated data, such that if i have 2 servers and one machine goes down the data would all still be available on the other machine. this with the understanding that some data may not yet have been replicated as replication isn't instantaneous. Can you clarify for me? thx, -nathan
Re: Replication ?
On Thu, Mar 3, 2011 at 2:44 PM, Roberto Spadim robe...@spadim.com.brwrote: more here: http://repcached.lab.klab.org/ This was the first thing that came up on google, just wanted to verify memcached doesn't replicate out-of-the-box. thx, -nathan
Re: Replication ?
On Thu, Mar 3, 2011 at 3:02 PM, Roberto Spadim robe...@spadim.com.brwrote: humm i will tell my experience there´s two ideas one replication (like raid1) other load balance (like raid0 stripe) client side can load balance like 'raid0' (i don´t know if libmemcache have this, in php i use it, and works well) repcache works like 'raid1' (for cluster and HA i´m using it without problems, and it´s very good!) memcachedb have non volatile tools (use harddisk/ssd 'cache' with a berkley db) there´s another memcache port but i don´t remember now, with more features (web administration and others tools) maybe in future we could put all in one package? i don´t know... A friend has mentioned membase: http://www.membase.org/ though I've hardly had a minute to investigate, it implements the memcache protocol, but it sounds like it may use something other than libmemcache on the backend. -nathan
Re: Replication ?
Hi Nathan, On 3/3/11 1:42 PM, Nathan Nobbe wrote: Hi all, I know I'll get blasted for not googling enough, but I have a quick question. Here's a dime. Get yourself a web browser and bring me back $0.10 change. :) (said jokingly...) I was under the impression memcached servers replicated data, such that if i have 2 servers and one machine goes down the data would all still be available on the other machine. this with the understanding that some data may not yet have been replicated as replication isn't instantaneous. There are a few memcached related things that do replication, but the core memcached server itself does not replicate. One is something related (forked?) from memcached called repcache. It does it on the server side as clustered pairs. One is something called Membase, which uses the memcached core (kinda forked, aiming to put back), which has special hashing called vbucket hashing. This can be transparent to the client though. One is that libmemcached does replication from the client, but has admittedly lots of interesting potential consistency issues depending on what fails when and how much it actually fails. I'll let you go further with that browser you just downloaded. :) Matt p.s.: full disclosure: I'm pretty heavily involved in Membase
Re: Replication ?
humm membase = memcache+repcache+memcachedb http://wiki.membase.org/display/membase/Membase+for+Memcached+Users nice =) 2011/3/3 Matt Ingenthron ingen...@cep.net: On 3/3/11 2:13 PM, Nathan Nobbe wrote: A friend has mentioned membase: http://www.membase.org/ though I've hardly had a minute to investigate, it implements the memcache protocol, but it sounds like it may use something other than libmemcache on the backend. It's a bit off topic for this list and should be discussed further over there, but to avoid the kinds of consistency challenges and topology change problems associated with repcached/libmemcached replication, Membase took a very different approach with vbuckets and vbucket hashing. Your browser will find you more. :) Clients don't have to be aware of this, as moxi can abstract away the details if the client doesn't know vbucket hashing. FWIW, Trond Norbye who I think did the libmemcached replication work is also a Membase guy. Regards, Matt -- Roberto Spadim Spadim Technology / SPAEmpresarial
Problems when one of the memcached server down
Hi, I use the .net clients and 2 servers at 2 different machines. When one of the memcached servers is down, I try to set an item to the cache (According the hashing algoritem of the client, this item should stored the inactive server!) and this item get stored at the active server. After I set the item, the second server get back to work. Now if I'll try to get this item I won't succeed, because the client will searche the item at the second server (According the hashing algoritem). What can i do to solve this problem??? (To set the item again at the second server isn't a solution, because I don't want to store any item at 2 servers)
Re: libmemcache: memcached_behavior_number_of_replicas working?
On Thu, Mar 3, 2011 at 11:18 AM, Felipe Cerqueira skylaz...@gmail.comwrote: The replication only works with binary mode. Did you set it up? Oh, no, I guess that would help. I must of missed that in the docs. I was running Memcached 1.2.8, so I just installed 1.4.5 and enabled binary in the Perl code with: $memc-memcached_behavior_set( MEMCACHED_BEHAVIOR_BINARY_PROTOCOL() = 1 ); $memc-memcached_behavior_set( MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS() = 1 ); If I write a sequence of keys using memcached_set( $key, 1 ) in a loop I only see (with memcached -vv) SET on the one server per key (the primary based on how it's hashed). But, then when I exit the program (closing the connection) the LAST key I wrote gets written to the replicant(s). Do I need to flush or something like that after every set call? And, remember, this feature is only for replication data, not consulting (get). The docs seem to indicate otherwise, and indeed if I kill the primary where a key was set and attempt to read then it reads from the replicant. MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS If you just want a poor mans HA, you may specify the numbers of replicas libmemcached should store of each item (on different servers). This replication does not dedicate certain memcached servers to store the replicas in, but instead it will store the replicas together with all of the other objects (*on the 'n' next servers specified in your server list*). MEMCACHED_BEHAVIOR_RANDOMIZE_REPLICA_READ Allows randomizing the replica reads starting point. *Normally the* *read is done from primary server and in case of miss the read is done from primary + 1, then primary + 2 all the way to 'n' replicas*. If this option is set on the starting point of the replica reads is randomized between the servers. This allows distributing read load to multiple servers with the expense of more write traffic. -- Bill Moseley mose...@hank.org
Re: Replication ?
Hi all, I know I'll get blasted for not googling enough, but I have a quick question. I was under the impression memcached servers replicated data, such that if i have 2 servers and one machine goes down the data would all still be available on the other machine. this with the understanding that some data may not yet have been replicated as replication isn't instantaneous. Can you clarify for me? thx, -nathan I sound like a broken record about this, but I like restating things nobody cares about; - memcached doesn't do replication by default - because not replicating your cache gives you 2x cache space - and when you have 10 memcached servers and one fails... - ... you get some 10% miss rate. - and may cache 2x more crap in the meantime. if your workload really requires cache data never disappear, you're looking more for a database (mysql, NoSQL, or otherwise). the original point (and something I still see as a feature) is the ability to elastically add/remove cache space in front of things which don't scale as well or take too much time to process. For everything else there's mastercard^Wredis^Wmembase^Wcassandra^Wsomeotherproduct -Dormando
Re: libmemcache: memcached_behavior_number_of_replicas working?
Hi! On Mar 3, 2011, at 5:12 PM, Bill Moseley wrote: But, then when I exit the program (closing the connection) the LAST key I wrote gets written to the replicant(s). I explained this on the libmemcached mailing list but I will comment do a short comment on it here as well. Libmemcached with replica set to 2 libmemcached will write to the hashed node and additionally to the next node in the list. The replicated key+pair is buffered on the client, so it is not sent out immediately, which is why you might not see it if you immediately go looking (it also means that a client crash could leave you with just the not replicated value in the cache). There are no lets vote to see if they both arrive and ordering could be thrown off to the replica under a number of different circumstances. There are no split brain solutions. This is far from bullet proof, buyer beware. Frankly though? It is probably good enough for the vast majority of problems. If you have greater needs? Go use a database. Cheers, -Brian
Re: libmemcache: memcached_behavior_number_of_replicas working?
Hi! On Mar 3, 2011, at 11:18 AM, Felipe Cerqueira wrote: About your question, its really wont replicate the data if fails on first server. You can see it on the source code of libmemcached: See: http://goo.gl/JSSro I dont like this behaivor too. I'd take a patch to change this behavior (well, add an additional mode). Write up a bug at http://bugs.launchpad.net/libmemcached and I will probably get to it sooner then later. Cheers, -Brian