CAS Value is not replicating in Memcache Master Master Replication
Hi , I am using Memcace server as active active (Master Master ) where two servers will be reciving requests , and applied repcached to those servers for replication What i observed is that the CAS Key stored by XMemcacheClient 1.3.5 is not being replicated among hese servers RIght now this is the way i am using CAS *cas(key, 0, value, casVal )* Please let me know do i need to do any changes for making the cas key replicated
Re: Memcached and Latency
hum, could you reproduce the tests with last 3.6.3 kernel? i'm not sure but i read some changes in latency of kernel maybe this could help too since you tested with 2.6.28 2012/10/22 Rishi : > Hi all, > > I am a graduate student at UC San Diego and we recently presented a paper on > latency bottlenecks in cloud. Our study used Memcached as an example of > latency sensitive application. I want to share some of our results with you > all. > > In our study we found that Memcached application logic is super fast 2-3 > microseconds. This latency number includes time to parse a request, look up > key in hash table and compose reply. We found other bottlenecks in the > system i.e. kernel stack limits Memcached throughput and increases latency. > We observed that kernel contribution to overall latency in cloud is between > 80% to 90%. > > Bypassing the kernel networking stack (i.e. deliver packets directly from > NIC to Memcached application) improves throughput by 5x but still results in > latency variance. We found that pthread lock contention on global hash table > in Memcached results in latency variance and reason being that locks not > only causes serialization but the contested pthread locks need to be > resolved in kernel, thus adding to variance. > > Caveat: We have done this study long back and I haven't tested with latest > Memcached 1.4.15 with Dormando lock fixes. I believe this fix would reduce > or eliminate the issue. > > We found that partitioning the global hash table into multiple buckets (we > use vbuckets feature in Memcached 1.6) and using NIC features to steer > requests directly to thread handling the bucket could result in reduced > latency. > > More details are in the paper. Link: > http://cseweb.ucsd.edu/~rkapoor/papers/rkapoor_chronos.pdf > > > We would welcome any comments/questions/criticisms. > > Thanks, > Rishi Kapoor > UCSD > -- Roberto Spadim
Re: Questions about memcached in general and failover
a point... a query like select * from session_table where session_id='x' where session_id is a primary key is very very fast, we are talking about < 0.1 seconds, with memcached you are trying to go down to <0.01 seconds why should you care about speed in internet if the time to consider a page slow is near 3 seconds? you could use database without problems when memcached goes down and a simple query (in primary key) isn't database intensive, some guys use a mysql table per session when session ends DROP TABLE session_xyz this make the system more proof of broken engine (that don't work fast with frequently delete+update+select queries) other option is a session_table with many partitions, just some ideas.. 2012/10/22 Perrin Harkins : > On Mon, Oct 22, 2012 at 5:09 AM, SAE wrote: >> But as explained above. If one of the memcached servers goes down. Some >> users experience the problem, that they not only get logged out but also >> have problems browsing the site at all or logging back in. Every page needs >> minutes to load then and it seems like the client is still trying to reach >> the server that got down and he's not trying that for only the first few >> clicks after one server went down, but for every click after that. > > That sounds like an issue with your memcached client, rather than the > server. See if there's a mailing list or other support for the client > you're using. > > - Perrin -- Roberto Spadim
Re: Memcached and repcached not replicating 2 master servers
On Mon, Oct 22, 2012 at 2:41 PM, Kiran Kumar wrote: > Hi , > > I have memcached-14.5 and -repcached-2.2 installed on memcached on ubuntu. > They install went great and when i start memcached i get: replication: > connect (peer=server1:11212) replication: marugoto copying replication: > close replication: listen This looks good and for testing purpose i set in a > Server 1 > > set my_key 0 2592000 1 > 5 > > and was able to get this in Server 2 . All this works fine . > > But my question / issue is that , when i set the Data through MemcacheClient > , the data is not being properly replicated (Means sometimes the Data is > reflected and sometimes not ) > > String location = server1:11211 server2:11211 > MemcachedClientBuilder builder = new > XMemcachedClientBuilder(AddrUtil.getAddressMap(location)); > builder.setSessionLocator(new KetamaMemcachedSessionLocator(true)); > > > Please let me know if i need to set anything more to make it work ?? If the memcache client knows about 2 servers it will send half to one and half to the other based on the key hashing. Does repcached replicate both directions? In any case this sounds like an issue with repcached so not something that most people on this list would know. -- Les Mikesell lesmikes...@gmail.com
Memcached and Latency
Hi all, I am a graduate student at UC San Diego and we recently presented a paper on latency bottlenecks in cloud. Our study used Memcached as an example of latency sensitive application. I want to share some of our results with you all. In our study we found that Memcached application logic is super fast 2-3 microseconds. This latency number includes time to parse a request, look up key in hash table and compose reply. We found other bottlenecks in the system i.e. kernel stack limits Memcached throughput and increases latency. We observed that kernel contribution to overall latency in cloud is between 80% to 90%. Bypassing the kernel networking stack (i.e. deliver packets directly from NIC to Memcached application) improves throughput by 5x but still results in latency variance. We found that pthread lock contention on global hash table in Memcached results in latency variance and reason being that locks not only causes serialization but the contested pthread locks need to be resolved in kernel, thus adding to variance. Caveat: We have done this study long back and I haven't tested with latest Memcached 1.4.15 with Dormando lock fixes. I believe this fix would reduce or eliminate the issue. We found that partitioning the global hash table into multiple buckets (we use vbuckets feature in Memcached 1.6) and using NIC features to steer requests directly to thread handling the bucket could result in reduced latency. More details are in the paper. Link: http://cseweb.ucsd.edu/~rkapoor/papers/rkapoor_chronos.pdf We would welcome any comments/questions/criticisms. Thanks, Rishi Kapoor UCSD
Memcached and repcached not replicating 2 master servers
Hi , I have memcached-14.5 and -repcached-2.2 installed on memcached on ubuntu. They install went great and when i start memcached i get: replication: connect (peer=server1:11212) replication: marugoto copying replication: close replication: listen This looks good and for testing purpose i set in a Server 1 *set my_key 0 2592000 1* *5* and was able to get this in Server 2 . All this works fine . But my question / issue is that , when i set the Data through MemcacheClient , the data is not being properly replicated (Means sometimes the Data is reflected and sometimes not ) String location = server1:11211 server2:11211 *MemcachedClientBuilder builder = new XMemcachedClientBuilder(AddrUtil.getAddressMap(location));* * builder.setSessionLocator(new KetamaMemcachedSessionLocator(true));* Please let me know if i need to set anything more to make it work ?? Thanks in advance .
Re: Questions about memcached in general and failover
On Mon, Oct 22, 2012 at 5:09 AM, SAE wrote: > But as explained above. If one of the memcached servers goes down. Some > users experience the problem, that they not only get logged out but also > have problems browsing the site at all or logging back in. Every page needs > minutes to load then and it seems like the client is still trying to reach > the server that got down and he's not trying that for only the first few > clicks after one server went down, but for every click after that. That sounds like an issue with your memcached client, rather than the server. See if there's a mailing list or other support for the client you're using. - Perrin
Memcache is automatically being stopped after one hour approx .
Hi , My Application is Using Memcache installed on two machines 10.1.1.12 and 10.1.1.12.203 on which Memcache server is started the following way memcached -d -u nobody -l 10.33.143.203 -p 11211 -m 64 -x 192.168.13.242 -v net.rubyeye.xmemcached.exception.MemcachedException: Xmemcached is stopped at net.rubyeye.xmemcached.XMemcachedClient.sendCommand(XMemcachedClient.java:247) at net.rubyeye.xmemcached.XMemcachedClient.fetch0(XMemcachedClient.java:573) at net.rubyeye.xmemcached.XMemcachedClient.get0(XMemcachedClient.java:971) at net.rubyeye.xmemcached.XMemcachedClient.gets(XMemcachedClient.java:984) at net.rubyeye.xmemcached.XMemcachedClient.gets(XMemcachedClient.java:1006) at net.rubyeye.xmemcached.XMemcachedClient.gets(XMemcachedClient.java:995) 2012-10-19 12:34:19,789[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:19,797[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:19,805[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:20,146[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211 2012-10-19 12:34:20,149[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211 2012-10-19 12:34:20,152[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211 2012-10-19 12:34:20,155[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:20,156[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211 2012-10-19 12:34:20,158[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:20,159[Thread-21] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211 2012-10-19 12:34:20,160[Thread-21] INFO (AbstractController.java::478)- Controller has been stopped. 2012-10-19 12:34:20,161[Thread-24] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:20,165[Thread-24] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:20,169[Thread-24] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211 2012-10-19 12:34:20,172[Thread-24] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:20,175[Thread-24] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:20,178[Thread-24] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211 2012-10-19 12:34:20,180[Thread-24] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211 2012-10-19 12:34:20,182[Thread-24] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12:11211 2012-10-19 12:34:21,092[ConcurrentSessionMonitorThread_1] WARN (ConcurrentSessionMonitorThread.java::173)- ConcurrentSessionMonitorThread :: END 2012-10-19 12:34:21,197[Thread-24] INFO (AbstractController.java::478)- Controller has been stopped. 2012-10-19 12:34:21,198[xmemcached-read-thread-2-thread-2] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211 2012-10-19 12:34:21,199[xmemcached-read-thread-2-thread-1] WARN (MemcachedConnector.java::348)- Remove a session: 10.1.1.12.203:11211
Re: Questions about memcached in general and failover
First of all I want to thank all of you for your replys and I want to apologize, that I didn't replied earlier, but part of last week I was ill. However, I may have not put enough emphasis on my real problem. Some users losing their sessions is one thing that isn't that pretty, but not the end of the world. But as explained above. If one of the memcached servers goes down. Some users experience the problem, that they not only get logged out but also have problems browsing the site at all or logging back in. Every page needs minutes to load then and it seems like the client is still trying to reach the server that got down and he's not trying that for only the first few clicks after one server went down, but for every click after that.
Starting Memcache with Replication Port
I am starting Memcache in my Linux machine the following way , the second ip below is the replication port , configured with the help of repcached *memcached -d -u nobody -l 10.1.1.101 -p 11211 -m 64 -x 10.1.2.102 -v (Server 1)* and *memcached -d -u nobody -l 10.1.2.102 -p 11211 -m 64 -x 10.1.1.101 -v (Server 2)* But what i am observing is that , after some time hardly one hour , memcached is being automatically stopped on the replication port Please let me know , Is Memcache starting the above way leads to any issues ?? Thanks in advance .
