Thanks Nicolas and Edward for your help. There was a problem in our code where it tries to set value without key which caused the problems.
TCP resets occurred when the server wants to use the socket for something else or to free up the socket so, it can be used later. On Tuesday, 29 December 2015 14:00:13 UTC+5:30, Nicolas Motte wrote: > > Hi Edward :) > > Maybe I misunderstood Bharathiraja's question. > > If it was a hot key problem, the server hosting the hot key(s) would > receive more traffic (as you explained with the Lady Gaga example), that's > clear. > But how can you be sure it would have more client connections if the > clients are using tcp-keepalive ? > I can understand that there is a higher possibility that the client will > connect to the server with the hot key, but over time the clients should > open a connection to the other servers too (well, of course it depends on > the nature of the traffic, but as there are 100 connections on the other > servers, I assume some keys are used there too). > So even with a hot key, the connection should slowly increase on the other > servers. That why i wanted to know if the 100 connections on each other > server was increasing or was constant. > > What I want to say is that I understand this *could* be a hot key problem, > but can t be 100% for sure without more information. > Moreover the hot key does not explain the TCP resets that are observed > (except if the traffic on the first server is so much bigger that the > queries are time-outing, Bharathiraja would have to check the cpu on the > server) > > Cheers > Nico > > 2015-12-28 9:52 GMT+01:00 Edward Goldberg <e...@edwardmgoldberg.com > <javascript:>>: > >> Raja, >> >> This issue is called the "Hot Key Problem". What happens is that you >> have "One" or more keys are are very popular and used by all ( or most ) of >> the code. >> >> For example: We have a key for each singer, one of the keys is "Lady >> Gaga", the rest are not very interesting ( this is a real example! ). >> >> So all of the posts to Gaga keep the server that is "selected" for GaGa >> Keys active. >> >> The Hash is to select one of the servers for each key, it does not: >> >> 1) Play FAIR with the distribution of Network packets, >> 2) Use the same number of connection for all servers. >> 3) Use the same amount of memory. >> >> So the even spread of the keys over the servers needs to be done by a >> very well selected Key. >> >> If you have a "Hot Key" then you see these issues of "Not equal" for >> memory, cpu, network and connections. >> >> The fix is to use a better Key that tends to have a nice spread over the >> servers. >> >> If that is NOT possible, then use more than one "Pool" of Memcache >> servers as needed to keep these Hot Keys from impacting other keys. >> >> Feel free to ask for more help. I have seen this issue and solved it for >> many projects. >> >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "memcached" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to memcached+...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
