Ok, thx for the figures. The only thing I could see to explain such a big difference is that most of your keys are hosted on the first server (~54%) If I m correct, the client opens a connection to a server only when it needs to retrieve a key from it (then the connection stays open). But if this is your problem, then the connections on other servers should increase slowly..
Can you tell me how many keys you have on each of your server ? (you can have a look here: http://lzone.de/cheat-sheet/memcached if you don t already monitor your cluster. The command is "stat" and the metric is "curr_items") Cheers Nico 2015-12-28 5:20 GMT+01:00 Bharathiraja P <pbrthemas...@gmail.com>: > Yes. First server always has around 2k connections. Other hosts have <100 > > Connection resets also happens with the first server. > > All the 18 servers have same configurations and tcp_ip settings. > > On Saturday, 26 December 2015 22:07:24 UTC+5:30, Nicolas Motte wrote: >> >> Hi Bharathiraja. >> >> TCP reset can have many causes,not easy to troubleshoot remotely. >> An easy one is a connection timeout (look at the diagram here: >> http://www.cs.northwestern.edu/~agupta/cs340/project2/TCPIP_State_Transition_Diagram.pdf >> ) >> If it is the case, using tcp keepalive should solve this problem. >> But if it was a connection timeout, i guess you would have this problem >> on every node.. >> You might have to investigate a bit further to find the root cause >> (different clients having the same IP maybe..) >> >> But I m not sure to understand now, >> do you have more connections on your first server all the time (so you >> monitor simply the number of opened connections every minute) ? >> or >> do you have more re-connections on your first server (so you monitor the >> number of new connections on each server) ? >> >> And how many connections and servers are we talking about exactly ? >> >> 2015-12-26 16:17 GMT+01:00 Bharathiraja P <pbrthe...@gmail.com>: >> >>> Thanks Nicolas for your reply. >>> >>> We don't create new connection every time. I checked with tcpdump and I >>> see more tcp resets happening on first host when compared to others. >>> >>> On Fri, Dec 25, 2015 at 10:00 PM, Nicolas Motte <lingu...@gmail.com> >>> wrote: >>> >>>> In some client libraries you can choose the hashing function, that will >>>> determine the distribution of your keys among the farm. >>>> For instance, in the C client, you have a dozen of hashing functions: >>>> http://docs.libmemcached.org/hashkit_functions.html >>>> There is no silver bullet, you need to study which one corresponds >>>> better to your data set. >>>> I couldn t find the equivalent in Cache:Memcached, but I guess there >>>> is... >>>> >>>> Even though this will change the distribution of your keys, that should >>>> not change the number of connections. >>>> Each of your clients should keep an open connection to each node of the >>>> farm. Do you keep your connections opened or do you re-create them for >>>> each call? >>>> Could you also check if your keys are correctly balanced across your >>>> farm? >>>> >>>> Cheers >>>> Nico >>>> >>>> 2015-12-25 17:00 GMT+01:00 Bharathiraja P <pbrthe...@gmail.com>: >>>> >>>>> Hi, >>>>> >>>>> We have 18 memcache servers and we use Cache::Memcached client to >>>>> store and retrieve values. Problem we see is, first server in the list >>>>> gets >>>>> more connections when compared to others. Any idea how to fix this? >>>>> >>>>> -- >>>>> Raja >>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "memcached" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to memcached+...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "memcached" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/memcached/onKftIeaZ18/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> memcached+...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "memcached" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to memcached+...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
