Have you been running the connection tester tool while observing the client slowdown?
The tool is there so you can rule if your client is an issue or not, ie; if the tool never sees a blip but all/most/some of your clients are seeing blips, it's the client's fault. If the tool sees a blip, you can see exactly where it's getting hung up and further narrow it down. On Mon, 21 Feb 2011, Patrick Santora wrote: > > Its just strange. Memcaced with verbose logging looks ok but the client > machines just take forever to get data. Like in the stats I don't > see anything out of the ordinary. The nic settings look ok too. Quite > frustrating... > > On Feb 21, 2011 11:51 AM, "Patrick Santora" <patwe...@gmail.com> wrote: > > I will need to look at those further today. This weekend went a little > > haywire for me. :) > > On Feb 21, 2011 11:42 AM, "dormando" <dorma...@rydia.net> wrote: > >> Have you walked through those links I gave you? You haven't mentioned > >> exactly what you're seeing and those links walk you through narrowing it > >> down a lot as well as listing a lot of things to look for. > >> > >> On Mon, 21 Feb 2011, Patrick Santora wrote: > >> > >>> Hrmm. Still having issues. Here is the latest stats dump. I also talked > > with my IT person and he mentioned the following setup, which does > >>> not look like an issue? > >>> NIC SETTINGS > >>> the servers should all be autonegotiating to 100/Full and we apply these > > additional kernel tuning parameters > >>> net.core.rmem_max = 16777216 > >>> net.core.wmem_max = 16777216 > >>> net.ipv4.tcp_rmem = 4096 87380 16777216 > >>> net.ipv4.tcp_wmem = 4096 65536 16777216 > >>> > >>> LATEST STATS > >>> STAT pid 1788 > >>> STAT uptime 44811 > >>> STAT time 1298311271 > >>> STAT version 1.4.5 > >>> STAT pointer_size 64 > >>> STAT rusage_user 178.875806 > >>> STAT rusage_system 763.939863 > >>> STAT curr_connections 811 > >>> STAT total_connections 2012 > >>> STAT connection_structures 813 > >>> STAT cmd_get 876886 > >>> STAT cmd_set 74747 > >>> STAT cmd_flush 0 > >>> STAT get_hits 858907 > >>> STAT get_misses 17979 > >>> STAT delete_misses 0 > >>> STAT delete_hits 2 > >>> STAT incr_misses 0 > >>> STAT incr_hits 0 > >>> STAT decr_misses 0 > >>> STAT decr_hits 0 > >>> STAT cas_misses 0 > >>> STAT cas_hits 0 > >>> STAT cas_badval 0 > >>> STAT auth_cmds 0 > >>> STAT auth_errors 0 > >>> STAT bytes_read 17426408671 > >>> STAT bytes_written 180479901035 > >>> STAT limit_maxbytes 536870912 > >>> STAT accepting_conns 1 > >>> STAT listen_disabled_num 0 > >>> STAT threads 4 > >>> STAT conn_yields 0 > >>> STAT bytes 3501518 > >>> STAT curr_items 3230 > >>> STAT total_items 74747 > >>> STAT evictions 0 > >>> STAT reclaimed 20950 > >>> END > >>> > >>> > >>> On Mon, Feb 21, 2011 at 8:12 AM, Patrick Santora <patwe...@gmail.com> > > wrote: > >>> @Dustin > >>> Thanks, I will be disabling them to see if that helps. > >>> > >>> -Pat > >>> > >>> > >>> On Mon, Feb 21, 2011 at 5:59 AM, Dustin <dsalli...@gmail.com> wrote: > >>> > >>> On Feb 21, 12:31 am, Patrick Santora <patwe...@gmail.com> wrote: > >>> > Heh. I had a funny feeling that was going to be the answer. I was > > curious > >>> > mostly because the Binary mode seemed to do quite a deal of good for > >>> > Facebook when it was used. I'm imagining that they cached images so > > binary > >>> > was a good idea, but for simple structures like json, it might not make > > much > >>> > sense. So thought I would get some opinions :). > >>> > >>> binary protocol doesn't make much of a difference wrt what you're > >>> caching, but can help you optimize some access patterns with a > >>> sufficiently smart client. If you're concerned that it may be making > >>> things worse (it probably doesn't have a huge effect from what I'm > >>> hearing here), you can just try disabling it. > >>> > >>> > >>> > >>> > >>> > >
