On Fri, Mar 06, 2009 at 11:49:39AM -0800, Michael Fortson wrote: > Oops, looks like it's actually Gb -> Gb: > http://pastie.org/409653
ah nice ! > Here's a netstat -s: > http://pastie.org/409652 Oh there are interesting things there : - 513607 failed connection attempts => let's assume it was for dead servers - 34784881 segments retransmited => this is huge, maybe your outgoing bandwidth is limited by the provider, causing lots of drops ? - 8325393 SYN cookies sent => either you've been experiencing a SYN flood attack, or one of your listening socket's backlog is extremely small - 1235433 times the listen queue of a socket overflowed 1235433 SYNs to LISTEN sockets ignored => up to 1.2 million times some client socket experienced a drop, causing at least a 3 seconds delay to establish. The errors your scripts detect certainly account for a small part of those. - 2962458 times recovered from packet loss due to SACK data => many losses, related to second point above. Could you post the output of "sysctl -a |grep ^net" ? I think that your TCP syn backlog is very low. Your stats page indicate an average of about 300 sessions/s over the last 24 hours. If your external bandwidth is capped and causes drops, you can nearly saturate the default backlog of 1024 with 300 sessions/s each taking 3s to complete. If you're interested, the latest snapshot will report the number of sess/s in the stats. > Haproxy and nginx are currently on the same box. Mongrels are all on a > private network accessed through eth1 (public access is via eth0). OK. > stats page attached (backend "everything" is not currently in use; > it'll be a use-when-full option for fast_mongrels once we upgrade to > the next haproxy). According to the stats, your avg output bandwidth is around 10 Mbps. Would this match your external link ? Regards, Willy
