On Sun, Jan 27, 2013 at 06:48:18PM -0800, Nivedita Singhvi wrote: [snip] > > Thanks, but what about this? > > > > pc2 $ nstat -z | grep -i drop > > TcpExtLockDroppedIcmps 0 0.0 > > TcpExtListenDrops 0 0.0 > > TcpExtTCPPrequeueDropped 0 0.0 > > TcpExtTCPBacklogDrop 0 0.0 > > TcpExtTCPMinTTLDrop 0 0.0 > > TcpExtTCPDeferAcceptDrop 0 0.0 > > That seems bogus. > > > > pc2 $ netstat -s | grep -i drop > > 470 outgoing packets dropped > > 5659740 SYNs to LISTEN sockets dropped > > > > Is this normal? > > That's a lot ofconnect requests dropped, but it depends on how > long you've been up and how much traffic you've seen. > > Hmm...you were on an older Ubuntu, right? The netstat source > was patched to translate it as follows: > > + { "ListenDrops", N_("%u SYNs to LISTEN sockets dropped"), opt_number }, > > (see the file debian/patches/CVS-20081003-statistics.c_sync.patch > in the net-tools src)
I have ubuntu 11.10 in all the servers I'm checking except for one with 12.04. Is weird, for the same ubuntu version (same kernel, same netstat, same nstat) I get different outputs. Some have some counters that coincide, some have more counters than other, some have different counters than other... pc121 # nstat -z | grep -i drop TcpExtLockDroppedIcmps 0 0.0 TcpExtListenDrops 0 0.0 TcpExtTCPPrequeueDropped 0 0.0 TcpExtTCPBacklogDrop 0 0.0 TcpExtTCPMinTTLDrop 0 0.0 TcpExtTCPDeferAcceptDrop 0 0.0 pc121 # netstat -s | grep -i drop 470 outgoing packets dropped 5659762 SYNs to LISTEN sockets dropped Other are like this: pc126 # nstat -z | grep -i drop TcpExtLockDroppedIcmps 0 0.0 TcpExtListenDrops 2968982 0.0 TcpExtTCPPrequeueDropped 0 0.0 TcpExtTCPBacklogDrop 0 0.0 TcpExtTCPMinTTLDrop 0 0.0 TcpExtTCPDeferAcceptDrop 0 0.0 pc126 # netstat -s | grep -i drop 2968982 SYNs to LISTEN sockets dropped Other like this: pc127 # nstat -z | grep -i drop TcpExtLockDroppedIcmps 0 0.0 TcpExtListenDrops 0 0.0 TcpExtTCPPrequeueDropped 0 0.0 TcpExtTCPBacklogDrop 0 0.0 TcpExtTCPMinTTLDrop 0 0.0 TcpExtTCPDeferAcceptDrop 0 0.0 pc127 # netstat -s | grep -i drop 1321958 SYNs to LISTEN sockets dropped pc128 # nstat -z | grep -i drop TcpExtLockDroppedIcmps 0 0.0 TcpExtListenDrops 6455507 0.0 TcpExtTCPPrequeueDropped 0 0.0 TcpExtTCPBacklogDrop 0 0.0 TcpExtTCPMinTTLDrop 0 0.0 TcpExtTCPDeferAcceptDrop 0 0.0 pc128 # netstat -s | grep -i drop 6 ICMP packets dropped because they were out-of-window 6455507 SYNs to LISTEN sockets dropped pc130 # nstat -z | grep -i drop TcpExtLockDroppedIcmps 0 0.0 TcpExtListenDrops 0 0.0 TcpExtTCPPrequeueDropped 0 0.0 TcpExtTCPBacklogDrop 0 0.0 TcpExtTCPMinTTLDrop 0 0.0 TcpExtTCPDeferAcceptDrop 0 0.0 pc130 # netstat -s | grep -i drop 3 ICMP packets dropped because they were out-of-window 6728909 SYNs to LISTEN sockets dropped And this is for the one with Ubuntu 12.04: pc106 # nstat -z | grep -i drop TcpExtLockDroppedIcmps 0 0.0 TcpExtListenDrops 2598140 0.0 TcpExtTCPPrequeueDropped 0 0.0 TcpExtTCPBacklogDrop 1711 0.0 TcpExtTCPMinTTLDrop 0 0.0 TcpExtTCPDeferAcceptDrop 0 0.0 TcpExtTCPReqQFullDrop 0 0.0 pc106 # netstat -s | grep -i drop 2598140 SYNs to LISTEN sockets dropped TCPBacklogDrop: 1711 Are this counters hardware-dependant? Or there anything else why they might be different in different servers? > i.e., the netstat pkg iS printing the value of the TCPEXT MIB counter > that's counting TCPExtListenDrops. Then why nstat show that counter in 0 and netstat with what I assume is the right value? > Theoretically, that number should be the same as that printed by nstat, > as they are getting it from the same kernel stats counter. I have not > looked at nstat code (I actually almost always dump the counters from > /proc/net/{netstat + snmp} via a simple prettyprint script (will send > you that offline). Mmm, ok, thanks! > If the nstat and netstat counters don't match, something is fishy. > That nstat output is broken. I using the one from iproute package 20110315-1build1 (except for the one with Ubuntu 12.04, which have 20111117-1ubuntu2). Any ideas on what could be wrong? > > And I don't know how could I narrow down the drops in any way. What I > > know is capturing traffic with tcpdump, I see some packets leaving one > > server but never arriving to the new one. About this, tcpdump should get all the packets received by the NIC, before the kernel have any chance to drop anything, right? > Hmm..do you have a switch between your two end points dropping pkts? I have no idea, I assume there is because the server have only one NIC and they are interconnected to several other servers, so there should be something in the middle, but we have the servers offsite, I can't do any sniffing myself in the middle of the endpoints. > Could be.. Basically, by looking at the statistics kept by each layer, you > should be able to narrow it down a little bit at least. You mean statistics provided by the switch? > It does still sound like some drops are occurring in TCP due to accept > backlog being full and you're overrunning TCP incoming processing (or > at least this contributing), going by that ListenDrops count. If that's so, then I guess you're implying tcpdump don't get the packets before the kernel can drop them. > Sorry, I'm on the road, travelling, and likely not online much this week. No worries! Thanks for the help, is very much appreciated. -- Leandro Lucarella sociomantic labs GmbH http://www.sociomantic.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/