I'm trying to track down the source of what is causing output errors on vlan interfaces for 2 separate physical systems. For example when looking at netstat between 2 different runs the values are always incrementing:
# netstat -s -f inet -I vlan800 && echo && sleep 5 && netstat -s -f inet -I vlan800 Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Colls vlan800 1500 <Link> 00:1c:23:e1:cf:48 187689428 0 148043392 262767 0 Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Colls vlan800 1500 <Link> 00:1c:23:e1:cf:48 187691085 0 148044677 262770 0 .... Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Colls vlan200 1500 <Link> 00:1c:23:e1:cf:48 18139570 0 18645286 40217 0 vlan300 1500 <Link> 00:1c:23:e1:cf:48 2562460 0 3460373 2720 0 vlan500 1500 <Link> 00:1c:23:e1:cf:48 112356993 0 141163651 158443 0 The hardware is 2 Dell PowerEdge 860 servers using the onboard Broadcom BCM5721 NICs. Each system is attached to a different Procurve switch with the 2 onboard NICs in a LACP trunk configuration. The 2 systems are setup in an HA configuration using carp/pf that runs very well. When looking for any type of issues on the switch ports they come back clean for all uplinks to the Dells: Errors (Since boot or last clear) : FCS Rx : 0 Drops Rx : 0 Alignment Rx : 0 Collisions Tx : 0 Runts Rx : 0 Late Colln Tx : 0 Giants Rx : 0 Excessive Colln : 0 Total Rx Errors : 0 Deferred Tx : 0 The same behavior is mimicked on both systems as the counters start incrementing when failing over the carp interfaces between the peers. Another oddity that is the physical interfaces show no output errors just input errors: # netstat -ni Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Colls bge0 1500 <Link> 00:1c:23:e1:cf:48 284587839 103261 120699706 0 0 bge0 1500 fe80::%bge0 fe80::21c:23ff:fe 284587839 103261 120699706 0 0 bge1 1500 <Link> 00:1c:23:e1:cf:48 61755734 193 233219963 0 0 bge1 1500 fe80::%bge1 fe80::21c:23ff:fe 61755734 193 233219963 0 0 ... trunk0 1500 <Link> 00:1c:23:e1:cf:48 346220631 0 353798619 167 0 trunk0 1500 192.168.201 192.168.201.23 346220631 0 353798619 167 0 trunk0 1500 fe80::%trun fe80::21c:23ff:fe 346220631 0 353798619 167 0 ... Any advice would be appreciated on what else to look for that is causing these errors. Regards, Matt Additional info if it helps: # netstat -sn ip: 461996032 total packets received 21 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (duplicates or out of space) 0 malformed fragments dropped 0 fragments dropped after timeout 0 packets reassembled ok 136026433 packets for this host 0 packets for unknown/unsupported protocol 324667376 packets forwarded 550005 packets not forwardable 0 redirects sent 11050218 packets sent from this host 192 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 187636 output datagrams fragmented 187734 fragments created 17589 datagrams that can't be fragmented 0 fragment floods 0 packets with ip length > max ip packet size 0 tunneling packets that can't find gif 0 datagrams with bad address in header 347724405 input datagrams checksum-processed by hardware 356771381 output datagrams checksum-processed by hardware 410308 multicast packets which we don't join icmp: 1171928 calls to icmp_error 0 errors not generated because old message was icmp Output packet histogram: echo reply: 295802 destination unreachable: 1143260 time exceeded: 4 0 messages with bad code fields 0 messages < minimum length 46 bad checksums 4 messages with bad length 64 echo requests to broadcast/multicast rejected Input packet histogram: echo reply: 22496 destination unreachable: 22342 source quench: 17 routing redirect: 172 echo: 295866 time exceeded: 1172 295802 message responses generated igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent ipencap: 57134945 total input packets 35134787 total output packets 0 packets shorter than header shows 0 packets dropped due to policy 0 packets with possibly spoofed local addresses 0 packets were dropped due to full output queue 49913523155 input bytes 4430739176 output bytes 0 protocol family mismatches 0 attempts to use tunnel with unspecified endpoint(s) tcp: 568071 packets sent 380576 data packets (116608967 bytes) 191 data packets (58379 bytes) retransmitted 0 fast retransmitted packets 160818 ack-only packets (253609 delayed) 0 URG only packets 0 window probe packets 225 window update packets 26297 control packets 0 packets hardware-checksummed 665199 packets received 397789 acks (for 116554934 bytes) 29196 duplicate acks 0 acks for unsent data 0 acks for old data 254664 packets (6360099 bytes) received in-sequence 112 completely duplicate packets (17247 bytes) 0 old duplicate packets 0 packets with some duplicate data (0 bytes duplicated) 1792 out-of-order packets (123317 bytes) 0 packets (0 bytes) of data after window 0 window probes 4641 window update packets 2 packets received after close 70 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded for missing IPsec protection 0 discarded due to memory shortage 663599 packets hardware-checksummed 0 bad/missing md5 checksums 0 good md5 checksums 184 connection requests 25814 connection accepts 25907 connections established (including accepts) 27115 connections closed (including 11 drops) 0 connections drained 80 embryonic connections dropped 396180 segments updated rtt (of 357857 attempts) 413 retransmit timeouts 2 connections dropped by rexmit timeout 0 persist timeouts 330 keepalive timeouts 9 keepalive probes sent 66 connections dropped by keepalive 16172 correct ACK header predictions 189395 correct data packet header predictions 68511 PCB cache misses 0 ECN connections accepted 0 ECE packets received 0 CWR packets received 0 CE packets received 0 ECT packets sent 0 ECE packets sent 0 CWR packets sent cwr by fastrecovery: 56 cwr by timeout: 413 cwr by ecn: 0 4 bad connection attempts 25818 SYN cache entries added 0 hash collisions 25814 completed 0 aborted (no space to build PCB) 0 timed out 0 dropped due to overflow 0 dropped due to bucket overflow 4 dropped due to RST 0 dropped due to ICMP unreachable 1 SYN,ACK retransmitted 5 duplicate SYNs received for entries already in the cache 0 SYNs dropped (no route or no space) 56 SACK recovery episodes 119 segment rexmits in SACK recovery episodes 104056 byte rexmits in SACK recovery episodes 2111 SACK options received 121 SACK options sent udp: 12563471 datagrams received 0 with incomplete header 0 with bad data length field 15745905 with bad checksum 223 with no checksum 6389398 input packets hardware-checksummed 0 output packets hardware-checksummed 383238 dropped due to no socket 3840853 broadcast/multicast datagrams dropped due to no socket 0 dropped due to missing IPsec protection 0 dropped due to full socket buffers 18446744073702145091 delivered 8829333 datagrams output 7670323 missed PCB cache esp: 57135654 input ESP packets 35134787 output ESP packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 257 packets for which no TDB was found 0 input packets that failed to be processed 0 packets with bad encryption received 0 packets that failed verification received 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 2 possibly replayed packets received 0 packets with bad payload size or padding received 450 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed crypto processing 0 input UDP encapsulated ESP packets 0 output UDP encapsulated ESP packets 0 UDP packets for non-encapsulating TDB received 51434523840 input bytes 4430739176 output bytes ah: 0 input AH packets 0 output AH packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets that failed verification received 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 possibly replayed packets received 0 packets with bad authenticator length received 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed crypto processing 0 input bytes 0 output bytes etherip: 0 packets shorter than header shows 0 packets were dropped due to full output queue 0 packets were dropped because of no interface/bridge information 0 packets dropped due to policy 0 packets dropped for other reasons 0 input ethernet-in-IP packets 0 output ethernet-in-IP packets 0 input bytes 0 output bytes ipcomp: 0 input IPCOMP packets 0 output IPCOMP packets 0 packets from unsupported protocol families 0 packets shorter than header shows 0 packets dropped due to policy 0 packets for which no TDB was found 0 input packets that failed to be processed 0 packets for which no XFORM was set in TDB received 0 packets were dropped due to full output queue 0 packets where counter wrapping was detected 0 packets attempted to use an invalid TDB 0 packets got larger than max IP packet size 0 packets that failed (de)compression processing 0 packets less than minimum compression length 0 input bytes 0 output bytes carp: 172 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for unknown vhid 0 discarded because of a bad address list 3636624 packets sent (IPv4) 0 packets sent (IPv6) 123 send failed due to mbuf memory error 17 transitions to master pfsync: 8184881 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for bad ttl 0 packets shorter than header 0 packets discarded for bad version 0 packets discarded for bad HMAC 0 packets discarded for bad action 0 packets discarded for short packet 0 states discarded for bad values 4 stale states 2217516 failed state lookup/inserts 17209508 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 send error divert: 0 total packets received 0 dropped due to no socket 0 dropped due to full socket buffers 0 packets output 0 errors pflow: 189470 flows sent 8044 packets sent 0 send failed due to mbuf memory error 0 send error ip6: 1233164 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (duplicates or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 0 redirects sent 136 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 0 multicast packets which we don't join Input packet histogram: hop by hop: 19144 UDP: 1187768 ICMP6: 26252 Mbuf statistics: 0 one mbufs 1233164 one ext mbufs 0 two or more ext mbufs 0 tunneling packets that can't find gif 0 packets discarded due to too many headers 0 failures of source address selection source addresses on an outgoing I/F 1 node-local 17 link-locals source addresses of same scope 1 node-local 17 link-locals 0 forward cache hit 0 forward cache miss divert6: 0 total packets received 0 dropped due to no socket 0 dropped due to full socket buffers 0 packets output 0 errors icmp6: 1233181 calls to icmp6_error 0 errors not generated because old message was icmp6 or so 0 errors not generated because of rate limitation Output packet histogram: multicast listener report: 104 multicast listener done: 4 neighbor solicitation: 11 neighbor advertisement: 17 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 1233181 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes pim6: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 messages received with bad version 0 registers received 0 bad registers received 0 registers sent rip6: 0 messages received 0 checksum calculations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output
