Just adding a bit more information inline:
On Mon, 4 Jul 2011, Charles Sprickman wrote:
Hello,
We're running a few 8.1-R servers with Broadcom bce interfaces (Dell R510)
and I'm seeing occasional packet loss on them (enough that it trips nagios
now and then). Cabling seems fine as neither the switch nor the sysctl info
for the device show any errors/collisions/etc, however there is one odd one,
which is "dev.bce.1.stat_IfHCInBadOctets: 539369". See [1] below for full
sysctl output. The switch shows no errors but for "Dropped packets 683868".
pciconf output is also below. [2]
By default, the switch had flow control set to "on". I also let it run with
"auto". In both cases, the drops continued to increment. I'm now running
with flow control off to see if that changes anything.
With flow control set to "off" on the switch I am still seeing dropped
packets. The number shown by the switch after resetting the counters
tracks "dev.bce.1.stat_IfHCInBadOctets" very closely. Neither the switch
nor the host (in netstat) are reporting any errors, collisions, etc.
I'm seeing less correlation between cpu utilization and drops, which is
unfortunate. In fact, looking at all the graphs I have, such as disk
activity, system load, traffic on the interface and a handful of others,
I'm seeing no strong correlation in the last 24 hours or so between any
system activity and the drops.
Google isn't turn up much more than source code hits on
"stat_IfHCInBadOctets", and from what I can tell, this is something that's
just reading something off the card, so I'm having little luck figuring
out just what a "bad octet" is in this context.
Thanks,
Charles
I do see some correlation between cpu usage and drops - I have cpu usage
graphed in nagios and cacti is graphing the drops on the dell switch. There's
no signs of running out of mbufs or similar.
So given that limited info, is there anything I can look at to track this
down? Anything stand out in the stats sysctl exposes? Two things are
standing out for me - the number of changes in bce regarding flow control
that are not in 8.1, and the correlation between cpu load and the drops.
What other information can I provide?
Thanks,
Charles
[1] [root@h23 /home/spork]# sysctl -a |grep bce.1
dev.bce.1.%desc: Broadcom NetXtreme II BCM5716 1000Base-T (C0)
dev.bce.1.%driver: bce
dev.bce.1.%location: slot=0 function=1
dev.bce.1.%pnpinfo: vendor=0x14e4 device=0x163b subvendor=0x1028
subdevice=0x02f1 class=0x020000
dev.bce.1.%parent: pci1
dev.bce.1.l2fhdr_error_count: 0
dev.bce.1.mbuf_alloc_failed_count: 282
dev.bce.1.fragmented_mbuf_count: 2748
dev.bce.1.dma_map_addr_rx_failed_count: 0
dev.bce.1.dma_map_addr_tx_failed_count: 5
dev.bce.1.unexpected_attention_count: 0
dev.bce.1.stat_IfHcInOctets: 62708651108
dev.bce.1.stat_IfHCInBadOctets: 539369
dev.bce.1.stat_IfHCOutOctets: 434264587173
dev.bce.1.stat_IfHCOutBadOctets: 0
dev.bce.1.stat_IfHCInUcastPkts: 533441918
dev.bce.1.stat_IfHCInMulticastPkts: 3108746
dev.bce.1.stat_IfHCInBroadcastPkts: 1314905
dev.bce.1.stat_IfHCOutUcastPkts: 640961970
dev.bce.1.stat_IfHCOutMulticastPkts: 26
dev.bce.1.stat_IfHCOutBroadcastPkts: 8909
dev.bce.1.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.1.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.1.stat_Dot3StatsFCSErrors: 0
dev.bce.1.stat_Dot3StatsAlignmentErrors: 0
dev.bce.1.stat_Dot3StatsSingleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.1.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.1.stat_Dot3StatsLateCollisions: 0
dev.bce.1.stat_EtherStatsCollisions: 0
dev.bce.1.stat_EtherStatsFragments: 0
dev.bce.1.stat_EtherStatsJabbers: 0
dev.bce.1.stat_EtherStatsUndersizePkts: 0
dev.bce.1.stat_EtherStatsOversizePkts: 0
dev.bce.1.stat_EtherStatsPktsRx64Octets: 34048797
dev.bce.1.stat_EtherStatsPktsRx65Octetsto127Octets: 431844366
dev.bce.1.stat_EtherStatsPktsRx128Octetsto255Octets: 25946173
dev.bce.1.stat_EtherStatsPktsRx256Octetsto511Octets: 39936369
dev.bce.1.stat_EtherStatsPktsRx512Octetsto1023Octets: 2296565
dev.bce.1.stat_EtherStatsPktsRx1024Octetsto1522Octets: 3931392
dev.bce.1.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
dev.bce.1.stat_EtherStatsPktsTx64Octets: 60122571
dev.bce.1.stat_EtherStatsPktsTx65Octetsto127Octets: 221041349
dev.bce.1.stat_EtherStatsPktsTx128Octetsto255Octets: 40177071
dev.bce.1.stat_EtherStatsPktsTx256Octetsto511Octets: 24099944
dev.bce.1.stat_EtherStatsPktsTx512Octetsto1023Octets: 44493532
dev.bce.1.stat_EtherStatsPktsTx1024Octetsto1522Octets: 251036438
dev.bce.1.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
dev.bce.1.stat_XonPauseFramesReceived: 61778
dev.bce.1.stat_XoffPauseFramesReceived: 76315
dev.bce.1.stat_OutXonSent: 0
dev.bce.1.stat_OutXoffSent: 0
dev.bce.1.stat_FlowControlDone: 0
dev.bce.1.stat_MacControlFramesReceived: 0
dev.bce.1.stat_XoffStateEntered: 0
dev.bce.1.stat_IfInFramesL2FilterDiscards: 145832
dev.bce.1.stat_IfInRuleCheckerDiscards: 0
dev.bce.1.stat_IfInFTQDiscards: 0
dev.bce.1.stat_IfInMBUFDiscards: 0
dev.bce.1.stat_IfInRuleCheckerP4Hit: 4448215
dev.bce.1.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.1.stat_CatchupInFTQDiscards: 0
dev.bce.1.stat_CatchupInMBUFDiscards: 0
dev.bce.1.stat_CatchupInRuleCheckerP4Hit: 0
dev.bce.1.com_no_buffers: 0
[2] pciconf -lvb
bce1@pci0:1:0:1: class=0x020000 card=0x02f11028 chip=0x163b14e4
rev=0x20 hdr=0x00
vendor = 'Broadcom Corporation'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xdc000000, size 33554432,
enabled
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
