RE: dev.bce.3.mbuf_alloc_failed_count increases permanently
> sysctl -a | g bce.3|g -vE '(%|stat)'; echo; sleep 10; sysctl -a | g bce.3| > g -vE '(%|stat)'; echo; netstat -m > > dev.bce.3.l2fhdr_error_count: 0 > dev.bce.3.mbuf_alloc_failed_count: 2098854 > dev.bce.3.mbuf_frag_count: 2655285 > dev.bce.3.dma_map_addr_rx_failed_count: 0 > dev.bce.3.dma_map_addr_tx_failed_count: 57 > dev.bce.3.unexpected_attention_count: 0 > dev.bce.3.com_no_buffers: 0 > > dev.bce.3.l2fhdr_error_count: 0 > dev.bce.3.mbuf_alloc_failed_count: 2098856 > dev.bce.3.mbuf_frag_count: 2655288 > dev.bce.3.dma_map_addr_rx_failed_count: 0 > dev.bce.3.dma_map_addr_tx_failed_count: 57 > dev.bce.3.unexpected_attention_count: 0 > dev.bce.3.com_no_buffers: 0 > > > Any suggestions? What is the reason of this? It's normal in a system under load, the kernel can't always allocate memory when requested by the driver. The result is that RX frames will be dropped as the driver reuses an existing mbuf, a response taken by many other drivers. If you notice rapid increases during certain system operations then you should consider increasing the amount of system memory. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: dev.bce.3.mbuf_alloc_failed_count increases permanently
> > > dev.bce.3.l2fhdr_error_count: 0 > > > dev.bce.3.mbuf_alloc_failed_count: 2098856 > > > dev.bce.3.mbuf_frag_count: 2655288 > > > dev.bce.3.dma_map_addr_rx_failed_count: 0 > > > dev.bce.3.dma_map_addr_tx_failed_count: 57 > > > dev.bce.3.unexpected_attention_count: 0 > > > dev.bce.3.com_no_buffers: 0 > > > > > > > > > Any suggestions? What is the reason of this? > > > > It's normal in a system under load, the kernel can't always > > allocate memory when requested by the driver. The result > > is that RX frames will be dropped as the driver reuses an > > existing mbuf, a response taken by many other drivers. > > > > If you notice rapid increases during certain system operations > > then you should consider increasing the amount of system > > memory. > > > > Thanks for you answer, Dave. > > What do you mean under "systems memory"? Is it the physical memory or the > virtual one? Virtual memory is likely sufficient in this case, though more frequent swapping may cause an equivalent performance loss to the dropped network traffic (i.e. you may be swapping one performance bottleneck for another). The counter "dma_map_addr_*" is incremented when the OS cannot map an MBUF for DMA access. If you see that incrementing as rapidly then you should definitely look at adding physical memory. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) on the Dell PE 2950
> | > Specifically, we've seen that newer (9 and higher) have issues with > | > the doing initial setup and negotiation and that Dell did indeed > | > release newer firmware to fix the issues. This requires a full > | > reboot into Linux (probably centos6) to get the update to execute. > | > | I could be wrong but I *think* that the firmware is loaded on device > | initialization, so there may be a chance that the driver can do it > | when starting? Additionally, maybe one can leverage the kernel > | firmware(9) framework so that the firmware can be more easily updated > | by user? The 5706/5708/5709/5716 devices have a set of RISC processors. One is loaded from NVRAM based firmware (MCP) while the others are loaded by firmware embedded in the driver. The MCP firmware is loaded at device power-on when the system is plugged into a socket in order to provide pre-OS access to the system (think Dell DRAC). There's a chicken and egg problem with your suggestion for firmware(9) support of the MCP firmware. Not only does MCP firmware provide pre-OS services but it also provides FreeBSD driver services. There are shared resources on the device which require synchronization between multiple FreeBSD driver instances and the MCP acts as an arbitrator for them. Additionally, I have some security concerns about making NVRAM write functionality easily accessible through the driver since it would be fairly easy to take down a system and require a motherboard swap to fix the problem. There's always a concern that a power-event could interrupt an NVRAM upgrade and make the system unusable so I think it's important to have the system administrator's blessings before making any NVRAM changes to firmware. Not sure how to accomplish that without making the driver load process interactive. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce packet loss
> You had 282 RX buffer shortages and these frames were dropped. This > may explain why you see occasional packet loss. 'netstat -m' will > show which size of cluster allocation were failed. > However it seems you have 0 com_no_buffers which indicates > controller was able to receive all packets destined for this host. > You may host lost some packets(i.e. non-zero mbuf_alloc_failed_count) > but your controller and system was still responsive to the network > traffic. > > Data sheet says IfHCInBadOctets indicates number of octets received > on the interface, including framing characters for packets that > were dropped in the MAC for any reason. The IfHcInBadOctets counter says the controller received X bytes that were bad on the wire (collisions, FCS errors, etc.). A value of 539,369 would equal about 355 frames @ 1518 bytes per frame. How bad that is really depends on the amount of time the server was running. The minimum bit-error rate (BER) for 1000Base-T is 10^-12, so running at line rate you'd expect to see an error very 1000 seconds according to the following link: http://de.brand-rex.com/LinkClick.aspx?fileticket=TFxnnLPedAg%3D&tabid=1956&mid=5686 Most vendors design to greater than 10^-12 and you're probably not running at line rate all the time so you should see fewer errors. In my testing I can go for days without seeing any errors, but if you run long enough or have marginal interconnects/cabling the error rate will rise. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce packet loss
> > > Data sheet says IfHCInBadOctets indicates number of octets received > > > on the interface, including framing characters for packets that > > > were dropped in the MAC for any reason. > > > > The IfHcInBadOctets counter says the controller received X bytes > > that were bad on the wire (collisions, FCS errors, etc.). A value > > I thought that too. But other counters such as FCS, FAE, Collisions, > Jabbers were all zero. > Also includes frames dropped due to an incorrect destination MAC address (i.e. perfect MAC filter mismatch). This means the NIC received a unicast frame for a MAC address that doesn't match the NIC's MAC address. Very common on a hub and occurs occasionally on switches as well. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce packet loss
> > Any thoughts on that? It's the only thing that differs between the > two > > switches. > > > > This makes me think possibility of duplex mismatch between bce(4) > and link partner. You should not use forced media configuration on > 1000baseT link. If you used manual media configuration on bce(4) > and link partner used auto-negotiation, resolved duplex would be > half-duplex. It's standard behavior and Duplex mismatch can cause > strange problems. > I would check whether link partner also agrees on the resolved > speed/duplex of bce(4). Forced link speed at 1000Mbps is not supported by the IEEE specification, you MUST use auto-negotiation at 1000Mbps (though you can advertise support for 1000Mb only to simulate forced operation). Duplex mismatch usually manifests with collision/deferred transmission errors on the link partner configured for half-duplex. The bce(4) driver does support those statistics. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce packet loss
> I was able to reproduce the drops in very large numbers on the internal > network today. I simply scp'd some large files from 1000/FD hosts to a > 100/FD host (ie: scp bigfile.tgz oldhost.i:/dev/null). Immediately the > 1000/FD hosts sending the files showed massive amounts of drops on the > switch. This makes me suspect that this switch might be garbage in that > it doesn't have enough buffer space to handle sending large amounts of > traffic from the GigE ports to the FE ports without randomly dropping > packets. Granted, I don't really understand how a "good" switch does > this > either, I would have thought tcp just took care of throttling itself. If you have flow control enabled end-to-end I wouldn't expect to see such behavior, frames should not be dropped. If you're seeing drops at the switch then I'd suspect that the traffic source connected to that switch doesn't honor flow control. Check if either the switch or traffic source keeps statistics on flow control frames generated/received. > Bear in mind that on the external switch our port to our ISP, which is > the > destination of almost all the traffic, is 100/FD and not 1000/FD. > > This of course does not explain why the original setup where I'd locked > the switch ports and the host ports to 100/FD showed the same behavior. > > I'm stumped. > > We are running 8.1, am I correct in that flow control is not implemented > there? We do have an 8.2-STABLE image from a month or so ago that we > are > testing with zfs v28, might that implement flow control? Flow control will depend on the NIC driver implementation. Older versions of the bce(4) firmware will rarely generate pause frames (frames would be dropped by firmware but statistics should show the frame drop occurring) and should always honor pause frames from the link partner when flow control is enabled. > > Although reading this: > > http://en.wikipedia.org/wiki/Ethernet_flow_control > > It sounds like flow control is not terribly optimal since it forces the > host to block all traffic. Not sure if this means drops are eliminated, > reduced or shuffled around. When congestion is detected the switch should buffer up to a certain limit (say 80% of full) and then start sending pause frames to avoid dropping frames. This will affect all hosts connecting through the switch so congestion at one host can spread to other hosts (see http://www.ieee802.org/3/cm_study/public/september04/thaler_3_0904.pdf). Small networks with a few hosts should be OK with flow control but if you have dozens of switches and hundreds of hosts then it's not a good idea. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce packet loss
> I'm running 8.1 and at least on the bce hosts, it looks like flow > control > isn't supported, it was added on 4/30/2010: > > http://svnweb.freebsd.org/base/head/sys/dev/bce/if_bce.c?r1=206268&r2=20 > 7411 > > In my 8.1 sources I still see this comment, which was removed in the > above > commit: > /* ToDo: Enable flow control support in brgphy and bge. */ This really applies to whether the user can set flow control manually. By default the NIC should auto-negotiate link speed and flow-control which is the most common case. For example, you can't set RX flow control and disable TX flow control with ifconfig using the current implementation, though it is possible in Linux with ethtool. > So at least on the bce hosts (and bge it seems), I do not have flow > control available on the NIC. Flow control will be set according to auto-negotiation results. For most cases that means flow control will be enabled since both sides normally support it. > The sysctl stats do show that it's > received > "XON/XOFF" frames, which I assume are flow control messages, but there's > no indication that the NIC does anything with them. There won't be any indication in the driver since flow control is managed in hardware. You'd need a wire capture to see that bce(4) has stopped sending frames in response to receiving an XOFF flow control frame or started sending frames in response to receiving an XON flow control frame. > >> We are running 8.1, am I correct in that flow control is not > implemented > >> there? We do have an 8.2-STABLE image from a month or so ago that we > >> are > >> testing with zfs v28, might that implement flow control? > > > > Flow control will depend on the NIC driver implementation. Older > > versions of the bce(4) firmware will rarely generate pause frames > > (frames would be dropped by firmware but statistics should show > > the frame drop occurring) and should always honor pause frames > > from the link partner when flow control is enabled. > > I think my nics probably lack it. I am also guessing that if any > high-traffic host ignores flow control frames, that's going to screw up > other hosts as well since the one causing the buffers to fill is not > going > to throttle and the overflow will continue, correct? Flow control is asymmetric and operates independently in both directions. If the traffic source ignores flow control frames or did not auto-negotiate flow control then it can certainly overwhelm the switch or traffic sink's buffers, causing frame drop and retransmits. > > >> > >> Although reading this: > >> > >> http://en.wikipedia.org/wiki/Ethernet_flow_control > >> > >> It sounds like flow control is not terribly optimal since it forces > the > >> host to block all traffic. Not sure if this means drops are > eliminated, > >> reduced or shuffled around. Frame drops should be eliminated, though congestion could spread upstream to other devices which don't have flow control and result in frame drops and retransmits there. > > When congestion is detected the switch should buffer up to a certain > > limit (say 80% of full) and then start sending pause frames to avoid > > dropping frames. This will affect all hosts connecting through the > > switch so congestion at one host can spread to other hosts (see > > > http://www.ieee802.org/3/cm_study/public/september04/thaler_3_0904.pdf). > > Wow. I did not catch that. I do recall something about the flow > control > frames being multicast - so every host gets them and pauses. That's... > interesting, isn't it? Pause frames are multicast frames but they are only transmitted between link partners (NIC to switch) and never sent further in the network. Flow control is intended to be a local behavior but the link indicates it can have an unintended global effect. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce packet loss
> > There won't be any indication in the driver since flow control > > is managed in hardware. You'd need a wire capture to see that > > bce(4) has stopped sending frames in response to receiving an > > XOFF flow control frame or started sending frames in response > > to receiving an XON flow control frame. > > Ah. I was hoping for something in the ifconfig output. I'll see if > tcpdump and wireshark can tell me anything about this host. > Flow control frames are consumed by the MAC and aren't passed to the OS stack so you won't see them with a host based packet capture application like tcpdump. You'd need something like a JDSU Xgig protocol analyzer to see them on the wire. > One the one host (w/bce) I just set to full auto, the switch claims to > have negotiated 1000FD w/flow control (this specifically shows as > "auto+enabled" on the switch side). > > I see that the "sysctl dev.bce.1" tree has some info, and I can see that > the NIC is receiving flow control frames: > > dev.bce.1.stat_XonPauseFramesReceived: 16638 > dev.bce.1.stat_XoffPauseFramesReceived: 17239 > > These lines are a bit puzzling though: > > dev.bce.1.stat_FlowControlDone: 0 > dev.bce.1.stat_XoffStateEntered: 0 Auto-negotiation occurs between PHYs and then the link comes up. When link is up bce(4) will configure the MAC based on the auto-neg results returned by brgphy(4) (the switch should follow a similar procedure on its side). Flow control occurs at the MAC level. It looks like the MAC is not configured to support flow control as negotiated by the PHY during auto-neg, resulting in flow control frames being ignored. The driver will set this in the MAC based on PHY results: http://fxr.watson.org/fxr/source/dev/bce/if_bce.c#L2046 Either the PHY results are coming back incorrectly or it may be firmware on bce(4) is disabling the behavior even though bce(4) correctly enables it. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: New bxe(4) Driver for Broadcom BCM57710 & BCM57711 Devices
> Thanks for the drivers, but at work we have some issues w/ the driver > on an HP Blade. We could do some basic pings, but when trying to > pass DNS traffic, it failed. Turning off both rxcsum and txcsum made > the driver work. If I only turned one or the other off, DNS traffic > would still not work. Are you using Flex-10? If so, can you tell me how the 57711E is configured? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) with IPMI [puzzling and puzzling]
> > so, I cracked open my R410 this morning to see what the Ethernet > chipset > > had for a indentification mark. It is definitely stamped as a > BCM5716, > > so I'm confused. > > > > bce(4) has code to handle the 5716 chipset, but its never being > executed > > AFAIK. What's going on here? > > > > I guess BCE_CHIP_NUM() returns 0x5709 for 5709 and 5716 > controllers. I don't know why bce(4) explicitly checks 5716 from > the BCE_MISC_ID register. Public data sheet indicates CHIP_NUM bits > have 0x5909 for 5716(NetXtremeII-PG203-R pp229). > To me 5716 is virtually the same as 5709 except for not supporting > TOE and iSCSI offloading. > That's a typo then, 5709 and 5716 share the same ASIC ID. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) with IPMI
> > Can't explain either but probably stable/6 bce(4) may have used old > > firmware. > > Ok, I can once again reach the IPMI controller if I remove this: > > http://svnweb.freebsd.org/base/head/sys/dev/bce/if_bce.c?r1=210263&r2=21 > 0262&pathrev=210263 > > Since the driver has control over the interface, not "upping" the > interface media causes the IPMI controller to not be able to access the > network. Ugh. > That's the current design. We could try moving bce_pulse() initialization from bce_attach() to bce_init() so the driver only takes control when you "up" the interface. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) with IPMI [puzzling and puzzling]
> Whoa ... ok, so now I've run into something horrible. > > According to dmidecode, a Dell R410 has a Broadcom 5716 chipset on it. > On Board Device 2 Information > Type: Ethernet > Status: Enabled > Description: Embedded Broadcom 5716 NIC 1 > On Board Device 3 Information > Type: Ethernet > Status: Enabled > Description: Embedded Broadcom 5716 NIC 2 > > > However, when I ask the chipset what type it is via: > if_bcereg.h:#define BCE_CHIP_NUM(sc) > > It clearly indicates that it is a 5709 Not horrible, expected. The two chips are virtually indistinguishable by software except for the PCI device ID. The two chips differ in that BCM5709 supports TCP/IP and iSCSI offload in Windows while the BCM5716 doesn’t. For the FreeBSD driver that really only means a different string is printed out, all other code remains the same. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) with IPMI
> Because driver already initialized CPUs and context I thought > moving bce_pulse() to bce_init() was too late. If that's doable it > would be more correct way to address this issue. > Moving bce_pulse() is a minimum. It might be necessary to move some other code out of bce_attach()/bce_detach() and relocate it to bce_init()/bce_stop() so that RX/TX MAC operations aren't interrupted. I've not seen it done that way in other Broadcom drivers so I wouldn't expect it to be a trivial effort. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) with IPMI [puzzling and puzzling]
> > That's a typo then, 5709 and 5716 share the same ASIC ID. > > > > Thanks for confirmation. > Could you review the attached patch? Looks good. ASIC ID matches my Dell R210 system: bce0: mem 0xda00-0xdbff irq 16 at device 0.0 on pci2 miibus0: on bce0 bce0: Ethernet address: b8:ac:6f:87:95:f1 bce0: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.2); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.8) Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) with IPMI [puzzling and puzzling]
> Thanks for testing. > Committed with r226123. > I also noticed that bce(4) may support BCM5716S because brgphy(4) > already supports BCM5709S. Can you test attached patch on BCM5716S? I don't have a 5716S system or NIC for testing. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) with IPMI
> > I attempted to disable IPMI via the Dell BIOS tool (DRAC) and I
> couldn't
> > see any driver detection of this status. So, when I add this:
> >
> > > if ((ifp->if_flags & IFF_UP) == 0 &&
> > >(sc->bce_flags & BCE_MFW_PRESENT_FLAG) == 0){
> > > printf("%s: BCE detected MFW not present\n",
> __func__);
> >
> >
> > I don't see any change in behavior with IPMI enabled or disabled via
> the
> > Dell "DRAC" ROM Menu.
> >
>
> I thought you may have seen "MFW(NOT RUNNING)" if management
> firmware is present and you disabled IPMI in device attach time.
> You may have to enable bootverbose to see it.
> If management firmware is present and running you may have seen
> actual management firmware version string.
The Broadcom MFW is configured to load/not load through an NVRAM
option that is likely not affected by the iDRAC BIOS settings.
To disable MFW you'd need to run the user diagnostic utility
(UXDIAG.EXE) with the following command line:
C:\>uxdiag -mfw 0
To turn it back on run the following:
C:\>uxdiag -mfw 1
You can find a bootable CD with the user diagnostic here:
http://www.broadcom.com/support/license.php?file=NXII/Xdiag-5.2.2.iso
Dave
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) with IPMI
> Ran this on my Dell R410. I can clearly see that the tool is > "disabling" the MFW bit, and that the dell bios interface to the IPMI > controller/DRAC is impaired, however ... > > The system still thinks that the MFW bit is "ON" and proceeds normally > in this case. The code still fails to detect that MFW is disabled. > > In addition, the IPMI driver attaches to "something" and I can poke at > it via ipmitool. So, from the driver perspective, the attempt to detect > MFW enabled doesn't work as the chipset still thinks that its "on". > > I've commented out the code block that we're trying to work around in > the yahoo code base for now. I have an R210 and running the uxdiag command definitely disabled management firmware on my system: ===[root] ~ # dmesg | grep bce bce0: mem 0xda00-0xdbff irq 16 at device 0.0 on pci2 miibus0: on bce0 bce0: Ethernet address: b8:ac:6f:87:95:f1 bce0: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.2); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI) bce1: mem 0xdc00-0xddff irq 17 at device 0.1 on pci2 miibus1: on bce1 bce1: Ethernet address: b8:ac:6f:87:95:f2 bce1: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.2); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI) bce0: Gigabit link up! bce0: Gigabit link up! bce1: Gigabit link up! bce1: Gigabit link up! ===[root] ~ # Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: stable/7 bce(4) out of buffers
> > > dev.bce.0.com_no_buffers: 130228 > > > > Hmm, in HEAD there looks to be a hw.bce.rx_pages that can be set to > > 1, 2, 4, or 8. It defaults to 2. In the 7 driver it looks like this > > is a #define in sys/dev/bce/if_bcereg.h (RX_PAGES on line 6710). Try > > changing RX_PAGES from 2 to 4 (or 8) and recompiling your kernel (or > > bce driver) and see if that helps. > > > > Indeed, I was eyeballing that just now. I'll post back with results. > Adding buffers might help but more often than not you really should adjust the interrupt coalescing sysctls (hw.bce.rx_quick_cons_trip, hw.bce.rx_ticks) so the hardware releases frames to the OS more frequently and fetches new buffers. Setting both values closer to 1 improves latency and small packet performance but does increase CPU load due to the increased number of interrupts. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce: jumbo not working since r218423
>> I also don't understand why sysctl hw.bce.loose_rx_mtu doesn't respect >> with tunnable hw.bce.strict_rx_mtu. Is there any reason to give them >> different names? >It may be an oversight. Personally I don't see any reason except >debugging purpose to limit RX frame size to interface MTU. It makes >sense when controller send frames but it should be able to receive >any sized RX frames(if controller allows it). Dropping RX frames >that are bigger than interface MTU would break path MTU discovery >of remote host that uses bigger MTU. The intent was to pass compliance tests such as those performed at UNH by limiting the MTU to the size allowed by the IEEE 802.3 specification. (No one likes explaining such failures to customers.) As you indicate, real world applications benefit from loose compliance on RX so a single tunable is all that's intended to enable/disable the behavior. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: Dell PowerEdge R820 Broadcom BCM57800 support
> > >I have an R820 I'm testing. The system seems to boot up fine, but > > > no network adapters show up. From pciconf -l : > > > > > > none4@pci0:1:0:0: class=0x02 card=0x1f5c1028 > chip=0x168a14e4 rev=0x10 hdr=0x00 > > > none5@pci0:1:0:1: class=0x02 card=0x1f5c1028 > chip=0x168a14e4 rev=0x10 hdr=0x00 > > > none6@pci0:1:0:2: class=0x02 card=0x1f671028 > chip=0x168a14e4 rev=0x10 hdr=0x00 > > > none7@pci0:1:0:3: class=0x02 card=0x1f671028 > chip=0x168a14e4 rev=0x10 hdr=0x00 > > > > > > which appears to be these: > > > > > > Broadcom BCM57800 NetXtreme II 10 GigE 1f5c > > > Broadcom BCM57800 NetXtreme II 1 GigE1f67 > > The chipid is 0x168a14e4 which indicates vendor is Broadcom and > device is NetXtreme II BCM57800 10G gigabit ethernet. I guess > bxe(4) would be right driver to pick up the controller but it seems > there is no support for BCM57800 in bxe(4) at this moment. > Probably David can add more comment on this(CCed). No committed plans to update bxe(4) to support BCM57712 or BCM578XX at the moment. If you have a near term need for 10Gb controllers on this platform you'll need to select an alternate rNDC adapter when you order the system. I posted a request on freebsd-jobs last month looking for someone who would be interested in the work if you can recommend someone. Not all of the documentation required for the effort is publically available so an NDA will likely be required to do the work. http://lists.freebsd.org/pipermail/freebsd-jobs/2012-August/000809.html Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: kern/134658: [bce] bce driver fails on PowerEdge m610 blade.
> >> Ooo!! > >> > >> Could you link me though to the patch Oliver, I don't seem to find > >> any post on this thread between today and mid last year? > >> > > > >Let's give more time to David. He can test my guess work and can > >fix bugs in the patch. When all goes well he may commit the patch. > > (please keep me CC'ed as I'm not subscribed to -net) > > Hi, > > Is there any news about the patch ? I have a bunch of M610 I > would like to install FreeBSD on :) Just committed a patch to head written by pyunyh and verified by me. Let me know if it works. I can get access to a M610 locally if you run into problems. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce+lagg
> If I configure the cards individually, I can reach other devices on the
> network. When I enable link aggregation, I can ping the hosts IP
> address,
> but I cannot ping its gateway.
I haven't played with lagg on bce(4) so I definitely haven't tested this.
Can you dump the hardware statistics before/after the test
("sysctl -a | grep bce") and "diff" them to see which statistics have
changed? Seems like it might be an RX filtering problem where the NIC
drops frames unexpectedly. Does a trace on the GW show that if receives
the ICMP echo requests and sends out the responses?
Dave
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) - com_no_buffers (Again)
> I'm under the impression the header splitting in bce(4) is for > LRO(opposite of TSO), not for VM magic to enable page flipping > tricks. Header splitting was implemented in the Linux version of bce(4) to prevent jumbo memory allocations. Allocating 9KB frames was causing problems on systems used for virtualization. (Harder to find a contiguous 9KB frame when a hypervisor is in use.) Using 4KB or smaller buffer sizes was considered more compatible with virtualization. LRO (Large Receive Offload, aka Transparent Packet Aggregation or TPA on the 10Gb controllers) is not supported on the 1Gb bce(4) devices. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) un hiding adapter info
> Would it be possible to unhide the output of bce_print_adapter_info() > from under boot verbose? > > This information is useful for comparing firmware and card versions > between machines. > > Alternatively what about adding a sysctl under dev.bce.X for this info? I have no problem doing that, just not sure that everyone would appreciate the spew. I like it for exactly the same reason, troubleshooting remote systems. What I'd really like to do is revamp the debug code so that it can be enabled/disabled on the fly rather than requiring that the driver be compiled. Adding some performance stuff would be awesome as well. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) - com_no_buffers (Again)
> >> Under testing I have yet to see a memory fragmentation issue with > this > >> driver. I follow up if/when I find a problem with this again. > >> > >> > So here we are again. The system is locking up again because of 9k > mbuf > allocation failures. Failure to allocate a new buffer should cause the driver to drop the received frame and reuse the buffer, not lock up the system. Are you seeing the lockup come from bce(4) or does it come from somewhere else due to the dropped data? > >> Is there a way to fix the RX buffer shortage issues (when header > >> splitting is turned on) so that they are guarded by flow control. > Maybe > >> change the low watermark for flow control when its enabled? > >> > >> > > I'm not sure how much it would help but try changing RX low > > watermark. Default value is 32 which seems to be reasonable value. > > But it's only for 5709/5716 controllers and Linux seems to use > > different default value. > > > These are: NetXtreme II BCM5709 Gigabit Ethernet > > So my next task is to turn the watermark related defines into sysctls > and turn on header splitting so that I can try to tune them without > having to reboot. > Do you have flow control enabled? There are arguments both for and against flow control. For bce(4), I haven't tested flow control for quite a while and it's behavior may have changed since it is controlled by firmware. Keep an eye on the hardware statistics to see that's it's actively generating pause frames. > > My next question is, is it possible to increase the size of the RX ring > without switching to RSS? > I have a change I've been working on to allow RX/TX ring size to be adjusted through a sysctl. Let me pretty it up a bit and send it to you for test. You should be able to adjust the ring size without enabling RSS. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) - com_no_buffers (Again)
> > Failure to allocate a new buffer should cause the driver to > > drop the received frame and reuse the buffer, not lock up the > > system. Are you seeing the lockup come from bce(4) or does > > it come from somewhere else due to the dropped data? > > > > > > The lockup is not from the NIC as such, the systems have the appearance > of locking up as home directories are on NFS and the user information > is > stored in a remote LDAP server. When the system starts to drop frames > due to lack of 9k memory regions it tends to last for a few minutes > (when it is really bad) and stop all traffic into the system. This > appears to the average user as a complete system pause. > Ack. > > I am under the impression that the best solution is to tune the RX ring > so that flow control can be disabled but I not sure I could do this. > Are small frames common in the actual failing case or are they restricted to your test case? At least one of the Broadcom Linux drivers addresses this issue by copying frames < 128 bytes to a standard size buffer, saving the jumbo buffer allocation and DMA mapping operation. If small frames are at the root of your problem then I think this would be a better solution than allocating more 9KB blocks by increasing the ring size. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) un hiding adapter info
> > What I'd really like to do is revamp the debug code so that it > > can be enabled/disabled on the fly rather than requiring that > > the driver be compiled. Adding some performance stuff would > > Couldn't it be implemented with sysctl? Users may set a variable > something like dev.bce.0.diag_bitmap to activate debugging code. > But it seems that requires a lot of code changes. I already have an extensive amount of conditionally compiled debug code in the driver which uses a bitmap to adjust the debug spew. For personal testing I've also added a sysctl to allow that value to be modified at runtime. My concerns are from a security standpoint (some of the debug code allows direct register access), a code size perspective (adds a lot of code most people don't use), and the performance penalty (lots of bitmap checking even when all debug spew is turned off). Probably worth the time to pull the most useful stuff into the production driver and leave the more obscure code in the debug only build. If there are no complaints I'll add that to my task list. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: driver for broadcom 57711E
> > > > Hi! I really need broadcom 57711E driver for my FreeBSD 8.0 > i386. > > > > Does anybody have this driver already? > > > > > > > > P.S. Also wanted David Christensen who had previously tried to > write a driver. > > > > > > LOL. Welcome to the club :-) > > > > > > ( Though I'm afraid David, or Broadcomm, respectively, are > incommunicado > > > on this particular issue. ) > > > > > Actually David is very responsive but I think he is somewhat busy > > for other work. For example, I had been working with David to > > support next generation Broadcom controller that would be widely > > available in next year. > > Personally I haven't have chance to try bnx(4) which was written > > by David to support BCM57710/57711/57711E controllers so I don't > > know how well it works. As David said long time ago, one of issue > > was testing wide variety of PHYs used in controller such that it > > made hard to release the driver. > > David, would you clarify current status of bnx(4)? > > Actually the driver name was bxe(4). Sorry for the delays on keeping the community updated on the status of the bxe(4) driver. If someone is willing to review the driver and help us address some of the style(9) violations in the source code, I'll commit the code within two weeks. Please email me off-list if you're able to help. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
New bxe(4) Driver for Broadcom BCM57710 & BCM57711 Devices
A number of people have asked about a FreeBSD driver for the Broadcom 10GbE devices (BCM57710 and BCM57711). I just submitted the initial release of the bxe(4) driver to -HEAD today so please test and provide your feedback. Thanks, Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Availability of Programmer's Guide for Broadcom 57XX LAN Controllers
For those who are interested in working on the bge(4) driver, Broadcom has now posted its Programmer's Guide at http://www.broadcom.com/collateral/pg/57XX-PG105-R.pdf. This document covers most of the older chips including 5700, 5701, 5703, 5704, 5714, and 5721, though since much of the architecture is shared with newer chips such as 5754 and 5761, it is still a useful reference. We're working on releasing the PG's for other members of the bge(4) family and also the PG for the bce(4) family soon. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: kern/134658: [bce] bce driver fails on PowerEdge m610 blade.
The following reply was made to PR kern/134658; it has been noted by GNATS. From: "David Christensen" To: "bug-follo...@freebsd.org" , "harald_jen...@dell.com" Cc: Subject: Re: kern/134658: [bce] bce driver fails on PowerEdge m610 blade. Date: Wed, 20 May 2009 17:38:48 -0700 This is the expected behavior as the mii(4) SerDes support=20 for the 5709 was removed from immediately prior to the 7.2=20 release. The PHY's ID is not present in miidevs.c. Dave= ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: broadcom 57710 support
> Is there any near plan to develop drivers for network cards > based on broadcom NetXtereme II 57710 10 GbE controller? I have written a driver to support the 57710/57711/57711E parts, bxe(9), but I haven't released it as I haven't been able to get much testing mileage on it. The biggest challenge is the wide array of PHYs used in the 10GbE space which makes testing a complex issue (I think we have 7 or 8 PHYs today which various customers use). If there are some developers who are interested in getting a version of the driver and helping work out some of the bugs please contact me directly. Ideally I'd like to have this driver ready for 8.1. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: kern/134658: [bce] bce driver fails on PowerEdge m610 blade.
> Anyone point me in the right direction on how to add the phy > to support these machine? > > Seems like its just a matter of adding the PHY details to > miidevs but how do I find out the specifics of that? Sorry, this is the 5709S and I haven't had an opportunity to implement this PHY yet. Unfortunately it's more than just a change to miidevs since the SerDes is actually an IEEE clause 45 compliant device (instead of the more common Clause 22 devices found in 1GbE controllers). The registers are diffrerent so the effort is more substantial. No estimate yet on when I can get to it. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: em(4): sending ARP regardless of NOARP flag
> >> I don't see how arping or not can be a driver problem, the driver > >> just sends packets queued by the stack, there exists NO > mechanism to > >> communicate that kind of thing down into the driver, -arp is > >> something that must be negotiated in the stack somewhere, > as for it > >> working with broadcom... > >> > >> > >> > > except for the system management stuff. On the bce(9) driver does it display the "MFW" flag during driver load? That would indicate whether NC-SI style management firmware is running which would be unexpected on a NIC card. If the Intel LOM is connected to a baseboard management controller or service processor then the BMC or SP are likely generating the ARP. What's the source MAC address of the ARP? Does it match the LOM's MAC address or the MAC address of any BMC or SP? The latter would generally be printed on a tag on the system or perhaps in a BMC setup screen visible during POST. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> Has anyone seen these errors before: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=135836&cat= > > The system is a Dell R610 and it happens on both cold and warm boots. > > I am about to check a second chassis, and test with 8, and > will follow up after my tests. Yes, I've seen these errors on a Dell r710. The problem is related to an interaction between the FreeBSD driver and the bootcode firmware on the BCM5709 controller which can cause the bootcode firmware to enable certain hardware blocks in the chip while the FreeBSD driver is attempting to reset the device. This allows RX traffic to enter the controller before the device is fully initialized. The fix is to upgrade the BCM5709 bootcode to version 5.02 or later. I'm checking if Dell has posted updated firmware which can address the problem. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> Thanks for the rapid response. > > Dell have firmware 5.0.9 on their website here: http://tiny.cc/ex834 > > Will that work? Yes, that release does include a good version of BCM5709 bootcode (v5.06). I couldn't really tell until I downloaded the file and looked at the temporary files inside. Thanks for the link. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> > After fighting with the update process for a bit and then learning > > about the live cycle manager I have managed to test the > R610 with the > > 5.0.9 firmware. > > > > On the face of it, it seems that this resolves the issue. > > > After a reboot this morning the error has resurfaced and the > box is again useless. Can you try a different test? Power-on the system with the network cable attached to an idle switch (i.e. keep all network traffic from being forwarded to the NIC during driver initialization). Does the system power up successfully? Repeatedly? The problem I saw was caused by network traffic being handled by the NIC during driver initialization. If you still see the same behavior then this might be a different issue. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> > Can you try a different test? Power-on the system with the network > > cable attached to an idle switch (i.e. keep all network > traffic from > > being forwarded to the NIC during driver initialization). Does the > > system power up successfully? Repeatedly? The problem I saw was > > caused by network traffic being handled by the NIC during driver > > initialization. If you still see the same behavior then > this might be > > a different issue. > > > > Looks like we are heading in the right direction here. > The next test is to diable the LOM's management firmware but boot to an active network. Get the User Diag utility at the bottom of http://www.broadcom.com/support/ethernet_nic/netxtremeii.php. Run the uxdiag utility with the command line: "C:\>uxdiag -c 0 -mfw 0 -c 1 -mfw 0 -c 2 -mfw 0 -c 3 mfw 0" The "-c X" specifies with LOM to use while the "-mfw 0" disables the firmware. Use the appropriate number of "-c X" values for the number of ports on your system (the r710 has 4 ports). To re-enable the firmware do the following: "C:\>uxdiag -c 0 -mfw 1 -c 1 -mfw 1 -c 2 -mfw 1 -c 3 mfw 1" Finally, the routine bce_print_adapter_info() in HEAD prints out both the bootcode and management firmware versions. If you can get those same changes into your release I'd like to see the versions reported on your system. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> > The next test is to diable the LOM's management firmware > but boot to > > an active network. Get the User Diag utility at the bottom of > > http://www.broadcom.com/support/ethernet_nic/netxtremeii.php. > > Run the uxdiag utility with the command line: > > "C:\>uxdiag -c 0 -mfw 0 -c 1 -mfw 0 -c 2 -mfw 0 -c 3 mfw 0" > > The "-c X" specifies with LOM to use while the "-mfw 0" > disables the > > firmware. Use the appropriate number of "-c X" values for > the number > > of ports on your system (the r710 has 4 ports). > > To re-enable the firmware do the following: > > "C:\>uxdiag -c 0 -mfw 1 -c 1 -mfw 1 -c 2 -mfw 1 -c 3 mfw 1" > > > > Finally, the routine bce_print_adapter_info() in HEAD > prints out both > > the bootcode and management firmware versions. If you can > get those > > same changes into your release I'd like to see the versions > reported > > on your system. > > > > Here is the info from a boot of 8.0 RC2. > > ASIC: 0x57092003 > B/C: 5.0.6 > Rev: C0 > Bus: PCIe x4, 2.5Gb/s > Flags: MSI|MFW > MFW: NCS 2.0.3 > > And looking at this it seems dells update CD was not up to > date enough and I only got 5.0.6 firmware not 5.0.9. The package version and the bootcode version are similar but they are not the same. The bootcode you have (v5.0.6) is sufficient to fix the problem. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
Hi, > we have two new Dell R610 machines with four bce NICs (only > bce0 is connected at this time). I tried cold (power cycle in > iDRAC) and warm reboot (shutdown -r now), NIC is working on > every reboot, but I am still seeing messages: > > bce0: /usr/src/sys/dev/bce/if_bce.c(1525): PHY write timeout! > bce0: /usr/src/sys/dev/bce/if_bce.c(1525): PHY write timeout! > bce0: link state changed to UP > > This is on FreeBSD 7.2-RELEASE-p4 amd64 > > I also see "PHY write timeout" on shutdown time for each NIC. > > Other than the log messages, I see no problems with bce NIC, > but I have not tested it with any high load. > > I did not upgraded FW or anything else. > > What can I do to help with solving this issue? You'll need to upgrade to 7-STABLE. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> Is there any progress on this issue? We can provide access > to hardware with this fault (Specifically the R610). I haven't been able to reproduce it on the r710 I have in house. Checking with other groups now to see if they have one a I can use, though I'm not sure why the system would make a difference for this particular issue. Just wanted to confirm that you're using the driver built into the kernel (as opposed to a module) and that a warm boot means running the "reboot" or "shutdown -r" commands while a cold boot means pressing the front panel power button or using the DRAC to power down the system. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> > I haven't been able to reproduce it on the r710 I have in house. > > Checking with other groups now to see if they have one a I can use, > > though I'm not sure why the system would make a difference for this > > particular issue. Just wanted to confirm that you're using > the driver > > built into the kernel (as opposed to a module) and that a warm boot > > means running the "reboot" or "shutdown -r" > > commands while a cold boot means pressing the front panel > power button > > or using the DRAC to power down the system. > > > > Dave > > Hi Dave, > > Thanks for the update. > > All these are true. > > warm - shutdown -r now > cold - from the power button (iDRACs are not configured yet). > > bce is compiled into the kernel (tested with GENERIC kernel > from 8-RC2 as well as 7.1 with the 7.2 driver plus the split > header patch). > > For the record we also have not been able to reproduce the > issue on the R710 only the R610. Still no luck on my r710, even after reverting to the production firmware and blasting broadcast traffic at the interface during reset. Let me check again on an r610. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> > I haven't been able to reproduce it on the r710 I have in house. > > Checking with other groups now to see if they have one a I can use, > > though I'm not sure why the system would make a difference for this > > particular issue. Just wanted to confirm that you're using > the driver > > built into the kernel (as opposed to a module) and that a warm boot > > means running the "reboot" or "shutdown -r" > > commands while a cold boot means pressing the front panel > power button > > or using the DRAC to power down the system. > > Thanks for the update. > > All these are true. > > warm - shutdown -r now > cold - from the power button (iDRACs are not configured yet). > > bce is compiled into the kernel (tested with GENERIC kernel > from 8-RC2 as well as 7.1 with the 7.2 driver plus the split > header patch). > > For the record we also have not been able to reproduce the > issue on the R710 only the R610. I got hold of an R610 system and I now understand why the issue was difficult to replicate on R710. The R610 ships without Enterprise iDRAC while the R710 ship with the add-in Enterprise iDRAC module. When the module is present the system is managed through the additional RJ45 port but when the module is absent iDRAC traffic will flow through the on-board 5709 adpaters. The error will only occur when management firmware is loaded on the 5709 AND when NC-SI management functionality is enabled. You should be able to confirm this by adding or removing the Enterprise iDRAC module on your systems. Now that I have a failure again I have some ideas to test which might help. Stay tuned. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> > For the record we also have not been able to reproduce the issue on > > the R710 only the R610. > > I got hold of an R610 system and I now understand why the > issue was difficult to replicate on R710. The R610 ships > without Enterprise iDRAC while the R710 ship with the add-in > Enterprise iDRAC module. When the module is present the > system is managed through the additional RJ45 port but when > the module is absent iDRAC traffic will flow through the > on-board 5709 adpaters. > The error will only occur when management firmware is loaded > on the 5709 AND when NC-SI management functionality is enabled. > > You should be able to confirm this by adding or removing the > Enterprise iDRAC module on your systems. Now that I have a > failure again I have some ideas to test which might help. > Stay tuned. Does the attached patch make a difference for you? FYI, I'll be out next week on vacation. Dave if_bce.diff Description: if_bce.diff ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> > Does the attached patch make a difference for you? > > This patch seems to do the trick, on at least one of the > R610's that we have. > > Just did cold boot, 5 warm boots, cold boot, 5 warm boots and > have not had any issues. > Thanks for testing. I want to pass the changes by a couple other people before I go ahead with a commit. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bge driver and MSI
> Can the bge driver use more than one MSI message? If > possible, what the advantage of this on a SMP system (better > CPU distribution on interrupts?)? > > I have an Broadcom BCM5703X, with 8 MSI messages: > -- > b...@pci0:1:2:0:class=0x02 card=0x00cb0e11 > chip=0x16a714e4 rev=0x02 > hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'BCM5703X NetXtreme Gigabit Ethernet' > class = network > subclass = ethernet > cap 07[40] = PCI-X 64-bit supports 133MHz, 2048 burst > read, 1 split transaction > cap 01[48] = powerspec 2 supports D0 D3 current D0 > cap 03[50] = VPD > cap 05[58] = MSI supports 8 messages, 64 bit > -- MSI is broken on this device (actually all PCI/PCI-X devices in the 57XX family). The interrupt will be asserted as expected but masking the interrupt does not work. For those newer 1Gb PCIe devices that do work correctly with MSI, only a single message is actually supported. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bge driver and MSI
> > MSI is broken on this device (actually all PCI/PCI-X devices in the > > 57XX family). The interrupt will be asserted as expected > but masking > > the interrupt does not work. For those newer 1Gb PCIe > devices that do > > work correctly with MSI, only a single message is actually > supported. > > Might be a good thing of someone would work up a man page > paragraph to this effect. > As a more general question, what would be the right medium to document such device errata as it relates to a driver? Is the man page the best location or would it be better to add a table near the top of the driver file which summarizes the bugs and the workarounds? Is there a general consensus? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: "PHY read timeout" with bce2 & 3 on FreeBSD 8.0
> > I'm working with a system (IBM System x3550 M2) that has > four onboard > > bce(4) NICs. Using FreeBSD 8.0, the first two seem to > function fine, > > but the second two do not, yielding messages like this when > a cable is > > inserted: > > > > Jan 12 10:37:19 dmz55 kernel: <2<>N2M>NINM IM INISMI AI S AIIS SAA > > 22cc2,,c , E2EIcIES,SIAS AA E I00 Jan 12 10:37:19 dmz55 kernel: > > Jan 12 10:37:19 dmz55 kernel: S0 > > Jan 12 10:37:19 dmz55 kernel: > > Jan 12 10:37:19 dmz55 kernel: <<22>>A > > Jan 12 10:37:19 dmz55 kernel: > > Jan 12 10:37:19 dmz55 kernel: 0 > > Jan 12 10:37:28 dmz55 kernel: bce2: > /usr/src/sys/dev/bce/if_bce.c(1533): > > Error: PHY read timeout! > > phy = 1, reg = 0x0001 > > > > > > What's are the best next steps to take in figuring this > out? Thanks in > > advance for any assistance. Kernel boot log appending below. > > > > Are you using ASF/IMPI/UMP on bce2 or bce3? > If so would you try attached patch? Publicly available datasheet > lacks details on management firmware handling so I'm not sure what > is really happening here. > davidch may shed light on this(CCed). The first lines include a lot of N's, M's, and I's, which make me think an NMI may be occurring. Does the garbled text and the PHY timeout error disappear if you disable bce2/bce3? Can you try adding "options PRINTF_BUFR_SIZE=256" to your kernel config file to try and de-garble the output? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: kern/134658: [bce] bce driver fails on PowerEdge m610 blade.
> While trying to debug the same issue I stumbled across this > thread ... We've got HS22 blades (IBM BladeCenter) which > habe the BCM5709S and suffer from exactly the same problem. > > Dave, are there any news regarding the PHY implementation? > > If there's decent documentation I might even give it a try > myself, provided that it's not too complex. (I've done > hardware programming before, but I've never touched a NIC/PHY > driver, except for very trivial fixes.) > Pyunh has taken an active interest in this effort and sent out a patch yesterday, though he doesn't have hardware to test. I'm planning on giving his patch a try today to see how close he got on his first attempt. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: bce(4) BCM5907 CTX write errors on 7.2 driver
> > Thanks for testing. I want to pass the changes by a couple other > > people before I go ahead with a commit. > > > > Dave > > > > Hi Dave, > > I was just wondering when these changes will hit the tree, I have been > running your patch in production for some time now and all seems well. Just committed the changes today. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: kern/134658: [bce] bce driver fails on PowerEdge m610 blade.
> > > If there's decent documentation I might even give it a > try myself, > > > provided that it's not too complex. (I've done hardware > programming > > > before, but I've never touched a NIC/PHY driver, except for very > > > trivial fixes.) > > > > > > > Pyunh has taken an active interest in this effort and sent > out a patch > > yesterday, though he doesn't have hardware to test. I'm > planning on > > giving his patch a try today to see how close he got on his first > > attempt. > > Where can I find this patch? We can also give it a try on our m610. Let me give it a try first, if I can get link I'll pass it along. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
RE: Some Broadcom GbE NIC(bge driver) suffers packet loss at receiving
> Some revision of Broadcom GbE NIC with bge driver > seems to loss 1 packet per 1000 packet at receiving on some occasion. > Sending doesn't have this problem(receiving only). > > This loss doesn't always happen, > but when happens, it seems that loss is exactly 1 per 1000, > as the last 3 digits of lost packets' sequence number are all same. > (i.e. example of lost packet seq-No. are: > 3284, 4284, 5284, 8284, 9284, ...) > The NIC maintains internal statistics which might give a clue to the cause of the packet loss. The latest bge driver in -CURRENT includes a number of "sysctl" nodes to bring out those statistics. Can you get that info? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: FreeBSD discarding received packets > MTU
> > It could certainly be argued by some that Cisco is not standards > > compliant in this case for sending an oversized Ethernet frame > > and expecting everyone to accept it. Hardware has limitations > > and assuming that all Ethernet controllers can support frames > > greater than 1522 bytes is not reasonable. Fortunately there is > > a suitable workaround which is setting a larger MTU for the > > interface. What size do you use? How did you arrive at that > > value? > > I use 1550 to make it work in the test harness. > > The trouble is that if I set the mtu to 1550, and the machine > talks to another > such machine with it's mtu also set to 1550 then they > negotiate a maximum sized > packet based on 1550, and the problem hits me again. This is > a web proxy > and that problem occurs when there are two layers of proxy > and one proxy talks to > another. I really just need it to to silently accept a packet some > 32 bytes or so larger than the stated MTU. > > I see no reason for the driver to not do what the em driver > does and allow > itself to receive any packet up to the MCLBYTES size. > > We only hit this problem recently because the data interfaces on our > devices are usually em NICs and we only just recently started > allowing the > users to use the built in (on DELL 2950) bce interfaces for > this purpose. > I'm not completely opposed to making such a change, but I don't want to make a default change in the driver's behavior that other people may be depending upon (whether they are aware of it or not). A tunable driver value could be the answer but I'm not entirely sure how it would fare in the hardware at the high end of MTU values such as 9000. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: FreeBSD discarding received packets > MTU
> Julian Elischer wrote:
> > Ok which clever person did this again?
> >
> > It just broke our product.
> > If it hasn't been removed from 7.0 and 6.x yet it's just
> about to be...
>
>
> on the topic above, I've found that the bce driver seems to
> be throwingaway packets
> larger than the mtu regardless of the rest of the system..
This has been part of the driver since it's initial release,
it's not an addition or change.
>
> the following code seems to be responsible..
>
> in the spirit of "be generous on receive" as mentionned before,
> should it be relaxed?
>
When this RX MTU size issue was being discussed previously I
was wondering why nobody actually discussed what the drivers
are doing. Both the NetXtreme (bge) and NetXtreme II (bce)
controllers have a setting in the RX MAC to specify the MTU
size of the medium. Anything larger than the MTU is discarded
and a statistic is recorded. Support for jumbo frames is
available (though not on all bge supported controllers) but the
general understanding from the hardware design perspective is
that the MTU is the maximum message size on the medium and that
all controllers on the medium should be configured for the same
message size. I've never seen an Ethernet driver that supports
different MTUs for RX and TX (is there one implemented under
FreeBSD?) and I would really hate to troubleshoot any higher
level application problems that might occur if the RX and TX MTU
were different for every system on a network.
>
>
> /* Calculate and program the Ethernet MTU size. */
> ether_mtu = ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN +
> ifp->if_mtu +
> ETHER_CRC_LEN;
>
> DBPRINT(sc, BCE_INFO, "%s(): setting mtu =
> %d\n",__FUNCTION__, ether_mtu);
>
>
> /*
> * Program the mtu, enabling jumbo frame
> * support if necessary. Also set the mbuf
> * allocation count for RX frames.
> */
> if (ether_mtu > ETHER_MAX_LEN + ETHER_VLAN_ENCAP_LEN) {
> REG_WR(sc, BCE_EMAC_RX_MTU_SIZE, ether_mtu |
> BCE_EMAC_RX_MTU_SIZE_JUMBO_ENA);
> sc->mbuf_alloc_size = MJUM9BYTES;
> } else {
> REG_WR(sc, BCE_EMAC_RX_MTU_SIZE, ether_mtu);
> sc->mbuf_alloc_size = MCLBYTES;
> }
>
>
> could probably be relaxed in the non jumbo case to accept
> packets up to
> MCLBYTES in size instead of ether_mtu.
Maybe MCLBYTES is good enough for you but what if someone has
a different requirement? How big is big enough for everyone?
I suppose supporting different MTUs for RX and TX is possible,
though the memory requirements are 5x larger for jumbo frames
over standard frames and many users might consider this a waste
of resources if they don't support jumbo frames on their
network (that would be almost 4MB). Support would obviously
be easier if "ifconfig" support both an RX and TX MTU size.
>
> manually I have set the mtu to larger values which increases
> ether_mtu and that seems to work..
> now, Why do we need this?
>
> WCCP is a protocol from Cisco.
> it encapsulates packets in GRE, including an extra
> intermediate header.
> when packets are encapsulated in GRE by cisco 6k switches
> (quite a few around)
> they do NOT frag the result, but instead, send an oversized packet.
>
> if you have a bce interface, it discards these packets which
> is a drag.
It could certainly be argued by some that Cisco is not standards
compliant in this case for sending an oversized Ethernet frame
and expecting everyone to accept it. Hardware has limitations
and assuming that all Ethernet controllers can support frames
greater than 1522 bytes is not reasonable. Fortunately there is
a suitable workaround which is setting a larger MTU for the
interface. What size do you use? How did you arrive at that
value?
Dave
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: FreeBSD discarding received packets > MTU
> > I'm not completely opposed to making such a change, but I don't want > > to make a default change in the driver's behavior that other people > > may be depending upon (whether they are aware of it or not). A > > tunable driver value could be the answer but I'm not entirely sure > > how it would fare in the hardware at the high end of MTU > values such > > as 9000. > > Dave: > > Internet ettiquette demands being gracious in what you accept. > The default policy of FreeBSD is to accept such packets. > This is a really weird bug to track down. > Other drivers support it. > > This isn't worth making a stand over, unless you're trying > to hold users of YOUR driver hostage. > I'm just being cautious about making changes before I understand all of the implications. The driver's current behavior is supported by IEEE 802.3 specification (802.3-2005, 4.2.4.2.1) and is implemented in the same way for other operating systems that are very widely deployed (including Windows and Linux) without any reported problems. The existing bge driver which was developed for FreeBSD 10 years ago also operates this way, so all of my references for porting this driver happen to agree on the same implementation. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: FreeBSD discarding received packets > MTU
> > > Internet ettiquette demands being gracious in what you accept. > > > The default policy of FreeBSD is to accept such packets. > > > This is a really weird bug to track down. > > > Other drivers support it. > > > > > > This isn't worth making a stand over, unless you're trying > > > to hold users of YOUR driver hostage. > > > > > > > I'm just being cautious about making changes before I understand > > all of the implications. The driver's current behavior is > > supported by IEEE 802.3 specification (802.3-2005, 4.2.4.2.1) > > and is implemented in the same way for other operating systems > > that are very widely deployed (including Windows and Linux) > > without any reported problems. The existing bge driver which > > was developed for FreeBSD 10 years ago also operates this way, > > so all of my references for porting this driver happen to agree > > on the same implementation. > > Which is all well and good, but the age of a bug does not a feature > make. > > Please think of the four points I raised. > > I think it makes sense to possibly add a "enforce rx mtu" knob > somewhere, but it should likely be turned off. That's the plan. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: BCM5704 chip "initialization failed"
> The motherboard of my HP proliant DL360 G4p was changed and now I > cannot see my network cards (BCM5704 netXtreme dual gigabit). I'm > running FreeBSD 6.2 p5... > > Here is what I have in dmsg > > bge0: mem > 0xfdef-0xfdef > irq 25 at device 2.0 on pci2 > bge0: Reserved 0x1 bytes for rid 0x10 type 3 at 0xfdef > bge0: firmware handshake timed out > bge0: RX CPU self-diagnostics failed! > bge0: chip initialization failed The NVRAM attached to the 5704 is likely bad, blank, or not programmed correctly. Check with HP support if they have any tools for testing/programming the 5704 (usually called b57udiag or q57udiag) otherwise you probably need to get the board replaced again. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Recommendation for a Big Endian System for NIC Driver Development
I'm looking for a recommendation for a big endian CPU system that works well with FreeBSD. It would be best if it didn't use a 4KB page size but that's a secondary issue. The system should support PCI Express with at least two x8 slots and PCI-X would be a plus. A half-height or full-height rack style chassis would be best, though a tower configuration would also be suitable. I want to do some wider testing on non-Intel based systems with current and new Broadcom drivers. Feel free to email me directly if you have a suggestion. David Christensen ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Recommendation for a Big Endian System for NIC Driver Development
> > I want to do some wider testing on non-Intel based systems with > > current and new Broadcom drivers. Feel free to email me directly > > if you have a suggestion. > > look at getting a PG powerPC off ebay.. > > (I can help you set it up if you wish.. > I just did a dozen of them for work). > > I like that option because then you can help debug why > this ppc system can't run reliably with this > broadcom phy on the gem driver :-) Do you have a specific make and model number you can suggest? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: bce(4) polling support?
> Just wondering if someone's been working on POLLING support for bce(4) > Broadcom NetXtreme II (BCM5706/BCM5708) PCI/PCIe Gigabit Ethernet > adapter driver? AFAIK late in 2006 brueffer@ decided[*] to remove the > erroneous polling(4) man entry claiming bce support... and that's it. > Any advancements since then? > > [*] http://monkey.org/freebsd/archive/freebsd-net/200612/msg4.html The only advancement is that I am taking the current code out of the driver since it's never really been tested or validated. No current plans to add the support back in any time soon. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Not All Symbols Present in a Loadable Kernel Module
I'm trying to build the "bce" driver as a kernel module under RELENG_7 but I'm finding that not all of the functions in the driver are exported as symbols. This makes it difficult to "call" a function from ddb because I get the error "Symbol not found". I'm building and loading the driver from /usr/src/sys/modules/bce. What am I doing wrong? How can I get all functions in the driver exported as symbols usable by the debugger? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Not All Symbols Present in a Loadable Kernel Module
> > I'm trying to build the "bce" driver as a kernel module under > RELENG_7 but I'm > > finding that not all of the functions in the driver are exported as > symbols. This > > makes it difficult to "call" a function from ddb because I get the > error "Symbol > > not found". I'm building and loading the driver from > /usr/src/sys/modules/bce. > > What am I doing wrong? How can I get all functions in the driver > exported as > > symbols usable by the debugger? > > Are you building a debug kernel or regular kernel? Have you turned on > debug symbols? > > makeoptions DEBUG=-g# Build kernel with gdb(1) > debug symbols > > Just a quick thought...I'm assuming these symbols are listed under > your final kernel image (nm it etc.). Yes, I'm building a debug kernel. I have the line listed above as well as the following: options KDB options DDB options GDB options INVARIANTS options INVARIANT_SUPPORT options WITNESS options WITNESS_SKIPSPIN Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Not All Symbols Present in a Loadable Kernel Module
> > Yes, I'm building a debug kernel. I have the line listed above as > well > > as the following: > > > > options KDB > > options DDB > > options GDB > > options INVARIANTS > > options INVARIANT_SUPPORT > > options WITNESS > > options WITNESS_SKIPSPIN > > Dave: > > What symbols can you not access exactly and how are you > installing/setting up debugging? > > I just built a RELENG_7 with DDB and I'm able to access bce symbols > without a problem. I can examine them and call them. I'm not using > options GDB however, only KDB/DDB. > > I would: > > - Make sure you have the right if_bce.ko/if_bce.ko.symbols files > generated/installed which contains the debug sections of your ko (from > the objcopy calls during the build - the binary is stripped with a > section pointer to the if_bce.ko.symbols file for debugging > information I believe) > - If you are using GDB, make sure its pointed to the right source base > so it can retrieve symbol information correctly > - If you are using GDB, stub it out and just use DDB to verify that > your build is sane (it works for me!) > - If all else fails, you can always build bce statically (just to move > forward etc.) - Enable the kernel debugger as described above - Build the driver in the /usr/src/sys/modules/bce directory with the command "make". - Run the command "nm if_bce.ko | grep dump_stat" in the same directory with the kernel module just built. In my case I only see a symbol for bce_dump_status_block, but there is a second routine called bce_dump_stats_block. In my working build there are 23 functions that start with "bce_dump" but only 8 are displayed with the command "nm if_bce.ko | grep bce_dump". Of course, I also get the symbol not present error when I try to use any of those missing symbols through a "call" command in the debugger. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Vlan EVENT patch
> > So, it won't hurt if it shows up in ifconfig, and it can be > controlled, but > > definitely helps our job, the administrators'. > > OK, if people feel strongly about this, and someone wants to implement > the > code in ifconfig I'll go along with it. I believe there are some valid reasons when you consider virtualization with Xen where you need to pass VLAN tags to the hypervisor. Not sure if Xen is running on FreeBSD yet but I can definitely see the need. dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Enabling MSI-X on -CURRENT for New Network Driver
I'm working on adding MSI-X support for a new network driver
and having some difficulty in actually getting an interrupt.
Does this look right?
/* Select and configure the IRQ. */
sc->bxe_msix_count = pci_msix_count(dev);
rid = 1;
/* Try allocating MSI-X interrupts. */
if ((sc->bxe_cap_flags & BXE_MSIX_CAPABLE_FLAG) &&
(bxe_msi_enable >= 2) && (sc->bxe_msix_count > 0)) {
int msix_needed = sc->bxe_msix_count;
if (pci_alloc_msix(dev, &sc->bxe_msix_count) == 0) {
if (sc->bxe_msix_count == msix_needed) {
DBPRINT(sc, BXE_INFO_LOAD, "%s(): Using %d
MSI-X "
"vector(s).\n", __FUNCTION__,
sc->bxe_msix_count);
sc->bxe_flags |= BXE_USING_MSIX_FLAG;
} else {
pci_release_msi(dev);
sc->bxe_flags &= ~BXE_USING_MSIX_FLAG;
sc->bxe_msix_count = 0;
}
}
}
/* Try allocating MSI interrupts if we didn't get MSI-X. */
...
/* Try legacy interrupt. */
...
/* Allocate the interrupt and report any errors. */
sc->bxe_res_irq = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid,
RF_ACTIVE);
/* Report any IRQ allocation errors. */
if (sc->bxe_res_irq == NULL) {
BXE_PRINTF("%s(%d): PCI map interrupt failed!\n",
__FILE__, __LINE__);
rc = ENXIO;
goto bxe_attach_fail;
}
sc->bxe_irq_rid = rid;
sc->bxe_intr = bxe_intr;
The allocation doesn't fail and I usually see an IRQ allocated to
the driver using "vmstat -i" (though not always):
===[root] /usr/src/sys/modules/bxe # vmstat -i
interrupt total rate
irq1: atkbd0 1 0
irq4: sio0 46432 6
irq6: fdc010 0
irq14: ata0 58 0
irq17: atapci1 42684 5
cpu0: timer 15331063 1999
irq256: em0 917 0
cpu3: timer 15330811 1999
cpu1: timer 15330808 1999
cpu2: timer 15330811 1999
cpu5: timer 15330811 1999
cpu6: timer 15330810 1999
cpu4: timer 15330806 1999
cpu7: timer 15330811 1999
irq258: bxe0 2 0
Total 122736835 16010
But my interrupt handler doesn't seem to be called.
The goal is to get a single interrupt working first,
multiple queue support comes next. Any ideas?
Dave
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Status of Multi-Queue (RSS) Support in -CURRENT
I'm working on implementing multi-queue support for a 10Gb device on FreeBSD and I wanted to find out the current state of the OS with regards to supporting this. It seems that support for multiple receive queues can be done today since most of the routing is done in hardware but the transmit side is a different story. I've seen some things in the cxgb driver that suggest changes to the OS (such as a m_pkthdr.rss_hash field) but I don't see any OS code to back that usage model up. What's the state of the art in multi-queue support for FreeBSD? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: kern/128833: [bge] Network packets corrupted when bge card is in 64-bit PCI slot
> Ok, thanks for testing anyway. > I still think that this isn't really a driver bug though but > you are hitting some hardware-related problem like f.e. a > silicon bug and the question is how to work around it. There is a documented errata for the 5701 A3 where a 64bit DMA read can be terminated early by the bridge and then completed later as a 32bit access, causing corruption on the 5701. The errata states this type of behavior is rare in bridges and that the workaround is to force 32bit operation (set bit 15 of register 0x6800). It's not clear whether this errata is occurring without seeing a but trace but it certainly sounds right. The only question would be knowing "when" to force 32bit operation. Should it be done in all cases or only for this bridge? > Looking at the bge(4) versions of the other BSDs and the > corresponding Linux and OpenSolaris drivers I can't spot > a such a workaround apart from the already known PCI-X > issue, unfortunately. The only other thing that comes to > my mind is that you might suffer from sort of the opposite > of the problem worked around by ti_64bitslot_war() (the NICs > driven by ti(4) are the predecessors of those supported by > bge(4)). Given that this also involves the BIOS that could > then explain why you're see first person to hit this problem. > Could you please instrument bge(4) to print the content > of the BGE_PCI_PCISTATE register and report back which > values it's initialized to depending on which type of slot > the card is plugged into? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: kern/128833: [bge] Network packets corrupted when bge card is in 64-bit PCI slot
The following reply was made to PR kern/128833; it has been noted by GNATS. From: "David Christensen" <[EMAIL PROTECTED]> To: "Marius Strobl" <[EMAIL PROTECTED]> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> Subject: RE: kern/128833: [bge] Network packets corrupted when bge card is in 64-bit PCI slot Date: Mon, 24 Nov 2008 11:28:10 -0800 > > There is a documented errata for the 5701 A3 where a 64bit DMA > > read can be terminated early by the bridge and then completed > > later as a 32bit access, causing corruption on the 5701. The > > errata states this type of behavior is rare in bridges and that > > the workaround is to force 32bit operation (set bit 15 of register > > 0x6800). It's not clear whether this errata is occurring without > > seeing a but trace but it certainly sounds right. The only > > question would be knowing "when" to force 32bit operation. Should > > it be done in all cases or only for this bridge? > > >=20 > David, >=20 > could it be that bug doesn't only affect 5701 A3 but also > B3 (i.e. chipid 0x0105) as in this case or even all 5701 > revisions?=20 I checked the assembly instructions for the 5701 and even though the ASIC ID decodes as B5, the revision of the chip is actually A3. (You should be able to verify this as the silkscreen on the part should show "P13".) Unfortunately the "friendly" revision of the chip doesn't match the "ASIC" revision of the chip for the 5701 and the errata references the "friendly" name. The result is that the part you know as B5 is affected by this errata. Other versions of the chip (A2 which you know as B2 and A1 which you=20 know as B1) are not subject to this errata. > How does the problem you describe relate to the > 5701 PCI-X issue, which we align the RX buffer differently > for as a workaround, would that problem also be avoided > by limiting 5701 to 32-bit operations? Or is the A3-errata > you described an entirely different issue and limited to 5701 > in a 64-bit non-PCI-X slot, or would 5701 in a PCI-X slot > even require both workarounds? Which PCI-X issue are you referring to? Can you point me to the line number on http://fxr.watson.org/fxr/source/dev/bge/if_bge.c? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Gathering Hardware State During a Driver Initiated Kernel Panic
Is there a method or callback in FreeBSD where a driver can be notified that it has caused a kernel panic in order to generate a dump of internal hardware state information? I've written a sysctl call for manual intervention and can handle some "expected" hardware events completely in the driver but I don't know of a way to get control again in cases where the driver wasn't expecting a failure. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: kern/128833: [bge] Network packets corrupted when bge card is in 64-bit PCI slot
The following reply was made to PR kern/128833; it has been noted by GNATS. From: "David Christensen" <[EMAIL PROTECTED]> To: "Marius Strobl" <[EMAIL PROTECTED]> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> Subject: RE: kern/128833: [bge] Network packets corrupted when bge card is in 64-bit PCI slot Date: Mon, 24 Nov 2008 14:38:35 -0800 > > > How does the problem you describe relate to the > > > 5701 PCI-X issue, which we align the RX buffer differently > > > for as a workaround, would that problem also be avoided > > > by limiting 5701 to 32-bit operations? Or is the A3-errata > > > you described an entirely different issue and limited to 5701 > > > in a 64-bit non-PCI-X slot, or would 5701 in a PCI-X slot > > > even require both workarounds? > > > > Which PCI-X issue are you referring to? Can you point me to > > the line number on http://fxr.watson.org/fxr/source/dev/bge/if_bge.c? > > >=20 > I was refering to BGE_FLAG_RX_ALIGNBUG, the lines dealing with it > are 874-875, 933-934, 2698-2708 and 3112-3122. The Linux tg3 > driver does pretty much the same via rx_offset. It's a different problem. The RX_ALIGNBUG is described in the errata as follows: "Description: In PCI-X mode, on rare instances, the DMA write engine=20 can incorrectly DMA duplicate data to the host if the first word of=20 the data being transferred is to a non-zero-offset address (offset=20 from the 8-byte boundary). Workaround: Align buffers to zero offset in the driver." I suppose you could force PCI mode for this problem too though at the expense of possible reduced performance. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Gigabit ethernet questions?
> Greeting colleagues. I've got two DL-360(pciX bus) servers, > with BCM5704 NetXtreme Dual Gigabit Adapters(bge). The Uname > is 6.1-RELEASE-p3. The bge interfaces of the both servers are > connected with each other with a cat6 patchcord. >Here are my settings: > kernel config: > options DEVICE_POLLING > options HZ=1000 # > > sysctl.conf: > kern.polling.enable=1 > net.inet.ip.intr_queue_maxlen=5000 > kern.ipc.maxsockbuf=8388608 > net.inet.tcp.sendspace=3217968 > net.inet.tcp.recvspace=3217968 > net.inet.tcp.rfc1323=1 > > bge1: flags=8843 mtu 9000 > options=5b > inet 192.168.0.1 netmask 0xff00 broadcast 192.168.0.255 > ether 00:17:a4:3a:e1:81 > media: Ethernet autoselect (1000baseTX ) > status: active > (note mtu 9000) > > and here are tests results: > > netperf: > > TCP STREAM TEST to 192.168.0.1 > Recv SendSend > Socket Socket Message Elapsed > Size SizeSize Time Throughput > bytes bytes bytessecs.10^6bits/sec > > 6217968 6217968 621796810.22 320.04 > > UDP UNIDIRECTIONAL SEND TEST to 192.168.0.1 > Socket Message Elapsed Messages > SizeSize Time Okay Errors Throughput > bytes bytessecs# # 10^6bits/sec > > 92169216 10.00 118851 1724281 876.20 > 41600 10.00 0 0.00] > > > > iperf: > gate2# iperf -s -N > > Server listening on TCP port 5001 > TCP window size: 3.07 MByte (default) > > [ 4] local 192.168.0.2 port 5001 connected with 192.168.0.1 > port 52597 > [ 4] 0.0-10.1 sec384 MBytes319 Mbits/sec > > Also I can say, that I've managed to achieve about 500mbit.s > by tuning tcp window with -w key in iperf. > > How can we explain such a low tcp performance? What else is > to tune? Is there somebody who achieved gigabit speed with > tcp on freebsd? You're test is non-optimal for the 5704 since the ports are linked together back-to-back. In a dual port configuration such as the 5704 each port must arbitrate for access to the PCI bus. Due to an errata for the 5704, the BGE_PCIDMAWCTL_ONEDMA_ATONCE bit is set which allows only one port access to the PCI bus at a time for the duration of the DMA transaction, rather than allowing the two ports to interleave DMAs. Your test configuration is a worst case scenario since both ports are active in both directions at the same time. If you can change your test to use a second system you should see the TCP performance rise substantially. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Problem with IBM NetXtreme 1000-T GigaEthernet Adapter
> Dear list, > I'm running the latest FreeBSD 6.1-RELEASE-p3 on the > following piece of > hardware:- > IBM xSeries 336 > - 1 add-on PCI dual-port NIC (IBM Netxtreme Dualpt Ethernet adapter) > matching the Broadcom BCM5704C Dual Gigabit Ethernet chipset (detected > as bge0 and bge1) > - 2 internal built-in NICs matching the Broadcom BCM5721 Gigabit > Ethernet chipset. (detected as bge2 and bge3) > > The internal NICs work fine (bge2 and bge3), however the > add-on PCI NIC > has the following issues:- > On a GB Ethernet switchport (I am using Cisco Catalyst C3750G):- > - When set to auto-detect, or forced 1000baseTX on both sides > (switchport and server), the LED on both switchport and NIC dies the > moment FreeBSD OS is booting (i.e. when FreeBSD bge driver activated). > Previously, it was working fine at BIOS level. > - The only working setup is forced 100baseTX full-duplex on both sides > (switchport and server), however with intermittent link (ping yields > packet loss every 2-3 out of 10 packets or so). > > I tried to use a FE version of the switch (Cisco Catalyst > C3750), and a > single-port version of the said NIC, with the same results > (auto detect > fails, and can only live with intermittent forced 100baseTX > full-duplex). There isn't a single port version of the 5704, it must be a different controller (maybe the 5703?). Which one is it exactly? Do either the LOM devices or the add-in boards support remote management (such as Serial-over-LAN or IPMI)? If so does disabling the feature change the problem? > > Any pointers would be appreciated -- thanks. :) > If you could attach a dump of dmesg that shows the messages from the driver that might help too. > --mendonan > "Yang mimpikan secangkir kopi panas dengan selimut.." > (Dreaming of a cup of hot coffee, and a blanket..") > ___ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > > ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: bge(4) one packet wedge
This "lost interrupt" type of problem is addressed by the use of the status_tag field in the status block. (Listed as bge_rsvd0 in the bge_status_block structure). Everytime the status block is updated a new tag value is written to the status block. When the ISR starts the driver should record the status_tag value. At the end of the ISR, the driver should compare the current status_tag value is the status block with the value recorded on entry to the ISR. If the values are the same then no additional status block updates have occurred so there shouldn't be any packets hanging around. If the values are different then additional packets or completions are waiting around so the ISR should loop around again. At the end of the ISR the driver will write the status_tag value it last handled to a mailbox register, letting the hardware know the last status block update handled. If necessary the hardware will generate a new interrupt and start the process over again. This entire process should be included in the Linux driver, I don't see it being used in the bge driver (bge_intr()). Dave > -Original Message- > From: Gleb Smirnoff [mailto:[EMAIL PROTECTED] > Sent: Wednesday, August 23, 2006 9:17 AM > To: David Christensen > Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: bge(4) one packet wedge > > Colleagues, > > I've faced a problem in bge(4) when a single packet is in the RX > ring, but it isn't noticed by the driver. A reception of a packet > triggers interrupt and both packets are processed - an old one > and the new one. > > To reproduce the problem you need to run netperf (from ports > collection): netserver on another host (10.0.0.1) and netperf on > the host, where tested bge(4) is installed - 10.0.0.2. No traffic > except netperf's should flow through this NIC, or the problem won't > be reproduced! > > So, I run netperf client and simultaneously tcpdump on the > another host. After few seconds there is a wedge. The last packet > seen on 10.0.0.1 is the packet sent by 10.0.0.1 to 10.0.0.2. However > it isn't seen on 10.0.0.2. > > Ok, let's look at the receive ring: > > (kgdb) p $sc->bge_rx_saved_considx > $14 = 51 > (kgdb) p $sc->bge_ldata.bge_status_block->bge_idx[0].bge_rx_prod_idx > $15 = 51 > > Looks like there is nothing to process. > > However, if I run 'ping -c 1 10.0.0.2' I will get an > interrupt and read > two packets: first the old packet, and then recently sent ping. > > -- > Totus tuus, Glebius. > GLEBIUS-RIPN GLEB-RIPE > > ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: bge(4) one packet wedge
> On Wed, Aug 23, 2006 at 12:53:49PM -0700, David Christensen wrote: > > This "lost interrupt" type of problem is addressed by the use of the > > status_tag > > field in the status block. (Listed as bge_rsvd0 in the > bge_status_block > > structure). > > Everytime the status block is updated a new tag value is > written to the > > status block. > > When the ISR starts the driver should record the status_tag > value. At > > the end > > of the ISR, the driver should compare the current > status_tag value is > > the status > > block with the value recorded on entry to the ISR. If the > values are > > the same > > then no additional status block updates have occurred so > there shouldn't > > be > > any packets hanging around. If the values are different > then additional > > packets > > or completions are waiting around so the ISR should loop > around again. > > At the > > end of the ISR the driver will write the status_tag value it last > > handled to a > > mailbox register, letting the hardware know the last status > block update > > handled. > > If necessary the hardware will generate a new interrupt and > start the > > process over > > again. > > > > This entire process should be included in the Linux driver, > I don't see > > it being > > used in the bge driver (bge_intr()). > > > > Dave > > > > Could you please answer few questions? > > 1) I've found status tag is returned in status block even if > bit 9 of Misc. >Host Control Register is not set, is it ok? Which controller are you using? This bit is reserved on the 5700 (which didn't support tagged status mode) but all other controllers should support it and all of the Broadcom drivers will always enable it. > > 2) Status tag is returned in bits 0-7 of status tag field of > status block, >as long as i know it should be returned in bits 31-24, is it ok? Not sure what you mean by this. If you're seeing it in bits 31-24 of the status block then there may be an endian issue on your system. Check that byte/word swapping is set correctly. > > 3) If i try to return processed tag (at the end of ISR) in > Mailbox 0 register: >CSR_WRITE_4(sc, BGE_MBX_IRQ0_LO, status_tag << 24); >it would lead to disabled interrupts. >I've thought this should not happen cause in_isr bits > (0-23) are cleared. Writing a non-zero value to bits 23:0 will cause the interrupt to be disabled: CSR_WRITE_4(sc, BGE_MBX_IRQ0_LO, 1); Writing all zeros to bits 23:0 will enable interrupts. CSR_WRITE_4(sc, BGE_MBX_IRQ0_LO, status_tag << 24); The tag value written to bits 31:24 does not affect the interrupt state. > > -- > Oleg. > > > ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: bge(4) one packet wedge
> > > Could you please answer few questions?
> > >
> > > 1) I've found status tag is returned in status block even if
> > > bit 9 of Misc.
> > >Host Control Register is not set, is it ok?
> >
> > Which controller are you using? This bit is reserved on the 5700
> > (which didn't support tagged status mode) but all other controllers
> > should support it and all of the Broadcom drivers will always
> > enable it.
>
> I tested it on bcm5701 & bcm5721
>
> >
> > >
> > > 2) Status tag is returned in bits 0-7 of status tag field of
> > > status block,
> > >as long as i know it should be returned in bits 31-24,
> is it ok?
> >
> > Not sure what you mean by this. If you're seeing it in bits 31-24
> > of the status block then there may be an endian issue on
> your system.
> > Check that byte/word swapping is set correctly.
> >
> > >
> > > 3) If i try to return processed tag (at the end of ISR) in
> > > Mailbox 0 register:
> > >CSR_WRITE_4(sc, BGE_MBX_IRQ0_LO, status_tag << 24);
> > >it would lead to disabled interrupts.
> > >I've thought this should not happen cause in_isr bits
> > > (0-23) are cleared.
> >
> > Writing a non-zero value to bits 23:0 will cause the interrupt to
> > be disabled:
> >
> > CSR_WRITE_4(sc, BGE_MBX_IRQ0_LO, 1);
> >
> > Writing all zeros to bits 23:0 will enable interrupts.
> >
> > CSR_WRITE_4(sc, BGE_MBX_IRQ0_LO, status_tag << 24);
> >
> > The tag value written to bits 31:24 does not affect the interrupt
> > state.
>
> Yes, this is i'm talking about:
> CSR_WRITE_4(sc, BGE_MBX_IRQ0_LO, 0x0100) would lead to _disabled_
> interrupts.
> Tested on bcm5701 & bcm5721.
Here's how it's done in Linux:
static void tg3_disable_ints(struct tg3 *tp)
{
tw32(TG3PCI_MISC_HOST_CTRL,
(tp->misc_host_ctrl | MISC_HOST_CTRL_MASK_PCI_INT));
tw32_mailbox_f(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW,
0x0001);
}
static void tg3_enable_ints(struct tg3 *tp)
{
tp->irq_sync = 0;
wmb();
tw32(TG3PCI_MISC_HOST_CTRL,
(tp->misc_host_ctrl & ~MISC_HOST_CTRL_MASK_PCI_INT));
tw32_mailbox_f(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW,
(tp->last_tag << 24));
if (tp->tg3_flags2 & TG3_FLG2_1SHOT_MSI)
tw32_mailbox_f(MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW,
(tp->last_tag << 24));
tg3_cond_int(tp);
}
>
> P.S. bcm5705 does not support tagged status mode, am i right?
No, the 5705 should support tagged status mode, only the 5788 doesn't
support it. The 5705 and 5788 are very closely related though.
> --
> Oleg.
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: bge(4) one packet wedge
> D> This "lost interrupt" type of problem is addressed by the > use of the > D> status_tag > D> field in the status block. (Listed as bge_rsvd0 in the > bge_status_block > D> structure). > D> Everytime the status block is updated a new tag value is > written to the > D> status block. > D> When the ISR starts the driver should record the > status_tag value. At > D> the end > D> of the ISR, the driver should compare the current > status_tag value is > D> the status > D> block with the value recorded on entry to the ISR. If the > values are > D> the same > D> then no additional status block updates have occurred so > there shouldn't > D> be > D> any packets hanging around. If the values are different > then additional > D> packets > D> or completions are waiting around so the ISR should loop > around again. > D> At the > D> end of the ISR the driver will write the status_tag value it last > D> handled to a > D> mailbox register, letting the hardware know the last > status block update > D> handled. > D> If necessary the hardware will generate a new interrupt > and start the > D> process over > D> again. > D> > D> This entire process should be included in the Linux > driver, I don't see > D> it being > D> used in the bge driver (bge_intr()). > > True, this isn't done in FreeBSD driver. First I started to > work on this, > but just noticed that I am seeing this problem on 5700 chip, that is > known not to support the status tag. > > What can be the problem for not updated status block in the 5700 case? In general the problem isn't that the status block isn't being updated, but that the status update occurs AFTER the ISR has stopped looking at the status block, but before the ISR has re-enabled interrupts, thus missing the update. That's why tagged status mode is an improvement, the ISR actively tells the hardware which status block update it last processed and forces the hardware to generate another interrupt if a status block update was missed by the ISR. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: bge(4) one packet wedge
> On Wed, Aug 23, 2006 at 12:53:49PM -0700, David Christensen wrote:
> D> This "lost interrupt" type of problem is addressed by the
> use of the
> D> status_tag
> D> field in the status block. (Listed as bge_rsvd0 in the
> bge_status_block
> D> structure).
> D> Everytime the status block is updated a new tag value is
> written to the
> D> status block.
> D> When the ISR starts the driver should record the
> status_tag value. At
> D> the end
> D> of the ISR, the driver should compare the current
> status_tag value is
> D> the status
> D> block with the value recorded on entry to the ISR. If the
> values are
> D> the same
> D> then no additional status block updates have occurred so
> there shouldn't
> D> be
> D> any packets hanging around. If the values are different
> then additional
> D> packets
> D> or completions are waiting around so the ISR should loop
> around again.
> D> At the
> D> end of the ISR the driver will write the status_tag value it last
> D> handled to a
> D> mailbox register, letting the hardware know the last
> status block update
> D> handled.
> D> If necessary the hardware will generate a new interrupt
> and start the
> D> process over
> D> again.
> D>
> D> This entire process should be included in the Linux
> driver, I don't see
> D> it being
> D> used in the bge driver (bge_intr()).
>
> In the Linux driver for the not tag capable controllers there
> is a following
> comment and code:
>
> /* In INTx mode, it is possible for the interrupt to arrive at
> * the CPU before the status block posted prior to
> the interrupt.
> * Reading the PCI State register will confirm whether the
> * interrupt is ours and will flush the status block.
> */
> if ((sblk->status & SD_STATUS_UPDATED) ||
> !(tr32(TG3PCI_PCISTATE) & PCISTATE_INT_NOT_ACTIVE)) {
>
> So, I tried to add (void)pci_read_config(sc->bge_dev,
> BGE_PCI_PCISTATE, 4)
> to the bge_intr(). Unfortunately, this didn't help.
>
This fix is really designed to handle bridges that haven't posted the
status
block memory write ahead of the PCI interrupt and I wouldn't expect it
to
resolve the reported problem. A better solution would be to try and
minimize the time between the last status block check and the time the
interrupt is re-enabled. Unfortunately I don't think the problem can be
completely eliminated with even the most creative coding solutions but
I'd be happy to be proven wrong.
Dave
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Rapid link state changes on bge(4) interface
> I'm testing network failover on IBM BladeCenter running FreeBSD 6.1 > STABLE for Sep 6th. > > I suspect a problem with link state change detection in bge code. When > I disable internal port on chassis built-in ethernet switch, kernel > floods syslog with messages about link state changes and coalescing > them. Log snippet follows: > > Sep 13 14:58:28 w3-6 kernel: bge1: link state changed to DOWN > Sep 13 14:58:28 w3-6 kernel: bge1: link state changed to UP > Sep 13 14:58:29 w3-6 kernel: bge1: link state changed to DOWN > Sep 13 14:58:29 w3-6 kernel: bge1: link state changed to UP > Sep 13 14:58:29 w3-6 kernel: bge1: link state changed to DOWN > Sep 13 14:58:29 w3-6 kernel: bge1: 4 link states coalesced > Sep 13 14:58:29 w3-6 kernel: bge1: link state changed to DOWN > Sep 13 14:58:29 w3-6 kernel: bge1: 11 link states coalesced > Sep 13 14:58:29 w3-6 kernel: bge1: link state changed to UP > Sep 13 14:58:30 w3-6 kernel: bge1: link state changed to DOWN > Sep 13 14:58:30 w3-6 kernel: bge1: 3 link states coalesced > Sep 13 14:58:30 w3-6 kernel: bge1: link state changed to UP > Sep 13 14:58:30 w3-6 kernel: bge1: 7 link states coalesced > Sep 13 14:58:30 w3-6 kernel: bge1: link state changed to DOWN > Sep 13 14:58:30 w3-6 kernel: bge1: 4 link states coalesced > Sep 13 14:58:30 w3-6 kernel: bge1: link state changed to DOWN > Sep 13 14:58:30 w3-6 kernel: bge1: 2 link states coalesced > > As you can see, messages are generated in rapid succession and > therefore any probing of link state change by ng_one2many for > interface failover is meaningless. Ethernet switch doesn't register > and log any interface state changes after disabling this port. LS20 > blades use chipset 8850. My firmware is 3.38, full changelog, if it is > of any help, is here: > > http://www-307.ibm.com/pc/support/site.wss/license.do?filename =pc_servers/brcm_fw_nic_12021_anyos_anycpu.chg > > Any ideas what might be wrong? I can't access the information on this web site through Mozilla after clicking "I Accept". Is this a 5704 controller using a SerDes link? I'm familiar with some Blade Center problems in the past (which I think were related to Sigdet) though I'm not in the office and can't look into it right away. A comparison between the serdes code in the Linux driver vs. the FreeBSD driver is probably the first area to investigate. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: bge(4) one packet wedge
> Finally I got a NIC that has a chip that does tagged status > block - 5701. > > I've prepared a patch, that mimics Linux. If a chip can do status tag, > then we write it to mailbox register at end of ISR, as you have > described. If the chip can't, then we force coalescing once > per second. > This should fix the problem correctly on the chips that support > status tag, and it is an ugly fix for chips that does not. > > Unfortunately, the attached patch doesn't fix the problem on 5701. The > wedge occurs as before. And I see status tag updated, while > the netperf > test has wedged: > > (kgdb) p $sc->bge_last_tag > $45 = 239 > (kgdb) p $sc->bge_last_tag > $46 = 240 > (kgdb) p $sc->bge_last_tag > $47 = 241 > (kgdb) p $sc->bge_last_tag > $48 = 242 > > I have no idea. :( May be you have one? What's happening to the status block consumer/producer indices at this time? Are they advancing? Do they match the driver maintained consumer/producer indices? I forget if this was a send or receive problem, is the packet sitting in the send or receive return ring? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Broadcom (bce) support and POLLING
Polling is not currently supported in the bce driver. Dave > Hi, > We have just upgraded to a Dell 2950 for our main BGP router. > With the constant rise in traffic through the box, I have started to > notice some dropped packets so I figured I would take a look > at putting > the interfaces into polling more to squeeze a little more packets > through them. > > I have added the DEVICE_POLLING and HZ=1000 to the kernel. I > turned on polling for the bce interfaces and they pass traffic at low > volumes (like ping tests). If I put my fluke traffic generators on > either side of the box and try to ramp it up to even 100Mb > (they are Gig > links) then the interfaces will stop responding. Turning off polling > will get them to respond again. > > I know that the bce support in still quite new. Has anyone else > testing with polling and the bce interfaces? Is there any more > information that I can get for the developers to help track this down. > The system is not in production right now so I can use it for testing. > > As another note... I added some Intel (em) cards into the box > and tested with them. Polling worked great on them but I was > only able > to get about 200k packets per second before it starts > dropping packets. > Is this about what I should expect? > > [root@ ~]# uname -a > FreeBSD 6.2-RC1 FreeBSD 6.2-RC1 #2: Tue Nov 28 18:10:56 MST 2006 > root@:/usr/obj/usr/src/sys/INET-GW i386 > > Kirk > > > > ___ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > > ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: [Fwd: Re: bge Ierr rate increase from 5.3R -> 6.1R]
> These happen under loads that can't be handled, and generally cause > thousands of input errors every second. The hardware records dropped > packets separately from other input errors, but unfortunately > all types > of input errors are counted together in if_ierrors, and I haven't done > more than muck around in ddb to separate them. There are a couple places I can suggest you look to understand if there are any resource limitations being encountered by the controller. The first is register 0x4450 (bits 15:0 for 5700 to 5704 and bits 8:0 for other controllers) which shows the number of internal free RX MBUFs on the controller. (In this case an MBUF is an internal 128 byte data structure used by the controller.) This value is especially important on 5700 to 5704 devices since those devices share a pool of MBUFs between the RX and TX data paths, unlike the 5705 and later devices which use separate MBUF pools for RX and TX data. As the value approaches 0, the controller will either start to generate pause frames or simply drop packets as specified by the values in registers 0x4410, 0x4414, and 0x4418. If you see ifInDiscards incrementing then this is a definite possibility. You would also see bit 4 of register 0x4400 set if this occurs. I also noticed on -CURRENT that register 0x2c18 was set to a rather high value (0x40). This register controls how many bge_rx_bd's will be fetched at one time. A high value will reduce PCI bus utilization as it will fetch more BD's in a single transaction, but it will increase latency and could potentially deplete the available internal MBUFs as the controller waits for the driver to populate additional bge_rx_bd's. Try using a smaller value such as 0x08. It would be really interesting if you could add a sysctl that would bring out the hardware statistics maintained by the device, similar to what exists in the "bce" driver. With this information we could focus on individual blocks to see where the packet loss is occurring and may be able to come up with more ideas on tuning the controller. > bge also has a "mini" rx ring which FreeBSD doesn't use. I > don't really > understand this or the interaction of the separate rx rings, but hope > that the mini ring can be used to handle small packets and would only > need an mbuf (not an mbuf cluster) for each packet, and with > it and the > jumbo ring the total hardware buffering would be 1024(mini) + > 512(std) + > 256(jumbo), with the 1024-entry event ring only used to communicate > with the host and its size not really limiting buffering. Meeting a > latency requirement of 1024+512 tiny-packet times is much easier than > meeting one of 192 or 20 tiny-packet times. (I only actually saw the > limits of 20 and 192 for full-sized (non jumbo) packets.) Mini rings are only supported by 57XX controllers that support external SSRAM, and the only controller that supports that is the 5700. This feature was never used in production so I wouldn't consider it an option since you'll never find the right hardware to support it. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Adding Proprietary Link Speed to sys/net/if_media.h
I'm working on improving the fiber support for the bce driver. One of the devices supported by the driver (the 5708) supports a 2.5G mode which is negotiated through next pages operations during normal auto negotiation. The mechanism for negotiating this speed uses standards based mechanisms, but the final speed is currently only supported on Broadcom switches and controllers. Since this speed is really intended for blade enclosures that's normally not a problem but since I need to modify sys/net/if_media.h to create a new media type I wanted to see if there were any strong objections to such a change. The media type will also be supported on the 5709, a forth-coming controller which will also support 2.5G operation, so the media type won't be limited to a single controller, though it may never be supported outside of Broadcom products. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Adding Proprietary Link Speed to sys/net/if_media.h
> > I'm working on improving the fiber support for the bce > driver. One of > > Should be fine but you might want to post a patch to be concrete. > > Sam I'm just looking at adding a media type as follows: #define IFM_2500_SX 21 /* 2500BaseSX - multi-mode fiber */ I've only briefly explored whether any other changes are necessary anywhere outside of my driver. If anyone has any thoughts on where I should consider more changes I'd appreciate the feedback. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Dell server responding with two mac addresses
> I have a rather serious problem, I have three dell server (PE1800, > PE2950 and SC1425) the PE1800 is running FreeBSD 6.1. The > problem that I > have is that the PE1800 is responding with two mac adresses > on the same > network interface, has anywone seen this before? This causes > our router > to be confused on where to send the traffic so one of the > other machines > can't access anything outside our router. The first thought was that > dells IPMI-interface would have something to do with it but after > disabling IPMI on all machines the problem is still there. > What LOM does the PE1800 use? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: HEADS UP: Plan to MFC new em driver
> I have a version of code ready to MFC, the big difference with CURRENT > is that TSO is #ifdef'd off until Andre is able to get that back. Is something broken with TSO? I just added TSO support to bce on CURRENT and was planning on MFC'ing to RELENG_6 within the next week. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Problems with BCE network adapter (Dell PE2950)
It's not a problem I've seen before. The driver seems to be passing an Ethernet frame up the stack with a length of -4. Is there anything unusual about your network setup like VLAN tagging, jumbo frames, or anything else that might cause the Ethernet frame to be formatted in an unusual manner? The error probably occurs in bce_rx_intr() but I'd need to see the packet that's causing the problem to figure out why the driver isn't calculating the packet length correctly. Dave > I am seeing some problems with one of my Dell PowerEdge > 2950's (running > RELENG_6_2) on board bce NICs. The interface seems to crash with the > following errors, to which the fix seems to be and "ifconfig > bce0 down; > ifconfig bce0 up": > > Jun 7 12:20:29 gonzo kernel: bce0: discard frame w/o leading > ethernet > header (len 4294967292 pkt len 4294967292) > Jun 7 12:20:29 gonzo last message repeated 54 times > Jun 7 12:20:58 gonzo kernel: nfs server nfs-server:/usr/home: not > responding > > > Is this a know problem? If so is there a solution? > > Tom > > #uname -a > FreeBSD gonzo.mintel.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE > #10: Thu Apr > 5 10:53:39 BST 2007 > [EMAIL PROTECTED]:/usr/obj/usr/src/sys/PE2950 amd64 > amd64 Intel(R) > Xeon(R) CPU 5160 @ 3.00GHz FreeBSD > > > ## dmesg.boot snippet > bce0: mem > 0xf400-0xf5ff irq 16 at device 0.0 on pci9 > bce0: ASIC ID 0x57081020; Revision (B2); PCI-X 64-bit 133MHz > miibus0: on bce0 > brgphy0: on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, > 1000baseTX-FDX, auto > bce0: Ethernet address: 00:18:8b:88:d8:81 > > ## pciconf -lv snippet > [EMAIL PROTECTED]:0:0: class=0x02 card=0x01b21028 > chip=0x164c14e4 rev=0x12 > hdr=0x00 > vendor = 'Broadcom Corporation' > class= network > subclass = ethernet > > #ifconfig bce0 > bce0: flags=8843 mtu 8192 > options=3b > inet 172.31.0.13 netmask 0xff00 broadcast 172.31.0.255 > inet 172.31.0.157 netmask 0x broadcast 172.31.0.157 > inet 172.31.0.161 netmask 0x broadcast 172.31.0.161 > ether 00:18:8b:88:d8:81 > media: Ethernet autoselect (1000baseTX ) > status: active > > #Switch port counters > console# show interfaces counters ethernet g10 >PortInOctets InUcastPkts InMcastPkts InBcastPkts > -- --- --- --- >g102615415297 875639870 13480 > >Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts > -- >g102535595313 136391705 10740316 1420686 > > FCS Errors: 0 > Single Collision Frames: 0 > Late Collisions: 0 > Excessive Collisions: 0 > Internal MAC Tx Errors: 0 > Oversize Packets: 0 > Internal MAC Rx Errors: 0 > Received Pause Frames: 0 > Transmitted Pause Frames: 0 ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Problems with BCE network adapter (Dell PE2950)
> Tom Judge wrote: > > [skip] > > > Vladimir, > > > > Is this the patch attached to your post titled "Serious bug in most > > (?) ethernet drivers (bge, bce, ixgb etc.).". > Exactly. > That patch was already incorporated in the bce driver (if_bce.c, v1.30) on May 4th, 2007. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Problems with BCE network adapter (Dell PE2950)
> David Christensen wrote: > >> Tom Judge wrote: > >> > >> [skip] > >> > >>> Vladimir, > >>> > >>> Is this the patch attached to your post titled "Serious > bug in most > >>> (?) ethernet drivers (bge, bce, ixgb etc.).". > >> Exactly. > >> > > > > That patch was already incorporated in the bce driver > (if_bce.c, v1.30) > > on May 4th, 2007. > > > > Dave > > > This wouldn't be in 6.2-p5 would it? Will the bce driver in RELENG_6 > have this patch if so will it back port easily to 6.2-p5? The patch was MFC'd to RELENG_6 in if_bce.c (v1.2.2.15) on June 5th. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Problems with BCE network adapter (Dell PE2950)
Tom, There's already some debug code to watch for unusual size packets. If you can recompile the driver from HEAD with the attached diffs we can printout the first 128 bytes of any unusual sized packets. This does enabled other debugging code so performance will drop but that should be OK since this doesn't present as a performance problem. Dave > -Original Message- > From: Tom Judge [mailto:[EMAIL PROTECTED] > Sent: Thursday, June 07, 2007 2:09 PM > To: David Christensen > Cc: freebsd-net > Subject: Re: Problems with BCE network adapter (Dell PE2950) > > David Christensen wrote: > > It's not a problem I've seen before. The driver seems to be > > passing an Ethernet frame up the stack with a length of -4. > > Is there anything unusual about your network setup like VLAN > > tagging, jumbo frames, or anything else that might cause > > the Ethernet frame to be formatted in an unusual manner? > > The error probably occurs in bce_rx_intr() but I'd need to > > see the packet that's causing the problem to figure out why > > the driver isn't calculating the packet length correctly. > > > > Dave > > > Hi Dave, > > The network is running Jumbo frames (8192 Byte frames). There are 5 > PE2950's and 7 PE1950's with this NIC on this network. However it is > always the same system that causes the crash in the same > manner. I will > look into reproducing the error tomorrow with the switch port > mirrored > to a second server (with an em NIC). Hopefully I will get > some useful > dump files from tcpdump > > Do you have any idea what the frame might look like that > causes this error? > > Here is the NIC config from the system: > > >> #ifconfig bce0 > >> bce0: flags=8843 mtu 8192 > >> > options=3b > >> inet 172.31.0.13 netmask 0xff00 broadcast > 172.31.0.255 > >> inet 172.31.0.157 netmask 0x broadcast > 172.31.0.157 > >> inet 172.31.0.161 netmask 0x broadcast > 172.31.0.161 > >> ether 00:18:8b:88:d8:81 > >> media: Ethernet autoselect (1000baseTX ) > >> status: active > > If its any use the switch is a Dell PowerConnect 5324. > > Tom > > >> I am seeing some problems with one of my Dell PowerEdge > >> 2950's (running > >> RELENG_6_2) on board bce NICs. The interface seems to > crash with the > >> following errors, to which the fix seems to be and "ifconfig > >> bce0 down; > >> ifconfig bce0 up": > >> > >> Jun 7 12:20:29 gonzo kernel: bce0: discard frame w/o leading > >> ethernet > >> header (len 4294967292 pkt len 4294967292) > >> Jun 7 12:20:29 gonzo last message repeated 54 times > >> Jun 7 12:20:58 gonzo kernel: nfs server nfs-server:/usr/home: not > >> responding > >> > >> > >> Is this a know problem? If so is there a solution? > >> > >> Tom > >> > >> #uname -a > >> FreeBSD gonzo.mintel.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE > >> #10: Thu Apr > >> 5 10:53:39 BST 2007 > >> [EMAIL PROTECTED]:/usr/obj/usr/src/sys/PE2950 amd64 > >> amd64 Intel(R) > >> Xeon(R) CPU 5160 @ 3.00GHz FreeBSD > >> > >> > >> ## dmesg.boot snippet > >> bce0: mem > >> 0xf400-0xf5ff irq 16 at device 0.0 on pci9 > >> bce0: ASIC ID 0x57081020; Revision (B2); PCI-X 64-bit 133MHz > >> miibus0: on bce0 > >> brgphy0: on miibus0 > >> brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, > 1000baseTX, > >> 1000baseTX-FDX, auto > >> bce0: Ethernet address: 00:18:8b:88:d8:81 > >> > >> ## pciconf -lv snippet > >> [EMAIL PROTECTED]:0:0: class=0x02 card=0x01b21028 > >> chip=0x164c14e4 rev=0x12 > >> hdr=0x00 > >> vendor = 'Broadcom Corporation' > >> class= network > >> subclass = ethernet > >> > >> #ifconfig bce0 > >> bce0: flags=8843 mtu 8192 > >> > options=3b > >> inet 172.31.0.13 netmask 0xff00 broadcast 172.31.0.255 > >> inet 172.31.0.157 netmask 0x broadcast > 172.31.0.157 > >> inet 172.31.0.161 netmask 0x broadcast > 172.31.0.161 > >> ether 00:18:8b:88:d8:81 > >> media: Ethernet autoselect (1000baseTX ) > >> status: active > >> > >> #Switch p
RE: Problems with BCE network adapter (Dell PE2950)
> Sorry for the top post, but I have just managed to repeat is > exact crash > twice on a new PE 1950 system. I have core files available. > > It seems that after a couple of reboots the problem goes away. The > system actually crashed 4 times but 2 of the cores where corrupt. > > It also seems that the system will be stable if the following > message is > not produced shortly after /etc/rc.d/netif start: > > bce0: /usr/src/sys/dev/bce/if_bce.c(3489): Too many free > rx_bd (0xFFF9 > > 0x01FE)! > The error indicates that too many receive buffer descriptors were freed from the receive chain. The driver must be losing count somewhere. The process for duplicating the error sounds simple enough, how much data is in your NFS mounted directory? Are you using TCP or UDP for the NFS mount? Any idea what type of network activity is happening just after /etc/rc.d/netif start (DHCP, NTP, anything else)? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Problems with BCE network adapter (Dell PE2950)
> > Sorry for the top post, but I have just managed to repeat is > > exact crash > > twice on a new PE 1950 system. I have core files available. > > > > It seems that after a couple of reboots the problem goes away. The > > system actually crashed 4 times but 2 of the cores where corrupt. > > > > It also seems that the system will be stable if the following > > message is > > not produced shortly after /etc/rc.d/netif start: > > > > bce0: /usr/src/sys/dev/bce/if_bce.c(3489): Too many free > > rx_bd (0xFFF9 > > > 0x01FE)! > > > > The error indicates that too many receive buffer descriptors > were freed from the receive chain. The driver must be losing > count somewhere. The process for duplicating the error sounds > simple enough, how much data is in your NFS mounted directory? > Are you using TCP or UDP for the NFS mount? > > Any idea what type of network activity is happening just after > /etc/rc.d/netif start (DHCP, NTP, anything else)? > > Dave One other thing, are you using jumbo frames? What's your MTU setting? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Problems with BCE network adapter (Dell PE2950)
> > > Sorry for the top post, but I have just managed to repeat is > > > exact crash > > > twice on a new PE 1950 system. I have core files available. > > > > > > It seems that after a couple of reboots the problem goes > away. The > > > system actually crashed 4 times but 2 of the cores where corrupt. > > > > > > It also seems that the system will be stable if the following > > > message is > > > not produced shortly after /etc/rc.d/netif start: > > > > > > bce0: /usr/src/sys/dev/bce/if_bce.c(3489): Too many free > > > rx_bd (0xFFF9 > > > > 0x01FE)! > > > > > > > The error indicates that too many receive buffer descriptors > > were freed from the receive chain. The driver must be losing > > count somewhere. The process for duplicating the error sounds > > simple enough, how much data is in your NFS mounted directory? > > Are you using TCP or UDP for the NFS mount? > > > > Any idea what type of network activity is happening just after > > /etc/rc.d/netif start (DHCP, NTP, anything else)? > > > > Dave > > One other thing, are you using jumbo frames? What's your MTU setting? > And one more thing. I've been passing line rate traffic for a few hours (with both netperf running the tcp_stream_script and a constant stream of UDP traffic to the discard server on the FreeBSD system) and I haven't seen a hiccup yet on the tip of RELENG_6 with jumbo frames enabled on my Dell PE2950 (one dual-core CPU, 4GB RAM). Of course I don't have any real services running on it (load average is 0.19). Any unusual settings I should be aware of? Does anyone know a simple way to drive up CPU utilization and consume large amounts of memory to try and simulate a heavily loaded system? Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Problems with BCE network adapter (Dell PE2950)
> Sorry for the top post, please try following patch: > http://people.freebsd.org/~sephe/if_bce.c.diff > > This is probably the cause; I noticed it when bce(4) was > ported to DragonFly. > Thanks Sephe, I think you're on to something. I have some debug code in the driver to simulate mbuf allocation failures and when I enable that I start receiving the same error messages Tom reported (along with various kernel panics), but when I include your change the system seems to keep humming along. I'll certainly add your code into an update shortly. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Question about bce driver
> Firstly I am not trying to slate David's efforts, however > there are and > have been some big problems with the driver. > > As for the IPMI integration, our management controllers are > set to used > the remote access controller NIC (Set as dedicated in the > bios) rather > than share the host nics. > > Unfortunately I can not justify spending the time on the driver to > management when the problem can be solved by spending a > relatively small > amount of money on hardware to fix the problem. > I'm sorry I haven't been able to get to the root of your problems Tom. My biggest challenge is trying to duplicate the a loaded system which seems to be an important component of the failure you're seeing. Simply blasting traffic with netperf hasn't been much of a challenge as it runs over the weekend at line rate without a hiccup. If anyone has an idea on how to load the system (which means regular failures by the driver to allocate mbufs) I'd definitely appreciate hearing it so that I can duplicate Tom's environment. It's certainly important to me that the driver perform well and I won't stop looking at the problem if you do decide to jump ship. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: Question about bce driver
> Dave, perhaps you could answer my question? > > I'm using the 0.9.6 driver with 6.1 RELEASE. I don't use > jumbo frames, so > the only problem I've experienced is the link state going up and down > occasionally. > > Does the more current driver solve the link state problem? > If it does, can > I use it on 6.1 merely by replacing the three files included > with it? Or > are there includes that refer to other libraries that have > changed since > 6.1? > > Am I better off upgrading the entire system to 6.2? (I > prefer not to do > this, because I have to take the site down to do so. > Patching files and > rebuilding world and kernel only takes the box out of service > during the > reboots. I've never seen the link up/down problem myself, so I don't know if any changes I've made have an effect on it. You would need to change more than the bce driver since it depends upon the brgphy driver under sys/dev/mii which would also need to change. There may be other dependencies upon changes in the 6.2 kernel which would prevent a quick/easy port without some debugging. Dave ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
