Re: bge problems (was: gigabit NIC of choice?)
While we're nitpicking: On Wed, 11 Sep 2002, John Polstra wrote: All of the documentation and errata for the BCM570x chips are protected by NDA, just like every other gigabit MAC in current production. Through the graciousness of Intel we also have a contributed driver for their gig-e cards (em). Thanks to all who lined that up, it works great. But broadcom does take the cake for being anal about NDAs. -- Doug White| FreeBSD: The Power to Serve [EMAIL PROTECTED] | www.FreeBSD.org To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: bge problems (was: gigabit NIC of choice?)
In article [EMAIL PROTECTED], Doug White [EMAIL PROTECTED] wrote: While we're nitpicking: On Wed, 11 Sep 2002, John Polstra wrote: All of the documentation and errata for the BCM570x chips are protected by NDA, just like every other gigabit MAC in current production. Through the graciousness of Intel we also have a contributed driver for their gig-e cards (em). Thanks to all who lined that up, it works great. What is it you are nitpicking about? My posting you quoted mentioned the em driver as the only game in town if you want a supported driver. I too am grateful to Intel for providing this and for keeping it up to date. It's more than the other vendors do for us. However, just like almost all the other gigabit chips, if you want to get docs or errata on the Intel devices you are going to have to sign an NDA. That's true even for their 10/100 chips. But broadcom does take the cake for being anal about NDAs. Intel is pretty anal about them too. :-( John -- John Polstra John D. Polstra Co., Inc.Seattle, Washington USA Disappointment is a good sign of basic intelligence. -- Chögyam Trungpa To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: bge problems (was: gigabit NIC of choice?)
In article p05111a10b9a45ebf5047@[10.1.1.73], Mark Peek [EMAIL PROTECTED] wrote: At 12:51 PM +0200 9/10/02, Birger Toedtmann wrote: So it seems there is some problem with the bge driver/card/64bit bus. Does the scenario ring a bell to someone of you? Yes, the bge driver in 4.6 is broken. John Polstra put fixes into -stable which will show up in 4.7. I doubt that those fixes will solve Birger's problem, unfortunately. John -- John Polstra John D. Polstra Co., Inc.Seattle, Washington USA Disappointment is a good sign of basic intelligence. -- Chögyam Trungpa To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: bge problems (was: gigabit NIC of choice?)
In article [EMAIL PROTECTED],
Birger Toedtmann [EMAIL PROTECTED] wrote:
just as this comes up, I have a question. Has anyone had experiences
with the bge driver and related cards? We had some weird problems here
lately:
- bought two 3com 996-SX (fiber) cards and plugged them into
ASUS A7M266-D boards with AMD Athlon 1.8GHz CPUs,
- brought up FBSD 4.6 and tried some networking
-- All packets (to and fro) that where bigger than total IP length
of 214 bytes were garbled. To be more specific, it seemed to
us that some 8 byte regions haven't been copied correctly from
the cards at some point when the packets were larger.
(The boards worked fine with normal 100Mbit cards, BTW.)
To make matters more interesting, we then put the (Gbit) cards into
AMD-Duron-700MHz-Systems with 32bit bus only - as apposed to the A7M266-D
which had a 64bit bus:
-- The cards worked just fine with all sorts (and sizes) of packets,
but performance was - surpise, surprise - limited to 0.25 Gbit.
So it seems there is some problem with the bge driver/card/64bit bus.
I use the 3c996B-T cards without any problems on a 64-bit bus. It's
an Intel-based machine (Dell 1550) rather than an Athlon, though.
It's possible that the problems you're seeing have something to do
with the chipset rather than with the width of the PCI bus.
All of the documentation and errata for the BCM570x chips are
protected by NDA, just like every other gigabit MAC in current
production. That makes it very difficult for us to maintain a working
driver. The only information we have is obtained by staring at
the Linux driver (which originates from Broadcom), making educated
guesses, and trying experiments. For example, in the Linux driver
obtained from the 3com web site, I see that there is a function
LM_DmaTest() which transfers a known data pattern using DMA and checks
for data corruption. This code might be present in the driver to
detect exactly the problem you've reported -- or maybe not. The only
way to find out is to try it.
It looks like if LM_DmaTest() finds corruption, it works around it by
putting a certain value into certain bits of a certain register.
You could try the equivalent in our driver and see if it works. In
if_bge.c, find the last occurrance of BGE_PCI_DMA_RW_CTL in
bge_chipinit():
#ifdef __brokenalpha__
/*
* Must insure that we do not cross an 8K (bytes) boundary
* for DMA reads. Our highest limit is 1K bytes. This is a
* restriction on some ALPHA platforms with early revision
* 21174 PCI chipsets, such as the AlphaPC 164lx
*/
PCI_SETBIT(sc, BGE_PCI_DMA_RW_CTL, BGE_PCI_READ_BNDRY_1024, 4);
#endif
After that, add this code (untested):
{ /* XXX experiment */
u_int32_t r;
r = CSR_READ_4(sc, BGE_PCI_DMA_RW_CTL);
r = (r ~BGE_PCIDMARWCTL_WRADDR_BNDRY) | BGE_PCI_WRITE_BNDRY_16BYTES;
CSR_WRITE_4(sc, BGE_PCI_DMA_RW_CTL, r);
}
Who knows? Maybe you'll get lucky. :-)
If you want a gigabit interface that is likely to keep working in
FreeBSD, your only option is to use the Intel chips and the em
driver. It's our only gigabit driver that's maintained by somebody
who has unrestricted access to the documentation and errata.
John
--
John Polstra
John D. Polstra Co., Inc.Seattle, Washington USA
Disappointment is a good sign of basic intelligence. -- Chögyam Trungpa
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-hackers in the body of the message
Re: bge problems (was: gigabit NIC of choice?)
John Polstra writes: | If you want a gigabit interface that is likely to keep working in | FreeBSD, your only option is to use the Intel chips and the em | driver. It's our only gigabit driver that's maintained by somebody | who has unrestricted access to the documentation and errata. Atleast one potential exception to this it that so far National seems to keep their programming on line without needed an NDA. I used that and peeked at the Netgear Linux driver on how to enable the fiber port. So now the driver supports copper and fiber cards. Doug A. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: bge problems (was: gigabit NIC of choice?)
In article [EMAIL PROTECTED], Doug Ambrisko [EMAIL PROTECTED] wrote: John Polstra writes: | If you want a gigabit interface that is likely to keep working in | FreeBSD, your only option is to use the Intel chips and the em | driver. It's our only gigabit driver that's maintained by somebody | who has unrestricted access to the documentation and errata. Atleast one potential exception to this it that so far National seems to keep their programming on line without needed an NDA. Whoops, thanks for the correction. That's good news. John -- John Polstra John D. Polstra Co., Inc.Seattle, Washington USA Disappointment is a good sign of basic intelligence. -- Chögyam Trungpa To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
bge problems (was: gigabit NIC of choice?)
Hi, What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) [ discussion followed ] just as this comes up, I have a question. Has anyone had experiences with the bge driver and related cards? We had some weird problems here lately: - bought two 3com 996-SX (fiber) cards and plugged them into ASUS A7M266-D boards with AMD Athlon 1.8GHz CPUs, - brought up FBSD 4.6 and tried some networking -- All packets (to and fro) that where bigger than total IP length of 214 bytes were garbled. To be more specific, it seemed to us that some 8 byte regions haven't been copied correctly from the cards at some point when the packets were larger. (The boards worked fine with normal 100Mbit cards, BTW.) To make matters more interesting, we then put the (Gbit) cards into AMD-Duron-700MHz-Systems with 32bit bus only - as apposed to the A7M266-D which had a 64bit bus: -- The cards worked just fine with all sorts (and sizes) of packets, but performance was - surpise, surprise - limited to 0.25 Gbit. So it seems there is some problem with the bge driver/card/64bit bus. Does the scenario ring a bell to someone of you? Regards, Birger To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
On Fri, 6 Sep 2002, Terry Lambert wrote: Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). Is anyone still making cards with the Tigon II chipset? I'm not finding them for sale anywhere... Thanks, -Dan To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Dan Ellard wrote: Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). Is anyone still making cards with the Tigon II chipset? I'm not finding them for sale anywhere... See Ken's post. You will basically have to buy them from stock from someone who has them on a shelf somewhere, or you will have to sign NDA and recreate the firmware work on another card. The Tigon III's are *significantly* cheaper, and don't have the firmware-download-each-time-the-IP-changes that the Tigon II driver has, so most people have switches to the Tigon III. I guess the next question is Anyone know a gigabit NIC that is currently in production, which has hack-friendly firmware?... -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Terry Lambert writes: I guess the next question is Anyone know a gigabit NIC that is currently in production, which has hack-friendly firmware?... I think our products are the only game in town. http://www.myri.com/myrinet/product_list.html http://www.myri.com/myrinet/performance/index.html Yes, they are a little pricy, but quite hackable. And the link speed is twice gig ethers's (ie, 2Gb/sec full duplex, rather than 1Gb/sec full duplex). Sorry for the shameless plug ;) Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Andrew Gallatin wrote: I guess the next question is Anyone know a gigabit NIC that is currently in production, which has hack-friendly firmware?... I think our products are the only game in town. http://www.myri.com/myrinet/product_list.html http://www.myri.com/myrinet/performance/index.html Yes, they are a little pricy, but quite hackable. And the link speed is twice gig ethers's (ie, 2Gb/sec full duplex, rather than 1Gb/sec full duplex). Sorry for the shameless plug ;) I'm a bit confused about these cards. Are they Gigabit ethernet cards, or are they 2 gigabit ethernet cards, which can only talk to other Myrinet cards, like ARCNet is not the same thing as ethernet? Are the FDX through a Cisco or Extreme Networks Gigabit switch, getting 2Gbit, and you are just defining 2G as total over the wire transfer rate in *both* directions, requiring that data go both ways (i.e. 2G is an aggregate number, but it's still standards compliant Gigabit)??? -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
On Tue, 10 Sep 2002, Terry Lambert wrote: Andrew Gallatin wrote: I think our products are the only game in town. http://www.myri.com/myrinet/product_list.html http://www.myri.com/myrinet/performance/index.html I'm a bit confused about these cards. Terry, put down the pipe and visit the URLs. ;-) They're not Ethernet at all. They're Myrinet. They're 2 GB/s full duplex. Saying that they are 4GB/s would be misleading, but they are most definitely capable of 2GB/s+2GB/s if your bus can pump that much data. Myrinet is currently only useful in specialized applications because it can't be bridged onto a standard Ethernet network without running it through a computer (AFAIK). There are rumors afloat of Gigabit Ethernet linecards for Myrinet switch hardware on the horizon though. The technology is great and the folks at myri.com are some of the most competent and helpful support staff I've ever had the pleasure of dealing with. Oh yeah, and it's f'ing fast. Brandon D. Valentine -- http://www.geekpunk.net [EMAIL PROTECTED] ++[++-][++-].[+-][+-]+.+++..++ +.+[++-]++.+++..+++.--..+. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Brandon D. Valentine writes: running it through a computer (AFAIK). There are rumors afloat of Gigabit Ethernet linecards for Myrinet switch hardware on the horizon Slightly more than rumours -- http://www.myri.com/news/02512/slides/Seitz_roadmap.pdf http://www.myri.com/news/02512/slides/Seizovic_lanai.pdf Cheers, Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: bge problems (was: gigabit NIC of choice?)
At 12:51 PM +0200 9/10/02, Birger Toedtmann wrote: So it seems there is some problem with the bge driver/card/64bit bus. Does the scenario ring a bell to someone of you? Yes, the bge driver in 4.6 is broken. John Polstra put fixes into -stable which will show up in 4.7. Mark To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
gigabit NIC of choice?
What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) I'm looking for: - Easy FreeBSD integration. - Reliability. - High performance. Thanks, -Dan To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
On Fri, Sep 06, 2002 at 11:59:05AM -0400, Dan Ellard wrote: ... What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) I'm looking for: - Easy FreeBSD integration. - Reliability. - High performance. i have had good results with the single-port Intel em cards They are reasonably priced too, at least the copper version. cheers luigi To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
On Fri, 6 Sep 2002, Luigi Rizzo wrote: i have had good results with the single-port Intel em cards They are reasonably priced too, at least the copper version. Thanks for the note. (and thanks for reminding me that I meant to ask about copper! I hope to never deal with fiber again...) -Dan To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) I'm looking for: - Easy FreeBSD integration. - Reliability. - High performance. The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). I recall from a while back that gigabit cards have relatively large caches on them, correct? How does the size of the cache impact performance, and what is considered a sufficient cache size? To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Darren Pilgrim wrote: Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). I recall from a while back that gigabit cards have relatively large caches on them, correct? How does the size of the cache impact performance, and what is considered a sufficient cache size? The best advice I have for you is to read the source code for the drivers, specifically any commentary by Bill Paul up top; he tells it like it is, with regard to the hardware. In general, cards with DMA engines that require better than two byte alignment require that the mbufs be copied again for transmit. Also, in general, the more queue descriptors, the better, since they limit the number of packets pending input or output that you can have outstanding simultaneously. Controllers that can't do scatter/gather are also problematic, because they mean you have to allocate a seperate buffer area out of memory and copy outbound data into thue buffer instead of scattering, and copy from the buffer to mbufs on the receive (gather). The smaller the amount of memory on the card, the worse things are, as well, because it limits the amount of data you can have outstanding, as well, which limits your throuput. Bad cards are also not capable of software interrupt coelescing (this was one of my contributions). Basically, what this means is that a card will not DMA, or does not have a modified register, or does not update it, while an interuppt is being processed (e.g. after the interrupt is raised in hardware, and has not yet been ACKed). The effect of this is that you can't poll at the end of the interrupt handler for new data, only exitting the handler when there is no new data to process (10 to 15% performance inmprovement, by my benchmarks). Bad cards will also have smaller on-chip buffers (as opposed to on-card buffers). For example, there are a number of cards that supposedly support both jumbograms and TCP checksum offloading, but have only 8K of space. A jumbogram is 9K, so when using jumbograms, it's impossible to offload checksums to the hardware. There are cards that supposedly support checksumming, but use the buggy incremental checksum update algorithm (two's complement vs. one's complement arithmatic), and will screw up the TCP checksum, yielding 0xfffe instead of 0x after summing, because they don't correctly handle negative zero (there is an RFC update on this). A really good card will allow you to align card buffers to host page boundaries, which can dignificantly speed up I/O. This is what I was referring to when I said there was a rewritten firmware for the Tigon II. The manufacturer won't reall share sufficient information for this interface to be implemented on the Tigon III. Basically, it eliminates another copy. The absolute worst one (according to Bill Paul) is the RealTek 8129/8139. See the comments in /usr/src/sys/pci/if_rl.c. Mostly, if you go by the comments in the drivers, you'll get a feel for what's done right and what's done wrong from a host interface perspective by the card manufacturer. As to your cache question... the size of the cache is the pool size. If you look at this as a queueing theory problem, then amount of buffer space translates directly into how much it's willing to tolerate delays in servicing interrupts -- pool retention time. Above a certain size, and it really won't effect your ability to shove data through it because there will be more and more free space available. Unless you are going card-to-card (unlikely; most firmware doesn't support the necessary ability to do incremental header rewriting, and flow monitoring, so that you can mark flows without in-band data that needs to be rewritten e.g. text IP addresses in FTP port commands, etc.), you will always end up with a certain amount of buffer space free, because the limiting factor is going to be your ability to shovel data over the PCI bus from the disk to main memory and back over the same bus to the network card. So my flip answer seems flip, but to get the best overall performance, you should use a Tigon II with the FreeBSD specific firmware, and the zero copy TCP patches that need the firmware patches. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Terry Lambert wrote: The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). References: http://people.freebsd.org/~ken/zero_copy/ -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Thank you! It was fun to watch questions come up and get shot down while reading the same email. Now if you'll excuse me, I need to go use the source, Terry. Terry Lambert wrote: Darren Pilgrim wrote: Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). I recall from a while back that gigabit cards have relatively large caches on them, correct? How does the size of the cache impact performance, and what is considered a sufficient cache size? The best advice I have for you is to read the source code for the drivers, specifically any commentary by Bill Paul up top; he tells it like it is, with regard to the hardware. long explaination snipped To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
On Fri, Sep 06, 2002 at 10:32:12 -0700, Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) I'm looking for: - Easy FreeBSD integration. - Reliability. - High performance. The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). You'll get good performance with the Tigon II with jumbo frames under -current with ZERO_COPY_SOCKETS and TI_JUMBO_HDRSPLIT turned on. Note that good performance == lower CPU utilization here, although it is difficult to see any improvement in -current with SMP enabled, and the improvement isn't as large in UP mode as it used to be. You can easily get wire rates with jumbo frames with a Tigon II without zero copy, given a reasonably fast machine. (At least on -stable. Reasonably fast == 1GHz Pentium III, 64 bit PCI.) With 1500 byte frames, though, the Tigon II won't perform as well as some other controllers. For most folks, performance with 1500 byte frames is what matters, since you usually need a jumbo-capable gigabit switch to take advantage of jumbo frames in anything more than a point to point environment. The modifications I made to the Tigon firmware (the ones that are in FreeBSD) are actually relatively minor. The main trick is to make sure that the header is in its own scatter/gather element, so the payload will be page aligned. (Assuming the second and subsequent scatter/gather elements are page aligned.) So if your workload consists of jumbo frames for the most part, and receive performance is important, I would suggest a Tigon II-based board, like the GA620T, if you can find one. (Obviously that's getting more and more difficult nowadays.) Otherwise, I would suggest another card with better performance with 1500 byte frames. (I haven't done any tests with other boards, so I can't make specific recommendations.) Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
