Re: gigabit NIC of choice?
On Fri, 6 Sep 2002, Terry Lambert wrote: Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). Is anyone still making cards with the Tigon II chipset? I'm not finding them for sale anywhere... Thanks, -Dan To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Dan Ellard wrote: Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). Is anyone still making cards with the Tigon II chipset? I'm not finding them for sale anywhere... See Ken's post. You will basically have to buy them from stock from someone who has them on a shelf somewhere, or you will have to sign NDA and recreate the firmware work on another card. The Tigon III's are *significantly* cheaper, and don't have the firmware-download-each-time-the-IP-changes that the Tigon II driver has, so most people have switches to the Tigon III. I guess the next question is Anyone know a gigabit NIC that is currently in production, which has hack-friendly firmware?... -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Terry Lambert writes: I guess the next question is Anyone know a gigabit NIC that is currently in production, which has hack-friendly firmware?... I think our products are the only game in town. http://www.myri.com/myrinet/product_list.html http://www.myri.com/myrinet/performance/index.html Yes, they are a little pricy, but quite hackable. And the link speed is twice gig ethers's (ie, 2Gb/sec full duplex, rather than 1Gb/sec full duplex). Sorry for the shameless plug ;) Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Andrew Gallatin wrote: I guess the next question is Anyone know a gigabit NIC that is currently in production, which has hack-friendly firmware?... I think our products are the only game in town. http://www.myri.com/myrinet/product_list.html http://www.myri.com/myrinet/performance/index.html Yes, they are a little pricy, but quite hackable. And the link speed is twice gig ethers's (ie, 2Gb/sec full duplex, rather than 1Gb/sec full duplex). Sorry for the shameless plug ;) I'm a bit confused about these cards. Are they Gigabit ethernet cards, or are they 2 gigabit ethernet cards, which can only talk to other Myrinet cards, like ARCNet is not the same thing as ethernet? Are the FDX through a Cisco or Extreme Networks Gigabit switch, getting 2Gbit, and you are just defining 2G as total over the wire transfer rate in *both* directions, requiring that data go both ways (i.e. 2G is an aggregate number, but it's still standards compliant Gigabit)??? -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
On Tue, 10 Sep 2002, Terry Lambert wrote: Andrew Gallatin wrote: I think our products are the only game in town. http://www.myri.com/myrinet/product_list.html http://www.myri.com/myrinet/performance/index.html I'm a bit confused about these cards. Terry, put down the pipe and visit the URLs. ;-) They're not Ethernet at all. They're Myrinet. They're 2 GB/s full duplex. Saying that they are 4GB/s would be misleading, but they are most definitely capable of 2GB/s+2GB/s if your bus can pump that much data. Myrinet is currently only useful in specialized applications because it can't be bridged onto a standard Ethernet network without running it through a computer (AFAIK). There are rumors afloat of Gigabit Ethernet linecards for Myrinet switch hardware on the horizon though. The technology is great and the folks at myri.com are some of the most competent and helpful support staff I've ever had the pleasure of dealing with. Oh yeah, and it's f'ing fast. Brandon D. Valentine -- http://www.geekpunk.net [EMAIL PROTECTED] ++[++-][++-].[+-][+-]+.+++..++ +.+[++-]++.+++..+++.--..+. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Brandon D. Valentine writes: running it through a computer (AFAIK). There are rumors afloat of Gigabit Ethernet linecards for Myrinet switch hardware on the horizon Slightly more than rumours -- http://www.myri.com/news/02512/slides/Seitz_roadmap.pdf http://www.myri.com/news/02512/slides/Seizovic_lanai.pdf Cheers, Drew To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
On Fri, Sep 06, 2002 at 11:59:05AM -0400, Dan Ellard wrote: ... What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) I'm looking for: - Easy FreeBSD integration. - Reliability. - High performance. i have had good results with the single-port Intel em cards They are reasonably priced too, at least the copper version. cheers luigi To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
On Fri, 6 Sep 2002, Luigi Rizzo wrote: i have had good results with the single-port Intel em cards They are reasonably priced too, at least the copper version. Thanks for the note. (and thanks for reminding me that I meant to ask about copper! I hope to never deal with fiber again...) -Dan To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) I'm looking for: - Easy FreeBSD integration. - Reliability. - High performance. The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). I recall from a while back that gigabit cards have relatively large caches on them, correct? How does the size of the cache impact performance, and what is considered a sufficient cache size? To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Darren Pilgrim wrote: Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). I recall from a while back that gigabit cards have relatively large caches on them, correct? How does the size of the cache impact performance, and what is considered a sufficient cache size? The best advice I have for you is to read the source code for the drivers, specifically any commentary by Bill Paul up top; he tells it like it is, with regard to the hardware. In general, cards with DMA engines that require better than two byte alignment require that the mbufs be copied again for transmit. Also, in general, the more queue descriptors, the better, since they limit the number of packets pending input or output that you can have outstanding simultaneously. Controllers that can't do scatter/gather are also problematic, because they mean you have to allocate a seperate buffer area out of memory and copy outbound data into thue buffer instead of scattering, and copy from the buffer to mbufs on the receive (gather). The smaller the amount of memory on the card, the worse things are, as well, because it limits the amount of data you can have outstanding, as well, which limits your throuput. Bad cards are also not capable of software interrupt coelescing (this was one of my contributions). Basically, what this means is that a card will not DMA, or does not have a modified register, or does not update it, while an interuppt is being processed (e.g. after the interrupt is raised in hardware, and has not yet been ACKed). The effect of this is that you can't poll at the end of the interrupt handler for new data, only exitting the handler when there is no new data to process (10 to 15% performance inmprovement, by my benchmarks). Bad cards will also have smaller on-chip buffers (as opposed to on-card buffers). For example, there are a number of cards that supposedly support both jumbograms and TCP checksum offloading, but have only 8K of space. A jumbogram is 9K, so when using jumbograms, it's impossible to offload checksums to the hardware. There are cards that supposedly support checksumming, but use the buggy incremental checksum update algorithm (two's complement vs. one's complement arithmatic), and will screw up the TCP checksum, yielding 0xfffe instead of 0x after summing, because they don't correctly handle negative zero (there is an RFC update on this). A really good card will allow you to align card buffers to host page boundaries, which can dignificantly speed up I/O. This is what I was referring to when I said there was a rewritten firmware for the Tigon II. The manufacturer won't reall share sufficient information for this interface to be implemented on the Tigon III. Basically, it eliminates another copy. The absolute worst one (according to Bill Paul) is the RealTek 8129/8139. See the comments in /usr/src/sys/pci/if_rl.c. Mostly, if you go by the comments in the drivers, you'll get a feel for what's done right and what's done wrong from a host interface perspective by the card manufacturer. As to your cache question... the size of the cache is the pool size. If you look at this as a queueing theory problem, then amount of buffer space translates directly into how much it's willing to tolerate delays in servicing interrupts -- pool retention time. Above a certain size, and it really won't effect your ability to shove data through it because there will be more and more free space available. Unless you are going card-to-card (unlikely; most firmware doesn't support the necessary ability to do incremental header rewriting, and flow monitoring, so that you can mark flows without in-band data that needs to be rewritten e.g. text IP addresses in FTP port commands, etc.), you will always end up with a certain amount of buffer space free, because the limiting factor is going to be your ability to shovel data over the PCI bus from the disk to main memory and back over the same bus to the network card. So my flip answer seems flip, but to get the best overall performance, you should use a Tigon II with the FreeBSD specific firmware, and the zero copy TCP patches that need the firmware patches. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Terry Lambert wrote: The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). References: http://people.freebsd.org/~ken/zero_copy/ -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
Thank you! It was fun to watch questions come up and get shot down while reading the same email. Now if you'll excuse me, I need to go use the source, Terry. Terry Lambert wrote: Darren Pilgrim wrote: Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). I recall from a while back that gigabit cards have relatively large caches on them, correct? How does the size of the cache impact performance, and what is considered a sufficient cache size? The best advice I have for you is to read the source code for the drivers, specifically any commentary by Bill Paul up top; he tells it like it is, with regard to the hardware. long explaination snipped To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: gigabit NIC of choice?
On Fri, Sep 06, 2002 at 10:32:12 -0700, Terry Lambert wrote: Dan Ellard wrote: What's the gigabit ethernet NIC of choice these days? (I've had good experiences with the NetGear G620T, but apparently this card is no longer being sold.) I'm looking for: - Easy FreeBSD integration. - Reliability. - High performance. The Tigon II has the best performances, but that's because software people rewrote the firmware, instead of hardware engineers moonlighting as programmers. 8-) 8-). You'll get good performance with the Tigon II with jumbo frames under -current with ZERO_COPY_SOCKETS and TI_JUMBO_HDRSPLIT turned on. Note that good performance == lower CPU utilization here, although it is difficult to see any improvement in -current with SMP enabled, and the improvement isn't as large in UP mode as it used to be. You can easily get wire rates with jumbo frames with a Tigon II without zero copy, given a reasonably fast machine. (At least on -stable. Reasonably fast == 1GHz Pentium III, 64 bit PCI.) With 1500 byte frames, though, the Tigon II won't perform as well as some other controllers. For most folks, performance with 1500 byte frames is what matters, since you usually need a jumbo-capable gigabit switch to take advantage of jumbo frames in anything more than a point to point environment. The modifications I made to the Tigon firmware (the ones that are in FreeBSD) are actually relatively minor. The main trick is to make sure that the header is in its own scatter/gather element, so the payload will be page aligned. (Assuming the second and subsequent scatter/gather elements are page aligned.) So if your workload consists of jumbo frames for the most part, and receive performance is important, I would suggest a Tigon II-based board, like the GA620T, if you can find one. (Obviously that's getting more and more difficult nowadays.) Otherwise, I would suggest another card with better performance with 1500 byte frames. (I haven't done any tests with other boards, so I can't make specific recommendations.) Ken -- Kenneth Merry [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
