Re: 10GE router resource
On Thu, Mar 27, 2008, Andrew C Burnette wrote: Indeed. PCI-X is already an EOL'ed interface, if only cheap PCI-X cards were available. Once you add extensive ACL's, there's loads more [central] processing to be done than just packet routing (100k choices versus 2 to 4 interfaces). System throughput gets slammed rather quickly. Linux IPtables grumbles painfully at 100k line ACLs :) Not to mention the options of what to do with a packet are very limited. I agree, and the rest of the discussion is interesting, but the iptables deployments I've seen which do massive ACLs like this almost certainly end up having ACLs you can collapse into a small number of set-lookup-and-act rules. Those set-lookup-and-act rules are much faster than the linear ACL lookups which ipfw/iptables/ipf/pf/etc do by default (and all of them support IP sets in some form or other); I did this trick recently to reduce the CPU overhead on an old revision 2.8ghz P4 from 99% to 10% when routing 100mbit of average-pps TCP. Adrian
RE: 10GE router resource
At 09:59 AM 3/26/2008, you wrote: Is there a multiport card out there on to which some of the forwarding responsibilities can be offloaded? Perhaps the CPU doesn't need to see every packet that arrives on the machine. Am I the only person who has heard of Google? It didn't take me long to find this wiki page http://www.bro-ids.org/wiki/index.php/ClusterFrontends for an Opensource Intrusion Detection System that lists various 10G cards for Linux and a couple of FPGA cards so that you can roll your own ASICs. Anyway, this one http://www.lewiz.com/talon3220.html has two ports and claims to reach 8.8 Gbps with 1500 byte packets. People rolling their own router are not the only ones who want to do 10G on Linux. Anyone who wants to roll your own more advanced apps on Linux without reinventing the wheel may want to check out my friend's company: http://www.bivio.net/products/bivio7000.htm Even with their specialized hardware platform, bus, and extensive tuning, they only get 10Gb/s throughput on the dual or quad 10G modules. However you can do 100,000 line ACLs at that speed. It is built for a different application than core routing. However, an XMR or Sup720 will still be a lot cheaper and give better performance. -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin
Re: 10GE router resource
[EMAIL PROTECTED] writes: People rolling their own router are not the only ones who want to do 10G on Linux. speaking of which, has anybody run xorp in production? it looks as much like JunOS as quagga/zebra looks like IOS. if click works on current hardware and if the xorp/click integration is good, this could be a great science fair project for smaller network operators who need big PPS. -- Paul Vixie
Re: 10GE router resource
On Wednesday 26 March 2008, Robert Boyle wrote: Even with their specialized hardware platform, bus, and extensive tuning, they only get 10Gb/s throughput on the dual or quad 10G modules. However you can do 100,000 line ACLs at that speed. It is built for a different application than core routing. However, an XMR or Sup720 will still be a lot cheaper... The chassis and switch fabric would generally be affordable (it'd normally be a bundle). It's the cost of the 10-Gig-E line cards that is the enemy. Mark. signature.asc Description: This is a digitally signed message part.
Re: 10GE router resource
Paul Vixie wrote: [EMAIL PROTECTED] writes: People rolling their own router are not the only ones who want to do 10G on Linux. speaking of which, has anybody run xorp in production? it looks as much like JunOS as quagga/zebra looks like IOS. if click works on current hardware and if the xorp/click integration is good, this could be a great science fair project for smaller network operators who need big PPS. Vyatta is built on top of xorp. You can download the bootable iso from their site and take a low-commitment look: http://www.vyatta.com/download/index.php --Peter
Re: 10GE router resource
Actually the latest version of Vyatta uses Quagga. If anyone is interested in discussing the differences in running the two in production networks feel free to contact me off list. In full disclosure, I work for Vyatta. Cheers, Robert. Peter Wohlers wrote: Vyatta is built on top of xorp. You can download the bootable iso from their site and take a low-commitment look: http://www.vyatta.com/download/index.php --Peter
Re: 10GE router resource
Actually, soon this will no longer be true. Vyatta's new platform, Glendale, will be moving to Quagga. Quagga is much more stable, and slow-moving compared to Xorp, which makes me slightly more comfortable (less breakage between versions). There are some major features lacking inside of the platform. For example, it lacks the ability to do BFD, BGP over IPSec, Multicast, etc... This major lack of features makes this a hard to deploy piece of software. I am sure with enough customers Vyatta would be able to catch up to Cisco. Also, from a viewpoint of hardware, x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with a switch) 1 GigE ports in it. Though, the way that Linux works, it cannot handle high packet rates. If you are planning on handling large flows with mostly large packets, you are alright for the most part. Just be warned. Peter Wohlers wrote: Paul Vixie wrote: [EMAIL PROTECTED] writes: People rolling their own router are not the only ones who want to do 10G on Linux. speaking of which, has anybody run xorp in production? it looks as much like JunOS as quagga/zebra looks like IOS. if click works on current hardware and if the xorp/click integration is good, this could be a great science fair project for smaller network operators who need big PPS. Vyatta is built on top of xorp. You can download the bootable iso from their site and take a low-commitment look: http://www.vyatta.com/download/index.php --Peter -- +1.925.202.9485 Sargun Dhillon deCarta [EMAIL PROTECTED] www.decarta.com
Re: 10GE router resource
On Wed, Mar 26, 2008 at 4:26 PM, Sargun Dhillon [EMAIL PROTECTED] wrote: from a viewpoint of hardware, x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with a switch) 1 GigE ports in it. Though, the way that Linux works, it cannot handle high packet rates. Correction: The way DRAM works, it cannot handle high packet rates. Also note that the PCI-X bus tops out in the 7 to 8 gbps range and it's half-duplex. High-rate routers try to keep the packets in an SRAM queue and instead of looking up destinations in a DRAM-based radix tree, they use a special memory device called a TCAM. http://www.pagiamtzis.com/cam/camintro.html Regards. Bill Herrin -- William D. Herrin [EMAIL PROTECTED] [EMAIL PROTECTED] 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: 10GE router resource
I wonder how difficult it would be to integrate such a device on to an x86 board cheaply. Something like NetFPGA (http://netfpga.org/) would be an interesting place to start. The board has on board SRAM, a bit of DRAM, an FPGA, and 2 GigE interfaces. I know it definitely isn't normal for Network Operators to fund research like this, but it would still be fairly interesting if there was an Open Router Consortium (something for Vyatta to start?) with hardware acceleration to X86 routers. Possibly even making Quagga a mainstream control plane. Right now Quagga is controlled by a few engineers from Sun. This nearly produces a conflict on interest (Sun used to have their own routing platform). Anyways, to end my rambling... As network operators would you finance a low, medium end router with decent ROI. The question for developers (Vyatta primarily), could you do what Digium did for Asterisk--become business front, and provide platforms for Asterisk deployment in the enterprise--for Quagga, Linux, etc? William Herrin wrote: On Wed, Mar 26, 2008 at 4:26 PM, Sargun Dhillon [EMAIL PROTECTED] wrote: from a viewpoint of hardware, x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with a switch) 1 GigE ports in it. Though, the way that Linux works, it cannot handle high packet rates. Correction: The way DRAM works, it cannot handle high packet rates. Also note that the PCI-X bus tops out in the 7 to 8 gbps range and it's half-duplex. High-rate routers try to keep the packets in an SRAM queue and instead of looking up destinations in a DRAM-based radix tree, they use a special memory device called a TCAM. http://www.pagiamtzis.com/cam/camintro.html Regards. Bill Herrin -- +1.925.202.9485 Sargun Dhillon deCarta [EMAIL PROTECTED] www.decarta.com
Re: 10GE router resource
On Wed, Mar 26, 2008 at 6:54 PM, Sargun Dhillon [EMAIL PROTECTED] wrote: I wonder how difficult it would be to integrate such a device on to an x86 board cheaply. Something like NetFPGA (http://netfpga.org/) would be an interesting place to start. The board has on board SRAM, a bit of DRAM, an FPGA, and 2 GigE interfaces. Hi Sargun, SRAM != TCAM. With SRAM you can only access one word per cycle. The coolness of the TCAM is that the entire memory is queried in one cycle, spitting out the best match. Nevertheless, there is some interesting hardware out there. The Endace DAG card with the coprocessor has a TCAM on it, but it's not big enough to handle a full BGP table. Regards, Bill Herrin -- William D. Herrin [EMAIL PROTECTED] [EMAIL PROTECTED] 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
RE: 10GE router resource
High-rate routers try to keep the packets in an SRAM queue and instead of looking up destinations in a DRAM-based radix tree, they use a special memory device called a TCAM. FPGAs can be used to do both SRAM and TCAMs. All that is needed is an FPGA board with 10G or a 10G card with an FPGA on it. Although NetFPGA and RiceNIC are both 1G devices, there is a certain commercial market for programmable high-speed network cards for things like Intrusion Detection and data-center/GRID type applications. Anyone seriously interested in this area should start hunting amongst the developers (and researchers) of embedded systems. You might end up working with a university student in the Czech Republic to put his TCAM/FPGA implementation onto a 10G card because the Internet breaks down the barriers that high-margin vendors have used to create lock-in. Bleeding edge networks may not be able to do this type of deal but then, they are only 1% or less of the network operators out there. --Michael Dillon
RE: 10GE router resource
FPGAs can be used to do both SRAM and TCAMs. All that is needed is an FPGA board with 10G or a 10G card with an FPGA on it. The Xilinx Virtex family can already do 10G, if you are into FPGA development (I seem to recall the first Xilinx FPGA that could do 10G was 4-5 years ago; forever in Moore's law). Other vendors have equivalent parts. And the Xilinx family has an available PowerPC core. I seem to recall a couple of vendors making available a (micro)Linux kernel for running on same. All the hardware you need for building your own 10G router. Just add FPGA development resources, some planar board design, and software.
Re: 10GE router resource
William Herrin wrote: On Wed, Mar 26, 2008 at 4:26 PM, Sargun Dhillon [EMAIL PROTECTED] wrote: from a viewpoint of hardware, x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with a switch) 1 GigE ports in it. Though, the way that Linux works, it cannot handle high packet rates. Correction: The way DRAM works, it cannot handle high packet rates. Also note that the PCI-X bus tops out in the 7 to 8 gbps range and it's half-duplex. Indeed. PCI-X is already an EOL'ed interface, if only cheap PCI-X cards were available. Once you add extensive ACL's, there's loads more [central] processing to be done than just packet routing (100k choices versus 2 to 4 interfaces). System throughput gets slammed rather quickly. Linux IPtables grumbles painfully at 100k line ACLs :) Not to mention the options of what to do with a packet are very limited. The AMD chips with extra L1 cache perform better on *bsd platforms as the forwarding code is tight and likes to stay close to the CPU, and context switching kills packet processing performance (thus the small but notable increase in the multicore performance). The GP registers on the AMD platform are also easy to deal with (and in 64 bit mode, you get double the number for free) essentially working an end around a broken stack architecture from a few decades agoanyone recall the simplicity of assembly language of the 6800 or the 6502? :-) getting the latency down low enough for HPC clusters is a major hassle, as the x86 PC design just doesn't have the bandwidth. Of course, Intel makes some slick NPU's for custom work (e.g. cloudshield.com). If you like starting at bit 0. (isn't that like slot zero or port zero, it technically doesn't exist since zero is only a placeholder in larger numbers if you mean anything greater than none? I could swear back in the days of a SLC96, ports were 1-96, not 0-95 :-) ) http://developer.intel.com/design/network/products/npfamily/index.htm?iid=ncdcnav2+proc_netproc too bad they [Intel] don't make a hypertransport capable version, or you'd have one helluva multicore multiNPU system with no glue logic required. Fun to play around though. regards, andy High-rate routers try to keep the packets in an SRAM queue and instead of looking up destinations in a DRAM-based radix tree, they use a special memory device called a TCAM. http://www.pagiamtzis.com/cam/camintro.html Regards. Bill Herrin
Re: 10GE router resource
At 09:44 PM 3/25/2008, you wrote: On Tue, Mar 25, 2008 at 1:59 PM, Chris Grundemann [EMAIL PROTECTED] wrote: Greg has laid out a great bit of information and I would like to add just one possibility to the list of budget 10GE routers: Vyatta. According to a recent press release from that company (http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product that is 2 to 3X higher performance at a cost savings of more than 75 percent when compared to Cisco's 7200. Unfortunately I have not had the when did the 7200 go 10ge? Shh... It's a secret and hasn't been released yet. We have have a few NPE-40Gs with four 10G XFP interfaces. ;) Nah... I'm just wishing... -Robert Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin
Re: 10GE router resource
On Monday 24 March 2008, user user wrote: Hi everybody! Hello. Also I'd love to hear recommendatios for budget 10GE routers. The budget router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces. Today, from Cisco, the smallest router you'll get a 10Gbps Ethernet port on is the Cisco ASR1000 series. Mind you, though, FCS for this box isn't until about May. Also, this box is oversubscribed as the current switch fabric is 10Gbps. From Juniper, the smallest M-series box you'll get the same port on is the M120 platform. You could also look at smaller switches from both vendors, but if you plan on taking full BGP feeds from your upstream providers, this might be an issue. Cheers, Mark. signature.asc Description: This is a digitally signed message part.
Re: 10GE router resource
From: Mark Tinka [EMAIL PROTECTED] Date: Mon, 24 Mar 2008 19:12:57 +0800 Sender: [EMAIL PROTECTED] On Monday 24 March 2008, user user wrote: Hi everybody! Hello. Also I'd love to hear recommendatios for budget 10GE routers. The budget router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces. Today, from Cisco, the smallest router you'll get a 10Gbps Ethernet port on is the Cisco ASR1000 series. Mind you, though, FCS for this box isn't until about May. Also, this box is oversubscribed as the current switch fabric is 10Gbps. From Juniper, the smallest M-series box you'll get the same port on is the M120 platform. You could also look at smaller switches from both vendors, but if you plan on taking full BGP feeds from your upstream providers, this might be an issue. Depending on how the box will be used, Foundry is probably the cheapest, followed by Force10. Since yo will be connecting to two transit providers, you probably need the full routing table, but if you don't need full routes, the new Juniper EX8200 looks like an option. It is limited to about 12K routes in the FIB. It's not shipping at this time and I don't know when FSR is scheduled. Note that F10 does not do MPLS and neither F10 or Foundry has the software stability of either C or J, so you will need to look closely at exactly the features needed. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 pgp1RIp462wmu.pgp Description: PGP signature
Re: 10GE router resource
Also I'd love to hear recommendatios for budget 10GE routers. The budget router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces. If you don't need BGP-ish power, David Newman just published his test of 10GigE switches today in Network World. He was focusing mostly on switching in the enterprise, but he has a variety of other performance metrics and results which may be helpful: http://www.networkworld.com/reviews/2008/032408-switch-test.html?t51hb jms -- Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719 Senior Partner, Opus One Phone: +1 520 324 0494 [EMAIL PROTECTED]http://www.opus1.com/jms
Re: 10GE router resource
Joel Snyder wrote: Also I'd love to hear recommendatios for budget 10GE routers. The budget router would be used to hook up client networks through one 10GE interface and connect to different transit providers through two 10GE interfaces. If you don't need BGP-ish power, David Newman just published his test of 10GigE switches today in Network World. He was focusing mostly on switching in the enterprise, but he has a variety of other performance metrics and results which may be helpful: http://www.networkworld.com/reviews/2008/032408-switch-test.html?t51hb The author's specifications eliminated Cisco's 4900M from the competition. That not unexpected though since it was a evaluation of access switches w/ 10G uplinks. The 4900M has 8 on-board 10G interfaces and expansion modules that can carry 8 more (not oversubscribed) or 16 (oversubscribed). It has has GigE support via TwinGig modules in the expansion module bays. It also has a 320Gbps backplane and can handle up to 200k v4 routes. It's an impressive little switch if you need 10G aggregation. It can't handle a full table of course but it still has a lot of use. No MPLS options. It's based on the 4500's Sup 6-E. http://www.cisco.com/en/US/products/ps9310/index.html The base unit starts at $16k. Justin
