RE: Software router

[Dennis Burgess]

RouterOS does run in virtual environments, super small, and has BGP,
OSPF, firewalling, etc., all built right in.  

---
Dennis Burgess, CCNA, Mikrotik Certified Trainer, MTCNA, MTCRE, MTCWE,
MTCTCE, MTCUME 
Link Technologies, Inc -- Mikrotik & WISP Support Services
Office: 314-735-0270 Website: http://www.linktechs.net
LIVE On-Line Mikrotik Training - Author of "Learn RouterOS"


-Original Message-
From: Jeremy Parr [mailto:jeremyp...@gmail.com] 
Sent: Tuesday, June 01, 2010 4:14 PM
To: Andrey Khomyakov; nanog@nanog.org
Subject: Re: Software router

On 1 June 2010 16:50, Andrey Khomyakov 
wrote:

> Good times!
>
> We are starting to play around with VMware SRM and they "virtual" 
> subnets that supposedly have to be able migrate from site to site in 
> case of a failure of the local hardware (or software).
> Seems like to do that I'd have to run a software router on a VM that 
> would redistribute the "virtual" subnet into the physical routing
domain.
> does any one have any suggestions for a software router?
>
> I'm running EIGRP on the net, so I guess nothing will speak that, so 
> I'd have to redistribute OSPF. Any OSPF software router software 
> suggestion would be much appreciated.
>
> Or if anyone had implemented "floating" subnets, any other suggestions

> or what to look out for would be also much appreciated.
>
> Thank all in advance,
>

Mikrotik would fit the bill.

Re: Software router

[James Hess]

On Tue, Jun 1, 2010 at 3:50 PM, Andrey Khomyakov
 wrote:
>Seems like to do that I'd have to run a software router on a VM that would
[snip]
For a VM router  (for performance reasons is different than what i'd
suggest for a generic software router), I would suggest picking an
off-the-shelf OS that Vmxnet2 or Vmxnet3  drivers are available for,
see KB1001805, make sure to install the VM tools, change vNICs'  type
to vmx.Standard OS + quagga, openbgpd, or other.Vyatta should
be great, if you are able to compile the vmx drivers for it.

Hopefully you are not planning to forward high-PPS traffic through a
single VM;  vNICs are potentially a serious bottleneck in that
scenario.

 If traffic is not trivial,   I would suggest using third-party
next-hop routing,  that is, with VM-based routers  removed from
forwarding path,  by acting as route server, or announcing as next-hop
another (real)  third-party router's   IP  instead one of its own IPs
(requiring all 3 routers to share a subnet).

Or investigate layer 2 extension of an upstream subnet via  L2TPv3
pseudo-wire service,  or Cisco OTV, etc
then design failover scenario to not require a VM involvement.

Another thought is   OSPF /32  host advertisements on  some 'beacon'
VM(s),  with tracked routes for  'virtual subnet' selection, instead
of a "router" VM.

Those are some vague thoughts...   I'm just saying, almost anything,
other than having a VM forward packets for subnets, if it is
avoidable,  even  tunnelling -- on a non-VM router...:)

--
-J

Re: Software router

[Ernest McCaleb]

I second Vyatta.  I've played with it quite a bit and found it to be
extremely functional.

Ernest

On Tue, Jun 1, 2010 at 4:50 PM, Andrey Khomyakov  wrote:

> Good times!
>
> We are starting to play around with VMware SRM and they "virtual" subnets
> that supposedly have to be able migrate from site to site in case of a
> failure of the local hardware (or software).
> Seems like to do that I'd have to run a software router on a VM that would
> redistribute the "virtual" subnet into the physical routing domain.
> does any one have any suggestions for a software router?
>
> I'm running EIGRP on the net, so I guess nothing will speak that, so I'd
> have to redistribute OSPF. Any OSPF software router software suggestion
> would be much appreciated.
>
> Or if anyone had implemented "floating" subnets, any other suggestions or
> what to look out for would be also much appreciated.
>
> Thank all in advance,
>
> --
> Andrey Khomyakov
> [khomyakov.and...@gmail.com]
>



-- 
Regards,

Ernest McCaleb

Re: Software router

[Mike]

On 01/06/2010 22:13, Jeremy Parr wrote:
> On 1 June 2010 16:50, Andrey Khomyakov  wrote:
>
>   
>> Good times!
>>
>> We are starting to play around with VMware SRM and they "virtual" subnets
>> that supposedly have to be able migrate from site to site in case of a
>> failure of the local hardware (or software).
>> Seems like to do that I'd have to run a software router on a VM that would
>> redistribute the "virtual" subnet into the physical routing domain.
>> does any one have any suggestions for a software router?
>>
>> I'm running EIGRP on the net, so I guess nothing will speak that, so I'd
>> have to redistribute OSPF. Any OSPF software router software suggestion
>> would be much appreciated.
>>
>> Or if anyone had implemented "floating" subnets, any other suggestions or
>> what to look out for would be also much appreciated.
>>
>> Thank all in advance,
>>
>> 
> Mikrotik would fit the bill.
>   
Vyatta has a VMWare image. Have used and is pretty good.

http://www.vyatta.org community edition or
http://www.vyatta.com commercial supported.

Mike

Re: Software router

[Jeremy Parr]

On 1 June 2010 16:50, Andrey Khomyakov  wrote:

> Good times!
>
> We are starting to play around with VMware SRM and they "virtual" subnets
> that supposedly have to be able migrate from site to site in case of a
> failure of the local hardware (or software).
> Seems like to do that I'd have to run a software router on a VM that would
> redistribute the "virtual" subnet into the physical routing domain.
> does any one have any suggestions for a software router?
>
> I'm running EIGRP on the net, so I guess nothing will speak that, so I'd
> have to redistribute OSPF. Any OSPF software router software suggestion
> would be much appreciated.
>
> Or if anyone had implemented "floating" subnets, any other suggestions or
> what to look out for would be also much appreciated.
>
> Thank all in advance,
>

Mikrotik would fit the bill.

Re: Software router

[Andrew D Kirch]

Really not core network related as it never touches a wire, let alone 
the core, but try www.xorp.org.


Andrew

On 06/01/2010 04:50 PM, Andrey Khomyakov wrote:

Good times!

We are starting to play around with VMware SRM and they "virtual" subnets
that supposedly have to be able migrate from site to site in case of a
failure of the local hardware (or software).
Seems like to do that I'd have to run a software router on a VM that would
redistribute the "virtual" subnet into the physical routing domain.
does any one have any suggestions for a software router?

I'm running EIGRP on the net, so I guess nothing will speak that, so I'd
have to redistribute OSPF. Any OSPF software router software suggestion
would be much appreciated.

Or if anyone had implemented "floating" subnets, any other suggestions or
what to look out for would be also much appreciated.

Thank all in advance,

   

Re: Software router state of the art

[Henning Brauer]

* Stuart Henderson <[EMAIL PROTECTED]> [2008-08-01 19:06]:
> On 2008-07-28, Joe Greco <[EMAIL PROTECTED]> wrote:
> >> I have yet to look into *BSD based solutions, but hear very good things 
> >> about firewall performance. I don't know about BGP/OSPF/MPLS etc support 
> >> on FreeBSD but am going to wager a guess its on par with Linux if not 
> >> better.
> >
> > The underlying OS is responsible for packet forwarding, but none of them
> > do any significant routing protocols natively.
> 
> OpenBSD has OpenOSPFD/OpenBGPD in the base OS rather than as a port/
> package, so it's fully coupled with any kernel changes (and supports
> some things missing from the FreeBSD port).

can't be stressed enough; the concept of
OpenBGPD/OSPFD/RIPD/DVRMPD/OSPF6D (did I forget one again?) is not
too be just another daemon implementing the protocol at hand, they
come with massive changes to the OpenBSD kernel to offer an
alternative to other solutions, including "hardware" routers.

Now it is quite clear that you don't want to run 5 loaded 10GE ports
on any Hardware OpenBSD currently supports (it's not just PCs), but
there are enough installations with smaller bandwidth requirements
where it is a very viable alternative.

-- 
Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED]
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam

Re: Software router state of the art

[Stuart Henderson]

On 2008-07-28, Joe Greco <[EMAIL PROTECTED]> wrote:
>> I have yet to look into *BSD based solutions, but hear very good things 
>> about firewall performance. I don't know about BGP/OSPF/MPLS etc support 
>> on FreeBSD but am going to wager a guess its on par with Linux if not 
>> better.
>
> The underlying OS is responsible for packet forwarding, but none of them
> do any significant routing protocols natively.

OpenBSD has OpenOSPFD/OpenBGPD in the base OS rather than as a port/
package, so it's fully coupled with any kernel changes (and supports
some things missing from the FreeBSD port).

Re: Software router state of the art

[David E. Smith]

Andrew D Kirch wrote:
Anyone have experience with RouterOS (http://www.mikrotik.com/)? 
Created mostly to run on these guys I think 
(http://www.routerboard.com/comparison.html) which generally don't 
get above 200k pps on the higher models.. But will RouterOS run on 
bigger boxen?
Yes I do, and I'm still in therapy.  I was pushing 30mbit, and I can't 
remember how many PPS through one, and it crashed about once a month 
requiring onsite intervention (usually at midnight).  This was running 
on a Compaq Deskpro I think.  It doesn't have much support for good 
network cards.  I suspect the Realtek's were behind the crashes. 
The Realteks were almost certainly to blame. Just like any other 
"server," good hardware is well worth it. RouterOS 2.9 and 3.x support 
Intel's gigabit server NICs, which work quite well.


www.mikrotikrouter.com sells a few nice purpose-built rackmount 
appliances for this sort of thing. (Just a happy customer, don't work 
there or anything.)


If you have the budget, and need the single-purpose horsepower, you'll 
usually be happier with Cisco or Juniper or someone that makes dedicated 
routing kit. If money's tight, or you need one box to do several 
network-related jobs for whatever reason, Mikrotik's software is another 
useful tool to have.


David Smith
MVN.net


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Software router state of the art

[Eugeniu Patrascu]

Aaron Glenn wrote:

On 7/28/08, Seth Mattinen <[EMAIL PROTECTED]> wrote:
  

Junpier's J-series is a BSD based platform as far as I understand it.
ImageStream is *much* more affordable for me, but is Linux-based, and I fear


...snip...

AFAIK, none of Juniper's Juniper kit rocks BSD outside of the
management interfaces and control plane (not even sure about the
latter, tbh).

someone feel free to correct me...
  
In the M/T series, control plan is handled by the RE, and the forwarding 
by the ASICs on each PIC.
in the J series, the control and forwarding plane are controlled by the 
RE, although the forwarding plane has a real time thread in the BSD 
kernel (or so Juniper says it does).

Re: Software router state of the art

[Aaron Glenn]

On 7/28/08, Seth Mattinen <[EMAIL PROTECTED]> wrote:
>
> Junpier's J-series is a BSD based platform as far as I understand it.
> ImageStream is *much* more affordable for me, but is Linux-based, and I fear
...snip...

AFAIK, none of Juniper's Juniper kit rocks BSD outside of the
management interfaces and control plane (not even sure about the
latter, tbh).

someone feel free to correct me...

Re: Software router state of the art

[Bill Nash]

On Mon, 28 Jul 2008, Rev. Jeffrey Paul wrote:


As much as I hate to contribute to the problem, I'd like to point out
that the barrage of useless, off-topic, empty traffic on this list in
the last week is, in my estimation, quite a bit above the "usual" ruckus
of NANOG.

While I'm not one to thunk down the rulebook, can you all collectively
knock it off?


I gotta disagree with you, especially with regard to this thread. Much of 
the conversations on this topic have ancillary benefits, specifically for 
folks who need to populate networks with things like 10g forensic sensors 
or similiar. I don't see commodity hardware router discussions being any 
different from a foundry vs juniper vs crisco discussion, be it typical 
fanboy nonsense or otherwise. As far as active threads on nanog go, the 
signal to noise ratio on this one has already far exceeded more 
'operational' ones. Even anecdotal experiences noted thus far have been 
pretty insightful, and useful.


I even totally resisted the urge of bombing the thread by extolling the 
virtues of the Killer NIC as a solution to all the throughput problems 
people have, because I felt it would really derail what has thus far been 
a fairly educational thread.


That said though, the more I thought about it (the killer nic joke), the 
more I looked at it. What's the state of NPU offloading amongst software 
routers? Is the notion even viable? I've seen a couple remarks about 
various brands of network cards having various buffer and interrupt driven 
issues as serious limiters to pps throughput, which is what prompted me to 
think of it in the first place.


- billn

Re: Software router state of the art

[Joe Greco]

> I wasn't too thrilled about being accused of OS politics when I was 
> genuinely concerned about deploying a software router based on things 
> I've heard in passing or read about here previously. It *is* nice to 
> know that someone found out that FreeBSD 7 hates OSPF - since I actually 
> use OSPF - and that would have tormented me for a while had I gone that 
> route.

Nonononono ... QUAGGA hates FreeBSD 7, and therefore Quagga OSPF does 
not work on FreeBSD 7.  OpenOSPFD has worked like a CHAMP.  Any issues
I have with OpenOSPFD are related to it not being Quagga, or as flexible
as Quagga, but I have had >0< issues with OpenOSPFD's reliability.  With
only a relatively short period to judge it, I'll note, but still, easy
to install, easy to deploy...

Easily enough that I'm having semi-serious thoughts ...

The problem appears to be related to FreeBSD having made changes to the
multicast API to become RFC-compliant.  Quagga has a bunch of workarounds
for various forms of brokenness present in Linux, etc., and my reading
suggests that Quagga is doing the wrong thing.

Quite frankly, this, and the loopback implosion OpenOSPFD caused when we
misconfigured it, are the worst things I've heard about software routing
this year.  Unlike most Cisco or Juniper issues, it's not a "reboot the
router" or "that'll be fixed 'soon'" type of thing.  You're free to open
up the hood and experiment yourself.

If your Cisco OSPF wasn't working, and Cisco didn't show any signs of
fixing it, it's a little more difficult to just pop the top and drop a
different routing protocol engine in.

Sorry for any misinterpretation.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Software router state of the art

[Kevin Day]

On Jul 28, 2008, at 1:55 PM, Seth Mattinen wrote:
I'm aware of Cisco IOS, then BSD-based and Linux-based platforms  
that are actually sold as routing products. I also know there are a  
billion "yay, router!" things out there. T1 cards are easy to find.  
The only other place I know I could buy a T3 card from is Sangoma.  
Maybe someone has even used it* T3 card before. Rather than reinvent  
the wheel alone, nanog has to contain the highest concentration of  
people that have tried various things and already know what will  
work and what won't work. I'm not looking for OS politics, just  
operational experience from people who have access to more money and  
more hardware than I do to have tried more stuff.


If my best option is still from the big players, so be it. If  
there's something else that's just as stable, I want to hear about  
it. I'm not adverse to some dirty work, but I just don't have the  
time right now to jump in over my head into a software router  
project and then fight my way back to the surface. I'm not trying to  
create something for educational purposes, I need something suitable  
for a production environment.


~Seth



We use a lot of Sangoma's stuff ourselves, both for data and TDM voice  
applications. For the most part, it's worked flawlessly. The few  
problems we've had were dealt with amazingly quickly on their end -  
one of their developers stayed well after midnight and gave me a  
custom fix for a problem that was pretty insignificant to us.


They support Linux a bit more strongly than FreeBSD, but both should  
work for what you need. Unless you're trying to install it on a 486,  
you'll have no problem handling 45mbps of traffic, bgp, nat,  
firewalls, etc. no matter what the PPS rate is.


You get the full source to their drivers, the only exception is the  
firmware loaded onto the echo canceler DSP for voice applications.


That said, they are a small company. Don't buy if you're expecting TAC  
level support contracts, glossy manuals or a GUI web interface to set  
the card up.


T3 levels of bandwidth are well inside the "no longer a problem" scale  
of software routing. Quagga or Xorp, combined with your favorite  
software firewall, nat, or other goodies and you're up. If I remember  
right, someone made a Xorp bootable CD that had Sangoma's drivers  
included, so you were up and running pretty fast.


If you want more specific info about anything, ask off list.

-- Kevin

Re: Software router state of the art

[Seth Mattinen]

Andrew D Kirch wrote:

Rev. Jeffrey Paul wrote:

On Mon, Jul 28, 2008 at 10:08:32PM +0100, [EMAIL PROTECTED] wrote:
 

But if you want free suggestions, then you'll have to put up with
half answers, vendor fanboys, and the usual ruckus of NANOG.




As much as I hate to contribute to the problem, I'd like to point out
that the barrage of useless, off-topic, empty traffic on this list in
the last week is, in my estimation, quite a bit above the "usual" ruckus
of NANOG.

While I'm not one to thunk down the rulebook, can you all collectively
knock it off?

Cheers,
-jp
I haven't followed the other threads in the last week, but I don't think 
that a discussion of routers is off topic.  While Michael's opinion was 
expressed in a fairly tongue-in-cheek method as would be required by his 
response, I don't see anything offtopic, perhaps just hotly worded.




I wasn't too thrilled about being accused of OS politics when I was 
genuinely concerned about deploying a software router based on things 
I've heard in passing or read about here previously. It *is* nice to 
know that someone found out that FreeBSD 7 hates OSPF - since I actually 
use OSPF - and that would have tormented me for a while had I gone that 
route.


Back to the topic at hand, unfortunately I wouldn't have the luxury of 
converting T1/T3 to Ethernet. I've used cards from Digium and Sangoma in 
the past for T1 and been relatively pleased, although older Digium cards 
hated sharing an IRQ with anything.


~Seth

Re: Software router state of the art

[Christopher Morrow]

On Mon, Jul 28, 2008 at 2:55 PM, Seth Mattinen <[EMAIL PROTECTED]> wrote:

> The problem I'm facing is that if I want something from Cisco that can do at
> least line-rate T3, I'm looking at least $20k per router. I don't have a
> uber-budget, so for me, that's kind of painful when I start to need more
> than one plus spare parts. But, I have a high level of confidence that I can
> put cards in, some memory, power it up, configure it and I'm good to go.
>

it's interesting that no one has mentioned the Nokia platform in this
discussion... they have a pc-based rackable server platform (in the
ip530/ip560 sized box) which would do T3 interfaces (from nokia I
believe even). Looking at the nokia website now I don't see WAN
capabilities below the 1220 though :( so you'd have to be in for that
at least.

-Chris

Re: Software router state of the art

[Andrew D Kirch]

Rev. Jeffrey Paul wrote:

On Mon, Jul 28, 2008 at 10:08:32PM +0100, [EMAIL PROTECTED] wrote:
  

But if you want free suggestions, then you'll have to put up with
half answers, vendor fanboys, and the usual ruckus of NANOG.




As much as I hate to contribute to the problem, I'd like to point out
that the barrage of useless, off-topic, empty traffic on this list in
the last week is, in my estimation, quite a bit above the "usual" ruckus
of NANOG.

While I'm not one to thunk down the rulebook, can you all collectively
knock it off?

Cheers,
-jp
I haven't followed the other threads in the last week, but I don't think 
that a discussion of routers is off topic.  While Michael's opinion was 
expressed in a fairly tongue-in-cheek method as would be required by his 
response, I don't see anything offtopic, perhaps just hotly worded.


Anderw

Re: Software router state of the art

[Rev. Jeffrey Paul]

On Mon, Jul 28, 2008 at 10:08:32PM +0100, [EMAIL PROTECTED] wrote:
> 
> But if you want free suggestions, then you'll have to put up with
> half answers, vendor fanboys, and the usual ruckus of NANOG.
> 

As much as I hate to contribute to the problem, I'd like to point out
that the barrage of useless, off-topic, empty traffic on this list in
the last week is, in my estimation, quite a bit above the "usual" ruckus
of NANOG.

While I'm not one to thunk down the rulebook, can you all collectively
knock it off?

Cheers,
-jp

-- 

 Rev. Jeffrey Paul-datavibe- [EMAIL PROTECTED]
  aim:x736e65616b   pgp:0xD9B3C17D  phone:1-800-403-1126
   9440 0C7F C598 01CA 2F17  D098 0A3A 4B8F D9B3 C17D
"Virtue is its own punishment."

Re: Software router state of the art

[Seth Mattinen]

[EMAIL PROTECTED] wrote:

Click for instance 


Thanks for being oh-so-helpful with a serious question. Got 
any useful answers for me? Give me a vendor that offers your 
suggestion. I don't have time for a make-it-myself solution.


Sorry, but you're in the wrong place. The IP networking consultants
are over thataway, and if you pay them a nice daily fee they will
sort out your problem for you.

But if you want free suggestions, then you'll have to put up with
half answers, vendor fanboys, and the usual ruckus of NANOG.

--Michael Dillon

P.S. that was a serious suggestion up above. If you have an interest
in software routers, then you should look at it. If you just want to
buy products then all routers are software routers, most especially 
the ones that depend on something called IOS or Junos. Focus on the

capabilities that you need and the prices. Don't try to be pretend to
be a router designer.




I'm aware of Cisco IOS, then BSD-based and Linux-based platforms that 
are actually sold as routing products. I also know there are a billion 
"yay, router!" things out there. Rather than reinvent the wheel alone, 
nanog has to contain the highest concentration of people that have tried 
various things and already know what will work and what won't work. I'm 
not looking for OS politics, just operational experience from people who 
have access to more money and more hardware than I do to have tried more 
stuff.


~Seth

Re: Software router state of the art

[Chris Stebner]

Jack Bates wrote:

Chris Stebner wrote:
This solution can most be definitely be had for under 5 grand. with 
the RSP4+'s (ECC mem) youd be looking at greater than 99.99 percent 
uptime if configured with SSO.


But if you end up needing BGP with full routes, throw that out the 
window. The RSP16's are expensive (even used relative to the RSP4) and 
usually necessarily for memory due to the current global routing table 
size. They are still cheap on the used market compared to list of most 
vendors, though.


Jack
I was "assuming" some level of route filtering/summarization as he did 
mention a single t1/t3 (at least used the word "link" - singular). Good 
point though, if you need more than 512mb mem, your gonna have to shell 
out the extra $10k for the pair of RSP16's


-chris

Re: Software router state of the art

[Eugeniu Patrascu]

Rubens Kuhl Jr. wrote:

You can use Linux without conntrack. You can either do "rmmod
ip_conntrack" (unload the module), rm /var/lib/modules/ip_conntrack
(or something like that to erase the file) or use the RAW queue to
forward some packets without connection tracking (-j NOTRACK) and some
others with conntrack (proxy redirection, captive portal and thinks
like that requires stateful forwarding in any platform).

I would be more worried about the prefix match and route cache done by
the operating system you are considering for use as a router. That
cannot be circunverted by turning off conntrack, pf or anything that
might do more with the packet that plain simple routing.
  

Hi,

As of 2.6.x kernel version (at least on 2.6.17) there is a FIB 
implementation called LC_Trie which supposedly does an O(1) route lookup 
which is very fast.
Where I live there are a lot of linux boxes deployed as routers pushing 
line rate GE for hundreds to thousand nodes computer networks while also 
deliverying QoS for each and every node.
From what I see in this thread you're more worried about T3/E3 
linecards than the actual Linux performance as a router.


As a personal example, I use a celeron 2.53Ghz with 512Mb of ram to push 
line rate 3 x 100Mbps cards wihout any discernable load reported either 
by top or uptime and that on top of Quagga with about ~ 5k prefixes.
Also, as an experiment I loaded a full routing table from one of my 
peers and besides of the increased RAM usage by Quagga to about 50MB the 
machine forwarded at the same rate, _maybe_ 1% incresed load.

RE: Software router state of the art

[michael.dillon]

> > Click for instance 

> Thanks for being oh-so-helpful with a serious question. Got 
> any useful answers for me? Give me a vendor that offers your 
> suggestion. I don't have time for a make-it-myself solution.

Sorry, but you're in the wrong place. The IP networking consultants
are over thataway, and if you pay them a nice daily fee they will
sort out your problem for you.

But if you want free suggestions, then you'll have to put up with
half answers, vendor fanboys, and the usual ruckus of NANOG.

--Michael Dillon

P.S. that was a serious suggestion up above. If you have an interest
in software routers, then you should look at it. If you just want to
buy products then all routers are software routers, most especially 
the ones that depend on something called IOS or Junos. Focus on the
capabilities that you need and the prices. Don't try to be pretend to
be a router designer.

Re: Software router state of the art

[Florian Weimer]

* Joe Greco:

> I'm not sure where the claims about "{one, few} flow{s}" are coming from.
> Certainly the number of flows on a typical UNIX box acting as a router is
> not that relevant unless you specifically configure something like 
> stateful firewalling, because the typical UNIX box simply doesn't have a
> *concept* of "flows."  It deals with packets.

You are mistaken.  Linux routing is flow-based.  Ever wondered what
those "dst cache overflow" messages mean you see during a DoS attack?
It's the flow cache complaining that it can't expire records in an
organic manner.

I don't know much about FreeBSD.  I think it got a route cache after
FreeBSD 4, too.  That's the reason why the FreeBSD 4 IP stack is still
so popular.

Re: Software router state of the art

[Jack Bates]

Chris Stebner wrote:
This solution can most be definitely be had for under 5 grand. with the 
RSP4+'s (ECC mem) youd be looking at greater than 99.99 percent uptime 
if configured with SSO.


But if you end up needing BGP with full routes, throw that out the window. The 
RSP16's are expensive (even used relative to the RSP4) and usually necessarily 
for memory due to the current global routing table size. They are still cheap on 
the used market compared to list of most vendors, though.


Jack

Re: Software router state of the art

[Chris Stebner]

Deepak Jain wrote:


The problem I'm facing is that if I want something from Cisco that 
can do at least line-rate T3, I'm looking at least $20k per router. I 
don't have a uber-budget, so for me, that's kind of painful when I 
start to need more than one plus spare parts. But, I have a high 
level of confidence that I can put cards in, some memory, power it 
up, configure it and I'm good to go.


Junpier's J-series is a BSD based platform as far as I understand it. 
ImageStream is *much* more affordable for me, but is Linux-based, and 
I fear Linux as a router and I don't know what they've done to fix 
the common gripes with Linux-as-router. I have no idea if either of 
the two have hardware assist in the cards, but my impression is that 
they are essentially software platforms with custom interface cards. 
Interface cards are important to me because I'm operating in an 
environment where my link to the outside world is probably going to 
be T1/T3.


I'm aware of Cisco IOS, then BSD-based and Linux-based platforms that 
are actually sold as routing products. I also know there are a 
billion "yay, router!" things out there. T1 cards are easy to find. 
The only other place I know I could buy a T3 card from is Sangoma. 
Maybe someone has even used it* T3 card before. Rather than reinvent 
the wheel alone, nanog has to contain the highest concentration of 
people that have tried various things and already know what will work 
and what won't work. I'm not looking for OS politics, just 
operational experience from people who have access to more money and 
more hardware than I do to have tried more stuff.


If my best option is still from the big players, so be it. If there's 
something else that's just as stable, I want to hear about it. I'm 
not adverse to some dirty work, but I just don't have the time right 
now to jump in over my head into a software router project and then 
fight my way back to the surface. I'm not trying to create something 
for educational purposes, I need something suitable for a production 
environment.




[I didn't know what to cut from above, so I left it].

What I've used and seen used before that plays both to the strengths 
of the PC as a router and addresses some of the T3 related issues -- 
especially if you control both ends of the T3.


Using an FE to T3 bridge or FE to T1 bridge as the case may be. With a 
little tuning you can put a rate shaper on the PC (prior art, very 
stable) to not run into off-PC buffering issues. Your PC has plenty of 
cheap buffer. The interface to the comms network is done through a 
dedicated, telco or computer center grade piece of gear.


Everyone here (NANOG) can agree that a 10 or 100Mb/s PC router is a no 
brainer and as long as you aren't using irresponsible gear, this thing 
will route packets forever.


Putting telco interfaces into PCs has always been a little more odd, 
but telco to ethernet bridges are fairly standard and fairly dumb. 
Depending on how many of these you have etc, you can do creative 
things with switches, FR, etc. And cost can be all over the map. I 
know Pairgain used to make good ethernet to T1 bridges, and that's 
probably the last time I remember playing with this stuff.


YMMV.

Deepak Jain
AiNET

To echo Deepak's suggestion and draw attention to the original statement 
"because I'm operating in an environment where my link to the outside 
world is probably going to be T1/T3." Would lead one to question the 
PA-MC-T3 even. Could be even cheaper if you don't need the multi-channel 
component (of course the monthly cost of the DS3 pales here in 
comparison w/ the h/w setup, but thought Id mention it regardless as it 
could save you 2 grand.) If all you need is a few t1's just pick up the 
VIP 2-50 interface card and a 4 x T1 adapter.


This solution can most be definitely be had for under 5 grand. with the 
RSP4+'s (ECC mem) youd be looking at greater than 99.99 percent uptime 
if configured with SSO.


-chris

Re: Software router state of the art

[Rubens Kuhl Jr.]

>> It keeps track of Src/Dst/QoS/Ethernet adapters/etc.. Additionally most
>> systems have the iptables modules loaded in kernel and the conntrack
>> module in kernel. This immediately activates connection tracking,
>> therefore considerably slowing down software routing. The most optimal
>> way of speeding this up would be sticking the route cache into somewhat
>> faster memory. Though it would be fairly nice to get rid of the route
>> cache as that can cause problem with eccentric setups. Also, as cache
>> entries take a moment to be deleted, or degrade leading to convergence
>> times being higher.
>
> Note .. to .. self ..  Linux .. makes .. crappy .. router.  Got it.
>
> Guess we'll continue to use FreeBSD, and the lesson to come away with
> is that it probably pays to avoid technologies that are suboptimal
> for the task at hand.  Not everything is created equal.  It also pays
> to tune things.  If "conntrack" hurts, then remove it.

You can use Linux without conntrack. You can either do "rmmod
ip_conntrack" (unload the module), rm /var/lib/modules/ip_conntrack
(or something like that to erase the file) or use the RAW queue to
forward some packets without connection tracking (-j NOTRACK) and some
others with conntrack (proxy redirection, captive portal and thinks
like that requires stateful forwarding in any platform).

I would be more worried about the prefix match and route cache done by
the operating system you are considering for use as a router. That
cannot be circunverted by turning off conntrack, pf or anything that
might do more with the packet that plain simple routing.


Rubens

Re: Software router state of the art

[Deepak Jain]

Another option (if you want a pure Cisco platform) would be to buy a 
used Cisco 7500 or 7200 and put a T3 card in there. Those are probably 
super cheap through reseller channels. (<<$20K for a 1+1).


A quick scan of Ebay shows a PA-MC-T3 for <$3K, a 7505 +RSP4+PS for $300 
and a fast ethernet blade for $30.00.


Excluding software licenses (assuming its not running something that its 
not perpetually licensed to something that will run the T3 card) you are 
looking at about $3K per T3 in HW.


Deepak  

Re: Software router state of the art

[Deepak Jain]

The problem I'm facing is that if I want something from Cisco that can 
do at least line-rate T3, I'm looking at least $20k per router. I don't 
have a uber-budget, so for me, that's kind of painful when I start to 
need more than one plus spare parts. But, I have a high level of 
confidence that I can put cards in, some memory, power it up, configure 
it and I'm good to go.


Junpier's J-series is a BSD based platform as far as I understand it. 
ImageStream is *much* more affordable for me, but is Linux-based, and I 
fear Linux as a router and I don't know what they've done to fix the 
common gripes with Linux-as-router. I have no idea if either of the two 
have hardware assist in the cards, but my impression is that they are 
essentially software platforms with custom interface cards. Interface 
cards are important to me because I'm operating in an environment where 
my link to the outside world is probably going to be T1/T3.


I'm aware of Cisco IOS, then BSD-based and Linux-based platforms that 
are actually sold as routing products. I also know there are a billion 
"yay, router!" things out there. T1 cards are easy to find. The only 
other place I know I could buy a T3 card from is Sangoma. Maybe someone 
has even used it* T3 card before. Rather than reinvent the wheel alone, 
nanog has to contain the highest concentration of people that have tried 
various things and already know what will work and what won't work. I'm 
not looking for OS politics, just operational experience from people who 
have access to more money and more hardware than I do to have tried more 
stuff.


If my best option is still from the big players, so be it. If there's 
something else that's just as stable, I want to hear about it. I'm not 
adverse to some dirty work, but I just don't have the time right now to 
jump in over my head into a software router project and then fight my 
way back to the surface. I'm not trying to create something for 
educational purposes, I need something suitable for a production 
environment.




[I didn't know what to cut from above, so I left it].

What I've used and seen used before that plays both to the strengths of 
the PC as a router and addresses some of the T3 related issues -- 
especially if you control both ends of the T3.


Using an FE to T3 bridge or FE to T1 bridge as the case may be. With a 
little tuning you can put a rate shaper on the PC (prior art, very 
stable) to not run into off-PC buffering issues. Your PC has plenty of 
cheap buffer. The interface to the comms network is done through a 
dedicated, telco or computer center grade piece of gear.


Everyone here (NANOG) can agree that a 10 or 100Mb/s PC router is a no 
brainer and as long as you aren't using irresponsible gear, this thing 
will route packets forever.


Putting telco interfaces into PCs has always been a little more odd, but 
telco to ethernet bridges are fairly standard and fairly dumb. Depending 
on how many of these you have etc, you can do creative things with 
switches, FR, etc. And cost can be all over the map. I know Pairgain 
used to make good ethernet to T1 bridges, and that's probably the last 
time I remember playing with this stuff.


YMMV.

Deepak Jain
AiNET

Re: Software router state of the art

[Seth Mattinen]

Michael 'Moose' Dinn wrote:
 Thanks for being oh-so-helpful with a serious question. Got any useful 
 answers for me? Give me a vendor that offers your suggestion. I don't have 
 time for a make-it-myself solution.




What are your requirements?



The problem I'm facing is that if I want something from Cisco that can 
do at least line-rate T3, I'm looking at least $20k per router. I don't 
have a uber-budget, so for me, that's kind of painful when I start to 
need more than one plus spare parts. But, I have a high level of 
confidence that I can put cards in, some memory, power it up, configure 
it and I'm good to go.


Junpier's J-series is a BSD based platform as far as I understand it. 
ImageStream is *much* more affordable for me, but is Linux-based, and I 
fear Linux as a router and I don't know what they've done to fix the 
common gripes with Linux-as-router. I have no idea if either of the two 
have hardware assist in the cards, but my impression is that they are 
essentially software platforms with custom interface cards. Interface 
cards are important to me because I'm operating in an environment where 
my link to the outside world is probably going to be T1/T3.


I'm aware of Cisco IOS, then BSD-based and Linux-based platforms that 
are actually sold as routing products. I also know there are a billion 
"yay, router!" things out there. T1 cards are easy to find. The only 
other place I know I could buy a T3 card from is Sangoma. Maybe someone 
has even used it* T3 card before. Rather than reinvent the wheel alone, 
nanog has to contain the highest concentration of people that have tried 
various things and already know what will work and what won't work. I'm 
not looking for OS politics, just operational experience from people who 
have access to more money and more hardware than I do to have tried more 
stuff.


If my best option is still from the big players, so be it. If there's 
something else that's just as stable, I want to hear about it. I'm not 
adverse to some dirty work, but I just don't have the time right now to 
jump in over my head into a software router project and then fight my 
way back to the surface. I'm not trying to create something for 
educational purposes, I need something suitable for a production 
environment.


~Seth


* http://www.sangoma.com/products_and_solutions/hardware/data_only/a301.html

Re: Software router state of the art

[Charles Wyble]

Andrew D Kirch wrote:

Justin Sharp wrote:

[EMAIL PROTECTED] wrote:


  


Yes I do, and I'm still in therapy.  I was pushing 30mbit, and I can't 
remember how many PPS through one, and it crashed about once a month 
requiring onsite intervention (usually at midnight).  This was running 
on a Compaq Deskpro I think.  It doesn't have much support for good 
network cards.  I suspect the Realtek's were behind the crashes.


Yeah or any number of cheap consumer parts in the Deskpro.  I would 
love to see RouterOS running on an HP or Dell mid range server and how 
that performs. I'm guessing it would be quite nice.




--
Charles Wyble (818) 280 - 7059
http://charlesnw.blogspot.com
CTO Known Element Enterprises / SoCal WiFI project

Re: Software router state of the art

[Andrew D Kirch]

Justin Sharp wrote:

[EMAIL PROTECTED] wrote:
but knowing how bad Linux is at being a router and that their 
products are Linux-based, I'm afraid to give one a try. J products 
are based on a competing non-Linux platform that has a better 
reputation for routing.



Enough with the bipartisan politics. There are more choices than just 
Linux and FreeBSD for software routing.


Click for instance 

--Michael Dillon

  
Anyone have experience with RouterOS (http://www.mikrotik.com/)? 
Created mostly to run on these guys I think 
(http://www.routerboard.com/comparison.html) which generally don't get 
above 200k pps on the higher models.. But will RouterOS run on bigger 
boxen?


Yes I do, and I'm still in therapy.  I was pushing 30mbit, and I can't 
remember how many PPS through one, and it crashed about once a month 
requiring onsite intervention (usually at midnight).  This was running 
on a Compaq Deskpro I think.  It doesn't have much support for good 
network cards.  I suspect the Realtek's were behind the crashes.


Andrew

Re: Software router state of the art

[Joe Greco]

> H. Well then you probably don't want to use Linux/BSD as a router, 
> as a substantial amount of DIY is required for anything beyond 
> relatively simple routing. MPLS support (on Linux) for example is in 
> early phases and requires integrating separate pieces and is best 
> supported on Fedora9. Needless to say, Fedora isn't designed for 
> reliable/stable operation and long term deployment.
> 
> I have yet to look into *BSD based solutions, but hear very good things 
> about firewall performance. I don't know about BGP/OSPF/MPLS etc support 
> on FreeBSD but am going to wager a guess its on par with Linux if not 
> better.

The underlying OS is responsible for packet forwarding, but none of them
do any significant routing protocols natively.  Adding on a package
such as Quagga or OpenBGPD is required for that, and the results of
each should be relatively similar across platforms.

The only major caveat is that Quagga OSPF is currently a disaster on 
FreeBSD 7.  Don't try it.  We added a server that was advertising some
stuff, with multiple interfaces, using a config identical to what we
do under FreeBSD 6.  Not only did it randomly not work, but it also
randomly killed OTHER OSPF speakers elsewhere in the network, including
on non-directly attached networks in another OSPF area (we'd log in and
see no neighbors).

OpenOSPFD appears to be the fix for that.  Simpler, smaller, but dumb
enough that it advertised 127.0.0.1 into our OSPF environment when we
were trying to get some aliases on lo0 advertised, which caused 
freaking out of pretty much every OSPF-speaking UNIX server we have 
(sigh).

BGP is straightforward, except for things like MD5, which can be a bit
dicey.  Quagga is very good, and much less expensive than, something
like Cisco for a route server, from what I've heard over the years.
You'll notice some of the Route-views boxes are Quagga or Zebra (its
predecessor).

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Software router state of the art

[Charles Wyble]

Seth Mattinen wrote:

[EMAIL PROTECTED] wrote:
but knowing how bad Linux is at being a router and that their 
products are Linux-based, I'm afraid to give one a try. J products 
are based on a competing non-Linux platform that has a better 
reputation for routing.





Thanks for being oh-so-helpful with a serious question. Got any useful 
answers for me? Give me a vendor that offers your suggestion. I don't 
have time for a make-it-myself solution.


H. Well then you probably don't want to use Linux/BSD as a router, 
as a substantial amount of DIY is required for anything beyond 
relatively simple routing. MPLS support (on Linux) for example is in 
early phases and requires integrating separate pieces and is best 
supported on Fedora9. Needless to say, Fedora isn't designed for 
reliable/stable operation and long term deployment.


I have yet to look into *BSD based solutions, but hear very good things 
about firewall performance. I don't know about BGP/OSPF/MPLS etc support 
on FreeBSD but am going to wager a guess its on par with Linux if not 
better.


To address another point made in this thread, see 
http://ols.fedoraproject.org/OLS/Reprints-2007/zhu-Reprint.pdf which 
addresses hardware multiqueue device support under Linux. Its from 
2007.  I think there was a question about Linux/multiqueue support in 
this thread, but I am not 100% sure. :)


I think there was mention of Vyatta earlier in the thread and some talk 
about it switching from Xorp to Quagga, and a supposition that should 
improve it.



--
Charles Wyble (818) 280 - 7059
http://charlesnw.blogspot.com
CTO Known Element Enterprises / SoCal WiFI project

Re: Software router state of the art

[Seth Mattinen]

[EMAIL PROTECTED] wrote:
but knowing how bad Linux is at being a router and that their 
products are Linux-based, I'm afraid to give one a try. J 
products are based on a competing non-Linux platform that has 
a better reputation for routing.


Enough with the bipartisan politics. There are more choices than 
just Linux and FreeBSD for software routing.


Click for instance 

--Michael Dillon




Thanks for being oh-so-helpful with a serious question. Got any useful 
answers for me? Give me a vendor that offers your suggestion. I don't 
have time for a make-it-myself solution.


~Seth

Re: Software router state of the art

[Justin Sharp]

[EMAIL PROTECTED] wrote:
but knowing how bad Linux is at being a router and that their 
products are Linux-based, I'm afraid to give one a try. J 
products are based on a competing non-Linux platform that has 
a better reputation for routing.



Enough with the bipartisan politics. There are more choices than 
just Linux and FreeBSD for software routing.


Click for instance 

--Michael Dillon

  
Anyone have experience with RouterOS (http://www.mikrotik.com/)? Created 
mostly to run on these guys I think 
(http://www.routerboard.com/comparison.html) which generally don't get 
above 200k pps on the higher models.. But will RouterOS run on bigger boxen?

RE: Software router state of the art

[michael.dillon]

> but knowing how bad Linux is at being a router and that their 
> products are Linux-based, I'm afraid to give one a try. J 
> products are based on a competing non-Linux platform that has 
> a better reputation for routing.

Enough with the bipartisan politics. There are more choices than 
just Linux and FreeBSD for software routing.

Click for instance 

--Michael Dillon

Re: Software router state of the art

[Seth Mattinen]

Sargun Dhillon wrote:
This is not exactly true. The modern Linux kernel (2.6) uses some amount 
of flow tracking in order to do route caching. You can check this out on 
your system by:

"ip route show cache"



Did you mean "route -C" ?

I like the idea and price point of the ImageStream products, but knowing 
how bad Linux is at being a router and that their products are 
Linux-based, I'm afraid to give one a try. J products are based on a 
competing non-Linux platform that has a better reputation for routing.


~Seth

Re: Software router state of the art

[Joe Greco]

> This is not exactly true. The modern Linux kernel (2.6) uses some amount 
> of flow tracking in order to do route caching. You can check this out on 
> your system by:
> "ip route show cache"

Okay...

# ip route show cache
ip: Command not found.
#

So I guess that's all well and good for me.

> It keeps track of Src/Dst/QoS/Ethernet adapters/etc.. Additionally most 
> systems have the iptables modules loaded in kernel and the conntrack 
> module in kernel. This immediately activates connection tracking, 
> therefore considerably slowing down software routing. The most optimal 
> way of speeding this up would be sticking the route cache into somewhat 
> faster memory. Though it would be fairly nice to get rid of the route 
> cache as that can cause problem with eccentric setups. Also, as cache 
> entries take a moment to be deleted, or degrade leading to convergence 
> times being higher.

Note .. to .. self ..  Linux .. makes .. crappy .. router.  Got it.

Guess we'll continue to use FreeBSD, and the lesson to come away with
is that it probably pays to avoid technologies that are suboptimal 
for the task at hand.  Not everything is created equal.  It also pays
to tune things.  If "conntrack" hurts, then remove it.

With the emergence of computers with many cores, it will be very
interesting to see how this develops.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Software router state of the art

[Sargun Dhillon]

This is not exactly true. The modern Linux kernel (2.6) uses some amount 
of flow tracking in order to do route caching. You can check this out on 
your system by:

"ip route show cache"

It keeps track of Src/Dst/QoS/Ethernet adapters/etc.. Additionally most 
systems have the iptables modules loaded in kernel and the conntrack 
module in kernel. This immediately activates connection tracking, 
therefore considerably slowing down software routing. The most optimal 
way of speeding this up would be sticking the route cache into somewhat 
faster memory. Though it would be fairly nice to get rid of the route 
cache as that can cause problem with eccentric setups. Also, as cache 
entries take a moment to be deleted, or degrade leading to convergence 
times being higher.






Joe Greco wrote:

On Sat, Jul 26, 2008, Florian Weimer wrote:


Was this with one packet flow, or with millions of them?
  

I believe it was >1 flow. The guy is using an Ixia; I don't know how
he has it configured.



Traditionally, software routing performance on hosts systems has been
optimized for few and rather long flows.
  

Yup.

And I always ask that question when people claim really high(!) throughput on
software forwarding. It turns out their throughput was single source/single
dest, and/or large packets (so high throughput, but low pps.)



I'm not sure where the claims about "{one, few} flow{s}" are coming from.
Certainly the number of flows on a typical UNIX box acting as a router is
not that relevant unless you specifically configure something like 
stateful firewalling, because the typical UNIX box simply doesn't have a

*concept* of "flows."  It deals with packets.  This has its own problems,
of course, but handling high levels of traffic in many flows is not one of
them.

There are other software routing platforms that DO flows, but the above
mentions "host[s] systems", so I'm reading that as "UNIX router."

On the flip side, packet size is definitely a consideration.  This topic
has been beaten to death on the Zebra mailing lists by myself and others
in the past.  


With yesterday's technology (P4 3.0G, 512MB RAM, PCI-X, FreeBSD 4) we were
successfully dealing with >300Kpps about 3 years ago, without substantial
work.  That was single source/single dest, but with a full routing table.
There's no real optimization for that within the FreeBSD framework, so it
is about the same performance you'd have gotten with multi source/multi
dest.

... JG
  




--
+1.925.202.9485
Sargun Dhillon
deCarta
[EMAIL PROTECTED]
www.decarta.com

RE: Software router state of the art

[Darden, Patrick S.]

I have one of these (Imagestream T3 WAN adapter on an Imagestream router) for 
5+ years to back up my Cisco 7204 with a channelized T3 card.  I like the 
system, I like the card.

The other engineers in my office call it the "bling router".  Lots of gold 
chrome.  Pimped out.
--Patrick Darden


-Original Message-
From: Chris Adams [mailto:[EMAIL PROTECTED]

Once upon a time, Andrew D Kirch <[EMAIL PROTECTED]> said:
> there 
> are no Channelized T3/OC3 cards available for a PeeCee, which means you 
> still need to buy something from Cisco or Juniper.

First hit on a Google search:

http://www.imagestream.com/PCI_921-CDS.html

-- 
Chris Adams <[EMAIL PROTECTED]>

Re: Software router state of the art

[Tony Finch]

On Sat, 26 Jul 2008, Dorn Hetzel wrote:

> Ok, it's probably a stupid question, but given the relative ease of putting
> 4gb+ ram on a 64bit platform,
> could packet per second performance be improved by brute forcing the route
> lookup as an array of 1 byte destination interface indexes for a contiguous
> swath of /32's from bottom to top?

Much easier if you filter out any longer prefixes than /24 :-)

Tony.
-- 
f.anthony.n.finch  <[EMAIL PROTECTED]>  http://dotat.at/
IRISH SEA: VARIABLE 3 OR 4 BECOMING NORTHEAST 4 OR 5. SLIGHT. FOG PATCHES,
THUNDERY SHOWERS LATER. MODERATE OR GOOD, OCCASIONALLY VERY POOR.

Re: Software router state of the art

[Petri Helenius]

William Herrin wrote:

The pro/1000 does not need to generate an interrupt in order to start
a DMA transfer? Can you refer me to some documents which explain in
detail how a card on the bus sets up a DMA transfer?
  
The driver provides the adapter a ring buffer of memory locations to 
busmaster dma the data into (which does not require interrupting the 
CPU). The interrupts are triggered after the DMA completes and in 
moderation controllable by the driver. For FreeBSD the default maxes 
interrupts out at 8000 per second and on some of the adapters there are 
firmware optimizations for lowering the latency from the obvious maximum 
of 125 microseconds. When an interrupt is fired the driver restocks the 
ring buffer with new addresses to put data into, for one or for 4000 
frames, depending on how many were used up.


With IOAT and various offloads this gets somewhat more complicated and 
more effective.


Pete

Re: Software router state of the art

[Seth Mattinen]

Chris Adams wrote:

Once upon a time, Andrew D Kirch <[EMAIL PROTECTED]> said:
I'd like to be wrong, but there's no way that any PC/Commodity routing 
system is going to work (in any environment other than Ethernet).  For 
the small ISP starting out (you know, the ones selling T1's/xDSL), there 
are no Channelized T3/OC3 cards available for a PeeCee, which means you 
still need to buy something from Cisco or Juniper.


First hit on a Google search:

http://www.imagestream.com/PCI_921-CDS.html



ImageStream isn't really "look I just bought a Dell and now I need to 
route my DS3 with BGP" is it?


Does anyone reading this use any ImageStream gear in production? I'm 
curious what your experiences are.


~Seth

Re: Software router state of the art

[Chris Adams]

Once upon a time, Andrew D Kirch <[EMAIL PROTECTED]> said:
> I'd like to be wrong, but there's no way that any PC/Commodity routing 
> system is going to work (in any environment other than Ethernet).  For 
> the small ISP starting out (you know, the ones selling T1's/xDSL), there 
> are no Channelized T3/OC3 cards available for a PeeCee, which means you 
> still need to buy something from Cisco or Juniper.

First hit on a Google search:

http://www.imagestream.com/PCI_921-CDS.html

-- 
Chris Adams <[EMAIL PROTECTED]>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: Software router state of the art

[Andrew D Kirch]

Zed Usser wrote:

Hi all!

There's been some discussion on the list regarding software routers lately and 
this piqued my interest. Does anybody have any recent performance and 
capability statistics (eg. forwarding rates with full BGP tables and N ethernet 
interfaces) or any pointer to what the current state of art in software routers 
is?

- Zed
I'd like to be wrong, but there's no way that any PC/Commodity routing 
system is going to work (in any environment other than Ethernet).  For 
the small ISP starting out (you know, the ones selling T1's/xDSL), there 
are no Channelized T3/OC3 cards available for a PeeCee, which means you 
still need to buy something from Cisco or Juniper.  And the major 
carriers are already using Cisco/Juniper, because even at the price they 
charge they aren't unreasonable because they support their product.  I 
don't care how many packets per second, or simultaneous route flows you 
can do if I can't plug anything besides Ethernet into it.  If you can 
show me the hardware, great I'll take a look at it, otherwise these 
simply don't matter all that much.


Andrew

Re: Software router state of the art

[Florian Weimer]

* Dorn Hetzel:

> Ok, it's probably a stupid question, but given the relative ease of
> putting 4gb+ ram on a 64bit platform, could packet per second
> performance be improved by brute forcing the route lookup as an array
> of 1 byte destination interface indexes for a contiguous swath of
> /32's from bottom to top?

8 bits per destination are not enough because you really want to put all
the necessary L2 information into the FIB.  Perhaps even AS numbers, for
flow export.

Re: Software router state of the art

[Florian Weimer]

* William Herrin:

> The pro/1000 does not need to generate an interrupt in order to start
> a DMA transfer? Can you refer me to some documents which explain in
> detail how a card on the bus sets up a DMA transfer?

Busmaster DMA does not generate an interrupt on the host CPU.  The
interrupt is used to trigger processing on the host CPU; it can be
deferred until several frames have been written.

Re: Software router state of the art

[William Herrin]

On Sat, Jul 26, 2008 at 1:40 PM, Petri Helenius <[EMAIL PROTECTED]> wrote:
> William Herrin wrote:
>> But cards like the Intel Pro/1000 have 64k of memory for buffering
>> packets, both in and out. Few have very much more than 64k. 64k means
>> 32k to tx and 32k to rx. Means you darn well better generate an
>> interrupt when you get near 16k so that you don't fill the buffer
>> before the 16k you generated the interrupt for has been cleared. Means
>> you're generating an interrupt at least for every 10 or so 1500 byte
>> packets.
>>
>
> This is not true in the bus master dma mode how the cards are usually used.
> The mentioned memory is used only as temporary storage until the card can
> DMA the data into the buffers in main memory. Most Pro/1000 cards have
> buffering capability up to 4096 frames.

Pete,

I'll confess to some ignorance here. We're at the edge of my skill set.

The pro/1000 does not need to generate an interrupt in order to start
a DMA transfer? Can you refer me to some documents which explain in
detail how a card on the bus sets up a DMA transfer?

Thanks,
Bill


-- 
William D. Herrin  [EMAIL PROTECTED] [EMAIL PROTECTED]
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Software router state of the art

[Petri Helenius]

William Herrin wrote:

"ethtool -c". Thanks Sargun for putting me on to "I/O Coalescing."

But cards like the Intel Pro/1000 have 64k of memory for buffering
packets, both in and out. Few have very much more than 64k. 64k means
32k to tx and 32k to rx. Means you darn well better generate an
interrupt when you get near 16k so that you don't fill the buffer
before the 16k you generated the interrupt for has been cleared. Means
you're generating an interrupt at least for every 10 or so 1500 byte
packets.
  
This is not true in the bus master dma mode how the cards are usually 
used. The mentioned memory is used only as temporary storage until the 
card can DMA the data into the buffers in main memory. Most Pro/1000 
cards have buffering capability up to 4096 frames.


Pete

Re: Software router state of the art

[William Herrin]

On Sat, Jul 26, 2008 at 7:41 AM, Dorn Hetzel <[EMAIL PROTECTED]> wrote:
> Ok, it's probably a stupid question, but given the relative ease of putting
> 4gb+ ram on a 64bit platform,
> could packet per second performance be improved by brute forcing the route
> lookup as an array of 1 byte destination interface indexes for a contiguous
> swath of /32's from bottom to top?
>
> Route updates would be a little ugly, 2^24 bytes to rewrite for a /8, but
> forwarding lookups out to be a single indexed read ?

Dorn,

In theory with about 6 gigs of ram for the IPv4 FIB, sure. But:

You're significantly multiplying the likelihood of a cache miss when
performing a lookup. You can do a fair amount of tree traversal in
cache for the price of one miss.

You're a tad shy on ram to try this with IPv6.

Regards,
Bill Herrin

-- 
William D. Herrin  [EMAIL PROTECTED] [EMAIL PROTECTED]
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Software router state of the art

[Joe Greco]

> On Sat, Jul 26, 2008, Florian Weimer wrote:
> > Was this with one packet flow, or with millions of them?
> 
> I believe it was >1 flow. The guy is using an Ixia; I don't know how
> he has it configured.
> 
> > Traditionally, software routing performance on hosts systems has been
> > optimized for few and rather long flows.
> 
> Yup.
> 
> And I always ask that question when people claim really high(!) throughput on
> software forwarding. It turns out their throughput was single source/single
> dest, and/or large packets (so high throughput, but low pps.)

I'm not sure where the claims about "{one, few} flow{s}" are coming from.
Certainly the number of flows on a typical UNIX box acting as a router is
not that relevant unless you specifically configure something like 
stateful firewalling, because the typical UNIX box simply doesn't have a
*concept* of "flows."  It deals with packets.  This has its own problems,
of course, but handling high levels of traffic in many flows is not one of
them.

There are other software routing platforms that DO flows, but the above
mentions "host[s] systems", so I'm reading that as "UNIX router."

On the flip side, packet size is definitely a consideration.  This topic
has been beaten to death on the Zebra mailing lists by myself and others
in the past.  

With yesterday's technology (P4 3.0G, 512MB RAM, PCI-X, FreeBSD 4) we were
successfully dealing with >300Kpps about 3 years ago, without substantial
work.  That was single source/single dest, but with a full routing table.
There's no real optimization for that within the FreeBSD framework, so it
is about the same performance you'd have gotten with multi source/multi
dest.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Software router state of the art

[Dorn Hetzel]

Ok, it's probably a stupid question, but given the relative ease of putting
4gb+ ram on a 64bit platform,
could packet per second performance be improved by brute forcing the route
lookup as an array of 1 byte destination interface indexes for a contiguous
swath of /32's from bottom to top?

Route updates would be a little ugly, 2^24 bytes to rewrite for a /8, but
forwarding lookups out to be a single indexed read ?


On Sat, Jul 26, 2008 at 7:31 AM, Adrian Chadd <[EMAIL PROTECTED]>wrote:

> On Sat, Jul 26, 2008, Colin Alston wrote:
> > >And I always ask that question when people claim really high(!)
> throughput
> > >on
> > >software forwarding. It turns out their throughput was single
> source/single
> > >dest, and/or large packets (so high throughput, but low pps.)
> >
> > I assume though that all of this is on x86 platform hardware. How does
> > this compare to Linux or FreeBSD running on something else like the
> > Cavium Octeon and other 64bit MIPS based processors?
>
> You'll have to ask the people playing with it on that.
>
> Me, I've been looking for some multicore MIPS + fruit for some Squid
> related hackery but I've been busy with other things (like, you know,
> making Squid-2 be able to be run on multi-core hardware in the first
> place..) so it'll have to wait.. :)
>
>
>
>
> Adrian
>
>
>

Re: Software router state of the art

[Adrian Chadd]

On Sat, Jul 26, 2008, Colin Alston wrote:
> >And I always ask that question when people claim really high(!) throughput 
> >on
> >software forwarding. It turns out their throughput was single source/single
> >dest, and/or large packets (so high throughput, but low pps.)
> 
> I assume though that all of this is on x86 platform hardware. How does 
> this compare to Linux or FreeBSD running on something else like the 
> Cavium Octeon and other 64bit MIPS based processors?

You'll have to ask the people playing with it on that.

Me, I've been looking for some multicore MIPS + fruit for some Squid
related hackery but I've been busy with other things (like, you know,
making Squid-2 be able to be run on multi-core hardware in the first
place..) so it'll have to wait.. :)




Adrian

Re: Software router state of the art

[Colin Alston]

On 2008/07/26 01:05 PM Adrian Chadd wrote:

On Sat, Jul 26, 2008, Florian Weimer wrote:

Traditionally, software routing performance on hosts systems has been
optimized for few and rather long flows.


Yup.

And I always ask that question when people claim really high(!) throughput on
software forwarding. It turns out their throughput was single source/single
dest, and/or large packets (so high throughput, but low pps.)


I assume though that all of this is on x86 platform hardware. How does 
this compare to Linux or FreeBSD running on something else like the 
Cavium Octeon and other 64bit MIPS based processors?

Re: Software router state of the art

[Adrian Chadd]

On Sat, Jul 26, 2008, Florian Weimer wrote:

> Was this with one packet flow, or with millions of them?

I believe it was >1 flow. The guy is using an Ixia; I don't know how
he has it configured.

> Traditionally, software routing performance on hosts systems has been
> optimized for few and rather long flows.

Yup.

And I always ask that question when people claim really high(!) throughput on
software forwarding. It turns out their throughput was single source/single
dest, and/or large packets (so high throughput, but low pps.)




Adrian

Re: Software router state of the art

[Florian Weimer]

* Adrian Chadd:

> 1 mil pps has been broken that way, but it uses lots of cores to get there.
> (8, I think?)

Was this with one packet flow, or with millions of them?

Traditionally, software routing performance on hosts systems has been
optimized for few and rather long flows.

Anyway, with multi-core, you don't need funky algorithms for incremental
FIB updates anymore (if you don't need sub-second convergence and stuff
like that).  As a result, you can use really dumb multi-way trees for
which a lookup takes something like 100 CPU cycles (significantly less
for non-DoS traffic with higher locality).

Re: Software router state of the art

[Florian Weimer]

* William Herrin:

> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.

But they are receive-only, right?

The main problem for "software routing" seems to be that it's basically
Ethernet-only because other interfaces are very difficult to find.

Re: Software router state of the art

[Joe Greco]

> Would you rather deploy a $3000 cisco edge box which is a unexpandable, 
> 100 mbit piece of crap, or throw two $2000 Dell boxes and have a 1 GigE 
> platform?

You don't need two $2000 Dell boxes to get a 1G platform, but this isn't
the list for that.  You also don't need a ton of money to do open source
development (we do one major project here in-house, one that's responsible
for generating a noticeable amount of traffic on most networks).  But this
also isn't the list for that discussion.

I suspect software routing is going to continue to be a growing factor in
the packet herding world, as more people want to do more interesting
things, and the CPU's are able to deal with more, more, more.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Software router state of the art

[Sargun Dhillon]

It would be very useful if there was an effort from the telecom 
community to develop a dynamic routing frontend like Quagga. The amount 
of human work that it requires in order to build up a product is 
enormous. If only someone with millions of dollars could donate 
engineers. It would allow the deployment of small branch office systems 
at a much lower cost.


Would you rather deploy a $3000 cisco edge box which is a unexpandable, 
100 mbit piece of crap, or throw two $2000 Dell boxes and have a 1 GigE 
platform?



Joe Greco wrote:
Last thing to say is, I haven't tried upgrading since Vyatta abandoned 
the XORP platform and moved to the Quagga platform, but I'm guessing 
(based on experience w/ Quagga) that they have a lot fewer of these 
quirks that I've described.



Quagga is pretty decent, but it is not uncommon for serious bugs to go
unaddressed for a long time.  For example, this bug renders Quagga
nearly unusable for OSPF on FreeBSD 7,

http://bugzilla.quagga.net/show_bug.cgi?id=420

which resulted in some finger-pointing, but the last I heard, it was 
due to a kernel interface change where FreeBSD multicast code had been 
rewritten and was DTRT, while Linux was doing something else.


This is probably still better than the XORP platform, but it is 
unfortunate.


... JG
  



--
+1.925.202.9485
Sargun Dhillon
deCarta
[EMAIL PROTECTED]
www.decarta.com

Re: Software router state of the art

[Joe Greco]

> Last thing to say is, I haven't tried upgrading since Vyatta abandoned 
> the XORP platform and moved to the Quagga platform, but I'm guessing 
> (based on experience w/ Quagga) that they have a lot fewer of these 
> quirks that I've described.

Quagga is pretty decent, but it is not uncommon for serious bugs to go
unaddressed for a long time.  For example, this bug renders Quagga
nearly unusable for OSPF on FreeBSD 7,

http://bugzilla.quagga.net/show_bug.cgi?id=420

which resulted in some finger-pointing, but the last I heard, it was 
due to a kernel interface change where FreeBSD multicast code had been 
rewritten and was DTRT, while Linux was doing something else.

This is probably still better than the XORP platform, but it is 
unfortunate.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Software router state of the art

[Justin Sharp]

Yes. We put in some Vyatta routers to extend our corporate network into 
another building as a temporary solution (the building had a very short 
lease, so our boss didn't want to spend any money on Juniper which is 
our usual net gear vendor). Consequently, we are still there.. go figure.


When we started w/ them, they were still using the XORP routing engine 
(and we haven't upgraded to the new platform yet). My experience wasn't 
terribly good. The first issue was a bad memory leak in the router 
manager process when VRRP hello times were set to 1 second. The first 
indication of something wrong is that our master router crashed, 
followed by his backup. Had to physically reboot the boxes to get them 
back online, which involved driving there as no one onsite had access to 
the cage at the office. All voice and data ran through these routers, 
basically rendering every employee useless until we got it back online. 
It wasn't a happy day. After that we had to monitor memory and do 
controlled reboots every month or so. We eventually convinced Vyatta of 
this memory leak and they were able to fix it, but that was a very 
frustrating process, and time consuming for us, which is why the next 
problems I describe, we have just found our own workarounds.


The next problem was a combination of a problem with the Vyatta and a 
problem w/ our IP phones. The Vyatta was sending garp's for the data 
vrrp address out the voice vlan (same 2 routers are default gate on both 
data and voice vlans). All of the workstations run through the phones 
(which sit tagged on voice vlan, and pass traffic from workstation 
untagged to data vlan). The phone, seeing the arp for the data vrrp 
address on its voice vlan, would send traffic to that address out the 
voice vlan, effectively taking that workstation off the net for anything 
other than local traffic. That was a bugger to figure out, and basically 
we solved it with an arptables rule on the vyatta's. That was the one 
advantage of using a Linux (debian) based router platform, was that we 
could load other 'unsupported' packages to solve problems like this.


The last thing is that OSPF never really converges correctly. You can 
view the OSPF database, and see which default the routers should 
converge to, but they do not. They will sit converged to one path for a 
while, and if you reboot the other router that generates default, they 
will reconverge to it for a while. This hasn't been a big enough problem 
for me to worry about it.


Last thing to say is, I haven't tried upgrading since Vyatta abandoned 
the XORP platform and moved to the Quagga platform, but I'm guessing 
(based on experience w/ Quagga) that they have a lot fewer of these 
quirks that I've described.


IMHO, YMMV, etc

--Justin

Tim Sanderson wrote:

Is anyone using Vyatta for routing? I sure would like to know about any 
experience with it in production.

http://www.vyatta.com/

--
Tim Sanderson, network administrator
[EMAIL PROTECTED]


-Original Message-
From: randal k [mailto:[EMAIL PROTECTED]
Sent: Wednesday, July 23, 2008 1:46 PM
To: Adrian Chadd
Cc: [EMAIL PROTECTED]
Subject: Re: Software router state of the art

That is a very interesting paper. Seriously, 7mpps with an
off-the-shelf Dell 2950? Even if it were -half- that throughput, for a
pure ethernet forwarding solution that is incredible. Shoot, buy a
handful of them as hot spares and still save a bundle.

Highly recommended reading, even if (like me) you're anti-commodity routing.

Cheers,
Randal

On Wed, Jul 23, 2008 at 10:17 AM, Adrian Chadd <[EMAIL PROTECTED]> wrote:
  

On Wed, Jul 23, 2008, Charles Wyble wrote:



This might be of interest:

http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf
  

Various FreeBSD related guys are working on parallelising the forwarding
layer enough to use the multiple tx/rx queues in some chipsets such as the
Intel gig/10ge stuff.

1 mil pps has been broken that way, but it uses lots of cores to get there.
(8, I think?)

Linux apparently is/has headed down this path.




  

RE: Software router state of the art

[Tim Sanderson]

Is anyone using Vyatta for routing? I sure would like to know about any 
experience with it in production.

http://www.vyatta.com/

--
Tim Sanderson, network administrator
[EMAIL PROTECTED]


-Original Message-
From: randal k [mailto:[EMAIL PROTECTED]
Sent: Wednesday, July 23, 2008 1:46 PM
To: Adrian Chadd
Cc: [EMAIL PROTECTED]
Subject: Re: Software router state of the art

That is a very interesting paper. Seriously, 7mpps with an
off-the-shelf Dell 2950? Even if it were -half- that throughput, for a
pure ethernet forwarding solution that is incredible. Shoot, buy a
handful of them as hot spares and still save a bundle.

Highly recommended reading, even if (like me) you're anti-commodity routing.

Cheers,
Randal

On Wed, Jul 23, 2008 at 10:17 AM, Adrian Chadd <[EMAIL PROTECTED]> wrote:
> On Wed, Jul 23, 2008, Charles Wyble wrote:
>
>> This might be of interest:
>>
>> http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf
>
> Various FreeBSD related guys are working on parallelising the forwarding
> layer enough to use the multiple tx/rx queues in some chipsets such as the
> Intel gig/10ge stuff.
>
> 1 mil pps has been broken that way, but it uses lots of cores to get there.
> (8, I think?)
>
> Linux apparently is/has headed down this path.

sizing router buffers (Re: Software router state of the art )

[Mikael Abrahamsson]

On Wed, 23 Jul 2008, Kevin Oberman wrote:


be of any use at all. This would require 3 GB of buffers. This same
problem also make TCP off-load of no use at all.


3 Gigabyte? Why?

The newer 40G platforms on the market seems to have abandonded the 600ms 
buffers typical in the 10G space, in favour of 50-200ms of buffers (I 
don't remember exactly).


Aren't there TCP implementations that don't use exponential window 
increase, but instead can do smaller increments, which I would have 
believed would enable routers to still do well with ~50ms of buffering.


High speed memory is very expensive, also a lot of applications today 
would prefer to have their packets dropped instead of being queued for 
hundreds of milliseconds. Finding a good tradeoff level between the demand 
of different traffic types is quite hard...


Also, DWDM capacity seems to get cheaper all the time, so if you really 
need to move data at multigigabit speeds, it might make sense to just rent 
that 10G wave and put your own equipment there that does what you want.


--
Mikael Abrahamssonemail: [EMAIL PROTECTED]

Re: Software router state of the art

[Kevin Oberman]

> Date: Wed, 23 Jul 2008 16:51:50 -0400
> From: "William Herrin" <[EMAIL PROTECTED]>
> Sender: [EMAIL PROTECTED]
> 
> On Wed, Jul 23, 2008 at 3:59 PM, Kevin Oberman <[EMAIL PROTECTED]> wrote:
> >> The first bottleneck is the interrupts from the NIC. With a generic
> >> Intel NIC under Linux, you start to lose a non-trivial number of
> >> packets around 700mbps of "normal" traffic because it can't service
> >> the interrupts quickly enough.
> >
> > Most modern high performance network cards support MSI (Message Signaled
> > Interrupts) which generate real interrupts only in an intelligent
> > basis. and only at a controlled rate. Windows, Solaris and FreeBSD have
> > support for MSI and I think Linux does, too. It requires both hardware
> > and software support.
> 
> "ethtool -c". Thanks Sargun for putting me on to "I/O Coalescing."
> 
> But cards like the Intel Pro/1000 have 64k of memory for buffering
> packets, both in and out. Few have very much more than 64k. 64k means
> 32k to tx and 32k to rx. Means you darn well better generate an
> interrupt when you get near 16k so that you don't fill the buffer
> before the 16k you generated the interrupt for has been cleared. Means
> you're generating an interrupt at least for every 10 or so 1500 byte
> packets.

You have just hit on a huge problems with most (all?) 1G and 10G
hardware. The buffers are way too small for optimal performance in any
case where the RTT is anything more that half a millisecond, you exhaust
the window and stall the stream.

I need port move multi-gigabit streams across the country and between the
US and Europe. Those are a bit too far apart for those tiny buffers to
be of any use at all. This would require 3 GB of buffers. This same
problem also make TCP off-load of no use at all.
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: [EMAIL PROTECTED]   Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4  EADA 927D EBB3 987B 3751


pgpMiyt2bekzk.pgp
Description: PGP signature

Re: Software router state of the art

[William Herrin]

On Wed, Jul 23, 2008 at 3:59 PM, Kevin Oberman <[EMAIL PROTECTED]> wrote:
>> The first bottleneck is the interrupts from the NIC. With a generic
>> Intel NIC under Linux, you start to lose a non-trivial number of
>> packets around 700mbps of "normal" traffic because it can't service
>> the interrupts quickly enough.
>
> Most modern high performance network cards support MSI (Message Signaled
> Interrupts) which generate real interrupts only in an intelligent
> basis. and only at a controlled rate. Windows, Solaris and FreeBSD have
> support for MSI and I think Linux does, too. It requires both hardware
> and software support.

"ethtool -c". Thanks Sargun for putting me on to "I/O Coalescing."

But cards like the Intel Pro/1000 have 64k of memory for buffering
packets, both in and out. Few have very much more than 64k. 64k means
32k to tx and 32k to rx. Means you darn well better generate an
interrupt when you get near 16k so that you don't fill the buffer
before the 16k you generated the interrupt for has been cleared. Means
you're generating an interrupt at least for every 10 or so 1500 byte
packets.

Regards,
Bill



-- 
William D. Herrin  [EMAIL PROTECTED] [EMAIL PROTECTED]
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Software router state of the art

[Kevin Oberman]

> Date: Wed, 23 Jul 2008 14:17:53 -0400
> From: "William Herrin" <[EMAIL PROTECTED]>
> 
> On Wed, Jul 23, 2008 at 2:03 PM, Naveen Nathan <[EMAIL PROTECTED]> wrote:
> >> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
> >> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
> >
> > I wonder, has anyone heard of this used for IDS? I've been looking at
> > building a commodity SNORT solution, and wondering if a powerful network
> > card will help, or would the bottleneck be in processing the packets and
> > overhead from the OS?
> 
> The first bottleneck is the interrupts from the NIC. With a generic
> Intel NIC under Linux, you start to lose a non-trivial number of
> packets around 700mbps of "normal" traffic because it can't service
> the interrupts quickly enough.

Most modern high performance network cards support MSI (Message Signaled
Interrupts) which generate real interrupts only in an intelligent
basis. and only at a controlled rate. Windows, Solaris and FreeBSD have
support for MSI and I think Linux does, too. It requires both hardware
and software support.

With MSI, TSO, LRO, and PCI-E with hardware that supports these, 9.5
Gbps TCP flows between systems is possible with minimal tuning. That
puts the bottleneck back on the forwarding software in the CPU to do
the forwarding at high rates.
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: [EMAIL PROTECTED]   Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4  EADA 927D EBB3 987B 3751


pgpQ9niz4enfu.pgp
Description: PGP signature

Re: Software router state of the art

[Jeffrey Ollie]

On Wed, Jul 23, 2008 at 11:17 AM, Adrian Chadd <[EMAIL PROTECTED]> wrote:
>
> Various FreeBSD related guys are working on parallelising the forwarding
> layer enough to use the multiple tx/rx queues in some chipsets such as the
> Intel gig/10ge stuff.
>
> Linux apparently is/has headed down this path.

I've seen some notes from Dave Miller about adding multiple TX queues
to the 2.6.27 kernel.  See Dave's blog for the gory details:

http://vger.kernel.org/~davem/cgi-bin/blog.cgi
http://git.kernel.org/?p=linux/kernel/git/davem/net-tx-2.6.git;a=summary

AFAIK he hasn't made any claims about performance improvements.  I
don't know the state of RX queues in Linux.

Jeff

Re: Software router state of the art

[Wes Young]

We use them here and there (the 1Gig versions). The biggest thing to  
think about is the types of rule-sets you'll be using compounded by  
the number of flows being created / expired. Once tuned, they work  
quite well, but the balance is how fast you can pull/analyze out of  
RAM. Compiling the rules down to the card's level speeds things up a  
bit, but at the loss of using more dynamic rulesets.


If you can get the raw data to some sort of larger medium (say,  
rotating pcaps on a disk), you length the buffer-window. FWIW however,  
probably the best way to scale this is get an Xport fiber regen tap,  
populate with a few of these, tune them to monitor different segments  
based on address space or port ranges. You'll have yourself a  
relatively cheap solution, but extremely effective solution.


I've yet to test out the NinjaProbes... It's on my todo list...

On Jul 23, 2008, at 2:21 PM, Christopher Morrow wrote:

On Wed, Jul 23, 2008 at 11:05 AM, Naveen Nathan <[EMAIL PROTECTED]>  
wrote:
The Endace DAG cards claim they can move 7 gbps over a PCI-X bus  
from

the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.


I wonder, has anyone heard of this used for IDS? I've been looking at
building a commodity SNORT solution, and wondering if a powerful  
network
card will help, or would the bottleneck be in processing the  
packets and

overhead from the OS?


http://www.endace.com/our-products/ninja-appliances/NinjaProbe-NIDS

snort at 1g & 10g

-chris



--
Wes Young
Network Security Analyst
CIT - University at Buffalo
http://claimid.com/saxjazman9









smime.p7s
Description: S/MIME cryptographic signature

Re: Software router state of the art

[Christopher Morrow]

On Wed, Jul 23, 2008 at 11:05 AM, Naveen Nathan <[EMAIL PROTECTED]> wrote:
>> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
>> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
>
> I wonder, has anyone heard of this used for IDS? I've been looking at
> building a commodity SNORT solution, and wondering if a powerful network
> card will help, or would the bottleneck be in processing the packets and
> overhead from the OS?

http://www.endace.com/our-products/ninja-appliances/NinjaProbe-NIDS

snort at 1g & 10g

-chris

Re: Software router state of the art

[William Herrin]

On Wed, Jul 23, 2008 at 2:03 PM, Naveen Nathan <[EMAIL PROTECTED]> wrote:
>> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
>> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
>
> I wonder, has anyone heard of this used for IDS? I've been looking at
> building a commodity SNORT solution, and wondering if a powerful network
> card will help, or would the bottleneck be in processing the packets and
> overhead from the OS?

The first bottleneck is the interrupts from the NIC. With a generic
Intel NIC under Linux, you start to lose a non-trivial number of
packets around 700mbps of "normal" traffic because it can't service
the interrupts quickly enough.

The DAG card can be dropped in to replace the interface used for a
libpcap-based application. When I tested the 1gbps PCIE version, I
lost no packets to 1gbps and my capture application's CPU usage
dropped to about 1/5th of what it was with the generic NIC. YMMV.

Regards,
Bill Herrin


-- 
William D. Herrin  [EMAIL PROTECTED] [EMAIL PROTECTED]
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Software router state of the art

[Chris Adams]

Once upon a time, Adam Armstrong <[EMAIL PROTECTED]> said:
> Sounds like a Juniper J-series. Have a look at the forwarding figures 
> for the J6350. It does something around 2mpps and it's just an intel CPU 
> with some PCI/PCI-X interfaces. The device just below it, the J4350 uses 
> a 2.53Ghz celeron. I'm not sure what the J6350 uses.

IIRC the new slots (the EPIMs) are PCI-E.  The J6350 CPU is a P4 3.4GHz.
It is my understanding that the J-series use a real-time layer under the
FreeBSD kernel and have a real-time thread for forwarding (as opposed to
the M-series with a hardware forwarding engine).

-- 
Chris Adams <[EMAIL PROTECTED]>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: Software router state of the art

[Naveen Nathan]

> The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
> the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.

I wonder, has anyone heard of this used for IDS? I've been looking at
building a commodity SNORT solution, and wondering if a powerful network
card will help, or would the bottleneck be in processing the packets and
overhead from the OS?

- naveen

Re: Software router state of the art

[Adam Armstrong]

Adrian Chadd wrote:

On Wed, Jul 23, 2008, Charles Wyble wrote:


Sure its not a CRS-1, but reliably doing a mil pps with a smattering of
low-touch features would be rather useful, no?

(Then, add say, l2tp/ppp into that mix, just as a crazy on-topic example..)
  
Sounds like a Juniper J-series. Have a look at the forwarding figures 
for the J6350. It does something around 2mpps and it's just an intel CPU 
with some PCI/PCI-X interfaces. The device just below it, the J4350 uses 
a 2.53Ghz celeron. I'm not sure what the J6350 uses.


adam.

Re: Software router state of the art

[randal k]

That is a very interesting paper. Seriously, 7mpps with an
off-the-shelf Dell 2950? Even if it were -half- that throughput, for a
pure ethernet forwarding solution that is incredible. Shoot, buy a
handful of them as hot spares and still save a bundle.

Highly recommended reading, even if (like me) you're anti-commodity routing.

Cheers,
Randal

On Wed, Jul 23, 2008 at 10:17 AM, Adrian Chadd <[EMAIL PROTECTED]> wrote:
> On Wed, Jul 23, 2008, Charles Wyble wrote:
>
>> This might be of interest:
>>
>> http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf
>
> Various FreeBSD related guys are working on parallelising the forwarding
> layer enough to use the multiple tx/rx queues in some chipsets such as the
> Intel gig/10ge stuff.
>
> 1 mil pps has been broken that way, but it uses lots of cores to get there.
> (8, I think?)
>
> Linux apparently is/has headed down this path.

Re: Software router state of the art

[Joel Jaeggli]

[EMAIL PROTECTED] wrote:

On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:

There's been some discussion on the list regarding software routers


The performance of "software routers" has always had a hardware component.

Basically, for the vast majority of them, take your PCI bus bandwidth,
count how many times a packet has to cross it, and do the math.  You can't
forward more than that much traffic no matter *what* software you run on
that box.  If that number falls short, stop right there and look for
some box of different design that has the required backplane bandwidth.

You will, of course, take additional performance hits due to locking issues
and similar in your software stack (that, and most "software" routers will
suffer from not having special hardware assist for routing table lookups).


The current state of the art is around 2 million pps for fast intel arch 
system.



Let us know if you find a suitable chassis/motherboard that has enough
bandwidth to make it worth thinking about for anything other than the
smaller edge routers that most providers have zillions of... :)

Re: Software router state of the art

[Adrian Chadd]

On Wed, Jul 23, 2008, Chris Marlatt wrote:

> http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-06/msg00364.html 
> has all the details. It's rather long thread but 1mpps was achieved on a 
> single cpu IIRC (the server had multiple cpus but only one being used 
> for forwarding). Firewall rules slowed it down quite a bit but theres 
> also some work out there being done to minimize this.

Yah, all of that is happening. Some people keep asking why FreeBSD-4
forwarding was always much faster than same-hardware forwarding under
current FreeBSD but at least thats finally being worked on.

Of course, with my FreeBSD advocacy hat on, if you -want- to see
something like FreeBSD handle 1mil+ pps forwarding then you should
really drop the FreeBSD Foundation a line and introduce yourself.
There are developers working on this (note: not me! :) who would
benefit from equipment and funding.

Anyway. Some PC class hardware is pretty damned fast. Some vendors
even build highish-throughput firewalls and proxies out of PC class
hardware. :) The "wah wah PC class hardware has anemic bus IO/memory IO/
CPU speed/ethernet modules and is thus too crap for serious routing" argument
is pretty much over for at least 1 mil pps, perhaps more.

2c,


Adrian

Re: Software router state of the art

[Chris Marlatt]

Adrian Chadd wrote:


1 mil pps has been broken that way, but it uses lots of cores to get there.
(8, I think?)



http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-06/msg00364.html 
has all the details. It's rather long thread but 1mpps was achieved on a 
single cpu IIRC (the server had multiple cpus but only one being used 
for forwarding). Firewall rules slowed it down quite a bit but theres 
also some work out there being done to minimize this.


Regards,

Chris

Re: Software router state of the art

[Adrian Chadd]

On Wed, Jul 23, 2008, Charles Wyble wrote:

> This might be of interest:
> 
> http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf

Various FreeBSD related guys are working on parallelising the forwarding
layer enough to use the multiple tx/rx queues in some chipsets such as the
Intel gig/10ge stuff.

1 mil pps has been broken that way, but it uses lots of cores to get there.
(8, I think?)

Linux apparently is/has headed down this path.

If someone were to spend some time dissecting the rest of the code to
also optimise the single-core throughput then you may see some interesting
software routers using commodity hardware (for values of "commodity"
roughly equal to "PC servers", rather than "magic lotsacore core MIPS with
some extra glue for jacking packets around."

Sure its not a CRS-1, but reliably doing a mil pps with a smattering of
low-touch features would be rather useful, no?

(Then, add say, l2tp/ppp into that mix, just as a crazy on-topic example..)



Adrian

Re: Software router state of the art

[William Herrin]

On Wed, Jul 23, 2008 at 11:24 AM,  <[EMAIL PROTECTED]> wrote:
> On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
>> There's been some discussion on the list regarding software routers
>
> The performance of "software routers" has always had a hardware component.
>
> Basically, for the vast majority of them, take your PCI bus bandwidth,
> count how many times a packet has to cross it, and do the math.  You can't
> forward more than that much traffic no matter *what* software you run on
> that box.  If that number falls short, stop right there and look for
> some box of different design that has the required backplane bandwidth.

The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from
the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.

Regards,
Bill Herrin



-- 
William D. Herrin  [EMAIL PROTECTED] [EMAIL PROTECTED]
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Software router state of the art

[Charles Wyble]

[EMAIL PROTECTED] wrote:

On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
  

There's been some discussion on the list regarding software routers



The performance of "software routers" has always had a hardware component.

Basically, for the vast majority of them, take your PCI bus bandwidth,
count how many times a packet has to cross it, and do the math.  You can't
forward more than that much traffic no matter *what* software you run on
that box.  If that number falls short, stop right there and look for
some box of different design that has the required backplane bandwidth.
  


This might be of interest:

http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf


--

Charles Wyble (818) 280 - 7059
http://charlesnw.blogspot.com
CTO Known Element Enterprises / SoCal WiFI project

Re: Software router state of the art

[Valdis . Kletnieks]

On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
> There's been some discussion on the list regarding software routers

The performance of "software routers" has always had a hardware component.

Basically, for the vast majority of them, take your PCI bus bandwidth,
count how many times a packet has to cross it, and do the math.  You can't
forward more than that much traffic no matter *what* software you run on
that box.  If that number falls short, stop right there and look for
some box of different design that has the required backplane bandwidth.

You will, of course, take additional performance hits due to locking issues
and similar in your software stack (that, and most "software" routers will
suffer from not having special hardware assist for routing table lookups).

Let us know if you find a suitable chassis/motherboard that has enough
bandwidth to make it worth thinking about for anything other than the
smaller edge routers that most providers have zillions of... :)



pgp0gGVsylFmF.pgp
Description: PGP signature