Re: Online games stealing your bandwidth

[gordon b slater]

On Sun, 2010-09-26 at 07:47 +0800, Adrian Chadd wrote:

> I don't recall any protocols being standard.
> 
> Plenty of people sell p2p caches but they all work using magic, smoke
> and mirrors. 
> 
> 
> Adrian

Less smoky is the relatively common practice (at least in Europe) of
tech-friendly ISPs running bittorrent for "release days" of Linux and
BSD distros; Ubuntu releases especially because they have a large
proportion of release-day installs (!) and the servers get hit hard. 

How much of this is just staff doing their customers a favor is
debateable, but I know two places where it's written into SOPs for
Debian and FreeBSD major releases (about a week or two after the
release). Linux distribution by bittorrent is sometimes harder now that
more Tier 1 ISPs block or inspect the P2P traffic
By a bit of quick fiddling you can ensure that users outside your blocks
don't get served.  
I'm talking installation ISOs, not the ports or packages - the rsyncs
and mirrors take care of that as normal.

For the Debian 5.05 release I provided 700GB+ in a week for the x86
Gnome CD alone via BT, the AMD64 CD was about twice that, yet most of
the Debian stuff will be done using Jigdo, so that's a fraction of the
actual traffic. The Debian Netboot CD's seem quite popular too but
especially for exotic hardware archs. The 5.06 release is still flowing
nicely.

The last few FreeBSD releases I've pushed 500GB each time though I hold
them open for much longer for the less popular architectures.

The thought of it all flying round in such long circles dismays me
somewhat. There's probably an reasonable argument for temporary ISP-BT
of this stuff, as it'll save us all a tiny bit of peak and a lot of
packets, all for very little kit, space and man-hours.

The rest of the torrent users, (games or copytheft), can surely
be /dev/null-ed ?  :) Hmm, can I smell burning torches in the distance?

Gord
--
# Hahaha, hehehe, I'm a little Gnome and I hate KDE

Software-based Border Router

[Nathanael C. Cariaga]

Hi All! 


Just want to ask if anyone here had experience deploying software-based routers 
to serve as perimeter / border router? How does it gauge with hardware-based 
routers? Any past experiences will be very much appreciated. 


I wanted to know because we've been asked if we want to assume full control of 
the internet link (up to the router). By assuming control up to the router, we 
still want to configure iBGP with our parent network so that we can take 
advantage of some routes available to the parent network's gateway. The saddest 
part is presently we do not have the router to serve as our gateway this is why 
we are considering the use of software-based routers. 


Thank you. 

Re: Software-based Border Router

[sthaug]

> Just want to ask if anyone here had experience deploying software-based 
> routers to serve as perimeter / border router? How does it gauge with 
> hardware-based routers? Any past experiences will be very much appreciated. 

Software based routers (e.g. Cisco 7200 series) have been used as border
routers for many years - this is hardly anything new. The question you
should ask is probably: Can such a router handle a full link's worth of
DDoS using minimum sized packets? The answer, of course, depends on your
link capacity, the router itself, features enabled (ACLs, QoS, ...) etc.

There are quite a few people using Quagga based boxes running Linux or
FreeBSD as border routers - this is a possible solution too, giving
you more bang for the buck than a traditional software based router from
the big vendors. Make sure you have enough expertise for the relevant OS
and routing software available.

Steinar Haug, Nethelp consulting, sth...@nethelp.no

Re: Software-based Border Router

[Nathanael C. Cariaga]

Thank you for the prompt response.  Just to clarify my previous post, I was 
actually referring to Linux/Unix-based routers.  We've been considering this 
solution because presently we don't have any budget for equipment acquisition 
this year.

To be honest, I came across Vyatta Core while searching for viable 
Linux/Unix-based solution that we can adopt and I'm currently reading its 
reference guides.  Has anyone here used this software before?  

Thanks a lot.

- Original Message -
From: sth...@nethelp.no
To: nccari...@stluke.com.ph
Cc: nanog@nanog.org
Sent: Sunday, September 26, 2010 5:59:21 PM
Subject: Re: Software-based Border Router

> Just want to ask if anyone here had experience deploying software-based 
> routers to serve as perimeter / border router? How does it gauge with 
> hardware-based routers? Any past experiences will be very much appreciated. 

Software based routers (e.g. Cisco 7200 series) have been used as border
routers for many years - this is hardly anything new. The question you
should ask is probably: Can such a router handle a full link's worth of
DDoS using minimum sized packets? The answer, of course, depends on your
link capacity, the router itself, features enabled (ACLs, QoS, ...) etc.

There are quite a few people using Quagga based boxes running Linux or
FreeBSD as border routers - this is a possible solution too, giving
you more bang for the buck than a traditional software based router from
the big vendors. Make sure you have enough expertise for the relevant OS
and routing software available.

Steinar Haug, Nethelp consulting, sth...@nethelp.no

Re: large icmp packet issue

[Heath Jones]

> How can i be sure even if a device blocks my ping , it might have policy
> blocking ping at it at all.
Correct in a lot of cases and that is why icmp should not be used by
itself when diagnosing issues.

>> I am having problem getting ping to work to a specific destination host when
>> using large size icmp packet and i am hoping someone here can offer some
>> suggestion. With regular ping, i can ping this remote host without any 
>> problem,
>> but if i crank up the packet size to above 1500 (1500 still works), i won't 
>> get any icmp reply.
>> My first thought was this was a pmtu issue. but when I ran tcpdump on this 
>> remote host,
>> i saw the incoming ping requests and this host actually sent back icmp 
>> replies, so it appears
>> that there is some device in between blocking these large size icmp reply 
>> packets.
It is possible that the MTU for interface facing you and interface
facing away from you are different on some middle hop. It is
interesting that you state the packet size to be >1500, are you
talking about jumbo frames?
(and do you mean frame size, not packet size?)

>> Here is the question, how can i find out which hop on the path is causing 
>> this behavior?
Robert is correct. You need to use traceroute, or alter the TTL values
when you send the icmp requests.
By setting dont-fragment and varying ttl & frame sizes, you should
find your issue.

Re: Online games stealing your bandwidth

[Valdis . Kletnieks]

On Sat, 25 Sep 2010 17:41:16 CDT, Robert Bonomi said:
> On Sun, 26 Sep 2010 00:01:38 , Jeroen Massar said:
> > So it that is true, if you define "news server" as a "cache", even
> > though you have to buy several terabytes, make that several petabytes,
> > to "be able to "cache" this data one along with all the network
> > environment to support getting data out of this "cache", the ISP is
> > completely in the clear even though that "cache" is the sole single
> > point where one can retrieve that "cached" data from even years after
> > the data was originally put on the network, the original is gone and
> > that "cache" works without anything being attached to it ? :)

"sole single point" is a sticking point, since at that point you've crossed
over from "cache" to "archive" - see below.

> There is existant _recent case law, specifically with regard to operating
> a newsserver, that holds otherwise.  Of course the server operator was
> blatently _advertizing_ the copyright-infringing (and more) nature of 
> their content.

Haven't read that case law yes, but I'm willing to guess the operator of the
newsserver was running it in violation of 17 USC 512 (b)(2)(E)(i) - which
basically says you need to respond to takedown requests if the upstream copy
has already been removed (which will often be the case with netnews). So if
you're the sole single point caching something that's now gone from upstream,
you have a problem (and a stale cache). But of course, we all run technically
competent caches that expire stuff when it goes stale (unless your business
model includes advertising you have a stale cache)...

The *real* fun for Bittorrent is (b)(2)(E)(ii): "The party giving the
notification includes in the notification a statement confirming that the
material has been removed from the originating site or access to it has been
disabled or that a court has ordered that the material be removed from the
originating site or that access to the material on the originating site be
disabled." - which could be difficult to do when the material has been
assembled block-by-block from dozens of other sources and few records kept of
the actual provenance of any given block.  They might have to resort to finding
a helpful judge willing to sign a "John Doe" order for each takedown.



pgp5ketZssdCt.pgp
Description: PGP signature

Re: large icmp packet issue

[ML]

On 9/25/2010 10:57 PM, fedora fedora wrote:

I am having problem getting ping to work to a specific destination host when
using large size icmp packet and i am hoping someone here can offer some
suggestion.

With regular ping, i can ping this remote host without any problem, but if i
crank up the packet size to above 1500 (1500 still works), i won't get any
icmp reply.

My first thought was this was a pmtu issue. but when I ran tcpdump on this
remote host, i saw the incoming ping requests and this host actually sent
back icmp replies, so it appears that there is some device in between
blocking these large size icmp reply packets.

Here is the question, how can i find out which hop on the path is causing
this behavior?

FD


Can you provide a pcap file?

Re: Software-based Border Router

[Ingo Flaschberger]

Dear Nathanael,

Just want to ask if anyone here had experience deploying software-based 
routers to serve as perimeter / border router? How does it gauge with 
hardware-based routers? Any past experiences will be very much 
appreciated.



I wanted to know because we've been asked if we want to assume full 
control of the internet link (up to the router). By assuming control up 
to the router, we still want to configure iBGP with our parent network 
so that we can take advantage of some routes available to the parent 
network's gateway. The saddest part is presently we do not have the 
router to serve as our gateway this is why we are considering the use of 
software-based routers.


I operate freebsd / quagga core routers since 4 years.

pro: cheap, tcpdump at router
con: no support, no wirespeed

expected performance: 100kpps (1,2ghz pentium m) - 700kpps (quad intel
core 2, 3ghz) - and much more with 10gige cards

issues: 4byte asn produced a crash at quagga (downtime 2h in 4 years)

to develop a good core-router, this means not only to setup a pc with unix 
and for example quagga, but setup an embedded unix to an appliance, for 
example with cf-cards (readonly).


Kind regards,
Ingo Flaschberger

RE: Software-based Border Router

[Dennis Burgess]

While Vyatta is a good piece of software for the Free version, the costs 
quickly increases as you have to purchase support and the version updates are 
few and far between with the Free version.  The production (paid) version 
though is quite nice.

Another option though would be RouterOS.  If it is a small site, doing BGP 
could be as little as $399 including the hardware!  However, most people that 
do BGP will need a bit more horsepower.  RouterOS will do your iBGP, OSPF, 
bandwidth controls, firewalling etc.  The software license there is $45 beans! 
Super cheap.  Hardware runs as low as $49 bucks to 10k depending on what you 
are needing.  If you would like, please feel free to contact me off-list and I 
will be glad to recommend the proper hardware.  

---
Dennis Burgess, CCNA, A+, Mikrotik Certified Trainer
Link Technologies, Inc -- Mikrotik & WISP Support Services
Office: 314-735-0270 Website: http://www.linktechs.net
LIVE On-Line Mikrotik Training - Author of "Learn RouterOS" 

-Original Message-
From: Nathanael C. Cariaga [mailto:nccari...@stluke.com.ph] 
Sent: Sunday, September 26, 2010 5:15 AM
To: sth...@nethelp.no
Cc: nanog@nanog.org
Subject: Re: Software-based Border Router

Thank you for the prompt response.  Just to clarify my previous post, I was 
actually referring to Linux/Unix-based routers.  We've been considering this 
solution because presently we don't have any budget for equipment acquisition 
this year.

To be honest, I came across Vyatta Core while searching for viable 
Linux/Unix-based solution that we can adopt and I'm currently reading its 
reference guides.  Has anyone here used this software before?  

Thanks a lot.

- Original Message -
From: sth...@nethelp.no
To: nccari...@stluke.com.ph
Cc: nanog@nanog.org
Sent: Sunday, September 26, 2010 5:59:21 PM
Subject: Re: Software-based Border Router

> Just want to ask if anyone here had experience deploying software-based 
> routers to serve as perimeter / border router? How does it gauge with 
> hardware-based routers? Any past experiences will be very much appreciated. 

Software based routers (e.g. Cisco 7200 series) have been used as border 
routers for many years - this is hardly anything new. The question you should 
ask is probably: Can such a router handle a full link's worth of DDoS using 
minimum sized packets? The answer, of course, depends on your link capacity, 
the router itself, features enabled (ACLs, QoS, ...) etc.

There are quite a few people using Quagga based boxes running Linux or FreeBSD 
as border routers - this is a possible solution too, giving you more bang for 
the buck than a traditional software based router from the big vendors. Make 
sure you have enough expertise for the relevant OS and routing software 
available.

Steinar Haug, Nethelp consulting, sth...@nethelp.no

Re: Routers in Data Centers

[Chris Adams]

Once upon a time, Joel Jaeggli  said:
> On Sep 25, 2010, at 9:05, Seth Mattinen  wrote:
> >>> From the datacenter operator prospective, it would be nice if some of 
> >>> these vendors would acknowledge the need for front-to-back cooling. I 
> >>> mean, it is 2010.
> 
> Bakplanes make direct front to back cooling hard. non-modular platforms can 
> do it just fine however.

There are servers and storage arrays that have a front that is nothing
but hot-swap hard drive bays (plugged into backplanes), and they've been
doing front-to-back cooling since day one.  Maybe the router vendors
need to buy a Dell, open the case, and take a look.

The server vendors also somehow manage to make an empty case that costs
less than $10,000 (they'll even fill it up with useful stuff for less
than that).

-- 
Chris Adams 
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: Odd BGP AS Path

[Sriram, Kotikalapudi]

We (at NIST) performed some additional data analysis to characterize
AS_SETs in BGP updates in RIB entries. The results can be found
in the slides at this link:

http://www.antd.nist.gov/~ksriram/AS_SET_Aggregator_Stats.pdf

This work was presented at the IETF SIDR WG meeting in Maastricht in July 2010.
Please look at mainly slides #3 and #4.
The findings are consistent with Olaf Maennel's (that Randy posted) in term of 
how
few updates have AS_SETs in them.
We additionally looked at the the question of -- when AS_SET is present, how 
often
does the ASN in the AGGRGATOR field in the update matches the originating AS?
This question has relevance to origin AS validation using ROAs in RPKI --
to try to propose how origin validation can be done if AS_SET is present in an 
update.
Please look into my slides if you are interested in the details.
I will be happy to answer any questions.

Currently, the push from several of us (Warren Kumari, Randy, many others -- I 
support it too)
is to deprecate AS_SETs altogether in future BGP enhancements such as RPKI.
This would mean an update will not pass origin validation
(in fact, origin validation using ROAs will not even be attempted) if AS_SET
is present in an update.
Comments/reaction/feedback to this from ops people who currently use AS_SETs
would be very useful.
(This would be good for discussion of Warren's draft in the IDR WG.
http://tools.ietf.org/html/draft-wkumari-deprecate-as-sets-00 )

Sriram


>Date: Thu, 23 Sep 2010 10:59:07 -0400
>From: Randy Bush 
>Subject: Re: Odd BGP AS Path
>To: Warren Kumari 
>Cc: nanog@nanog.org
>
>just to uncloak a bit.  when we first decided to look at deprecating
>as-sets et alia, we begged olaf maennel to analyse the use thereof.  the
>following is edited from internal email from early august.  we then
>begged one of our number, warren, to do the dirty, and he was kind
>enough to do so.  we owe him.
>
>---
>
>using data from ripe r00

>yearstotal real
>stable  as-sets   as-sets
>  7 42
>  6 62
>  5203
>  4225
>  364   16
>  2   214   87
>  1   875  289
>
> years stable is how many years that as-set was announced for that
> prefix.
>
> total as-sets is the number of prefixes that had any as-set.
>
>real as-sets is the number of prefixes with as-sets which were not
>singleton asns and were not private asns.
>
>olaf scanned for ten years but none were stable for more than seven.
>
>in ten years, 1205 different prefixes with as-sets were seen.  removing
>private asns and removing singletons left only 404 prefixes over the ten
>years.
>
>he did not check for covering prefixes, i.e. two prefixes with the same
>as-set where one was a sub-prefix of the other, longer, one.
>
>and i suspect the data are somewhat self-similar.  i.e. the 289 that
>were stable for the last year may have shorter term components which
>were much higher.  i.e. looking at data for a week or a day may give
>higher values.
>
>a graph which should be pretty self-explanitory may be found at
>
>
> http://archive.psg.com/fraction-and-absolute-number-of-ASsets-in-table-over-time-valids-only.pdf
>
>it is interesting that, at no time, i.e. in no single rib dump, were
>there more than 23 prefixes with as-sets.  while this is suspicious, it
>seems to be true.
>
>randy

Re: Software-based Border Router

[Joel Jaeggli]

If one has a cisco 7200, then you have a software based border router.

Considerations, for a given router platform are capacity,  susceptability to 
dos, features required etc. Depending on the capacity required a software 
device could do fine. If it's in front of hosting environment you want to know 
that it doesn't take dirt nap from a couple hundred mb/s of small packet.

Joel's widget number 2

On Sep 26, 2010, at 2:41, "Nathanael C. Cariaga"  
wrote:

> Hi All! 
> 
> 
> Just want to ask if anyone here had experience deploying software-based 
> routers to serve as perimeter / border router? How does it gauge with 
> hardware-based routers? Any past experiences will be very much appreciated. 
> 
> 
> I wanted to know because we've been asked if we want to assume full control 
> of the internet link (up to the router). By assuming control up to the 
> router, we still want to configure iBGP with our parent network so that we 
> can take advantage of some routes available to the parent network's gateway. 
> The saddest part is presently we do not have the router to serve as our 
> gateway this is why we are considering the use of software-based routers. 
> 
> 
> Thank you. 
> 

Re: Routers in Data Centers

[Joel Jaeggli]

On Sep 26, 2010, at 8:26, Chris Adams  wrote:
> Once upon a time, Joel Jaeggli  said:
>> On Sep 25, 2010, at 9:05, Seth Mattinen  wrote:
> From the datacenter operator prospective, it would be nice if some of 
> these vendors would acknowledge the need for front-to-back cooling. I 
> mean, it is 2010.
>> 
>> Bakplanes make direct front to back cooling hard. non-modular platforms can 
>> do it just fine however.
> 
> There are servers and storage arrays that have a front that is nothing
> but hot-swap hard drive bays (plugged into backplanes), and they've been
> doing front-to-back cooling since day one.  Maybe the router vendors
> need to buy a Dell, open the case, and take a look.

The backplane for a sata disk array is 8 wires per drive plus a common power 
bus.
  
> 
> The server vendors also somehow manage to make an empty case that costs
> less than $10,000 (they'll even fill it up with useful stuff for less
> than that).

Unit volume is little higher, and the margins kind of suck. There's a reason 
why hp would rather sell you a blade server chassis than 16 1us.

Equating servers and routers is like equating bouncy castle prices with renting 
an oil platform.

> -- 
> Chris Adams 
> Systems and Network Administrator - HiWAAY Internet Services
> I don't speak for anybody but myself - that's enough trouble.
> 

Re: Routers in Data Centers

[Chris Adams]

Once upon a time, Joel Jaeggli  said:
> On Sep 26, 2010, at 8:26, Chris Adams  wrote:
> > There are servers and storage arrays that have a front that is nothing
> > but hot-swap hard drive bays (plugged into backplanes), and they've been
> > doing front-to-back cooling since day one.  Maybe the router vendors
> > need to buy a Dell, open the case, and take a look.
> 
> The backplane for a sata disk array is 8 wires per drive plus a common power 
> bus.

Server vendors managed cooling just fine for years with 80 pin SCA
connectors.  Hard drives are also harder to cool, as they are a solid
block, filling the space, unlike a card of chips.

I'm not saying the problems are the same, but I am saying that a
backplane making cooling "hard" is not a good excuse, especially when
the small empty chassis costs $10K+.
-- 
Chris Adams 
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: Routers in Data Centers

[Joel Jaeggli]

Joel's widget number 2

On Sep 26, 2010, at 10:47, Chris Adams  wrote:

> Once upon a time, Joel Jaeggli  said:
>> On Sep 26, 2010, at 8:26, Chris Adams  wrote:
>>> There are servers and storage arrays that have a front that is nothing
>>> but hot-swap hard drive bays (plugged into backplanes), and they've been
>>> doing front-to-back cooling since day one.  Maybe the router vendors
>>> need to buy a Dell, open the case, and take a look.
>> 
>> The backplane for a sata disk array is 8 wires per drive plus a common power 
>> bus.
> 
> Server vendors managed cooling just fine for years with 80 pin SCA
> connectors.  Hard drives are also harder to cool, as they are a solid
> block, filling the space, unlike a card of chips.

It's the same 80 wires on every single drive in the string.

There are fewer conductors embedded in 12 drive sca backplane as there are in a 
12 drive sata backplane, in both cases they are generally two layer pcbs. 
Compared to what 10+ layer pcbs that are a approaching 1/4" thick on the 
router. 

Hard drives are 6-12w each, a processor complex that's north of 200w per card 
is a rather different cooling exercise.  

> I'm not saying the problems are the same, but I am saying that a
> backplane making cooling "hard" is not a good excuse, especially when
> the small empty chassis costs $10K+.
> -- 
> Chris Adams 
> Systems and Network Administrator - HiWAAY Internet Services
> I don't speak for anybody but myself - that's enough trouble.
> 

Re: Routers in Data Centers

[Seth Mattinen]

On 9/26/10 11:09 AM, Joel Jaeggli wrote:
> 
> 
> Joel's widget number 2
> 
> On Sep 26, 2010, at 10:47, Chris Adams  wrote:
> 
>> Once upon a time, Joel Jaeggli  said:
>>> On Sep 26, 2010, at 8:26, Chris Adams  wrote:
 There are servers and storage arrays that have a front that is nothing
 but hot-swap hard drive bays (plugged into backplanes), and they've been
 doing front-to-back cooling since day one.  Maybe the router vendors
 need to buy a Dell, open the case, and take a look.
>>>
>>> The backplane for a sata disk array is 8 wires per drive plus a common 
>>> power bus.
>>
>> Server vendors managed cooling just fine for years with 80 pin SCA
>> connectors.  Hard drives are also harder to cool, as they are a solid
>> block, filling the space, unlike a card of chips.
> 
> It's the same 80 wires on every single drive in the string.
> 
> There are fewer conductors embedded in 12 drive sca backplane as there are in 
> a 12 drive sata backplane, in both cases they are generally two layer pcbs. 
> Compared to what 10+ layer pcbs that are a approaching 1/4" thick on the 
> router. 

Aw come on, that's no reason you can't just drill it full of holes. I
mean, it is 2010. It should be wireless by now.

~Seth

Re: Software-based Border Router

[William Herrin]

On Sun, Sep 26, 2010 at 6:15 AM, Nathanael C. Cariaga
 wrote:
> Thank you for the prompt response.  Just to clarify my previous
> post, I was actually referring to Linux/Unix-based routers.
> We've been considering this solution because presently we
> don't have any budget for equipment acquisition this year.

What's your time worth?

Quagga on Linux is a fine software, but messing with the
idiosyncrasies is far more time consuming than buying a Cisco 2811,
adding enough RAM to handle BGP, configuring it once and forgetting
about it.

Also bear in mind that while your ISP's engineers can help you
configure your Cisco router, Quagga is a mystery to them. You can
still get help... but not from someone who also knows how the ISP's
network is configured.

This is not a problem if you have lots of experience with BGP routing. Do you?

Regards,
Bill Herrin



-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Software-based Border Router

[Fletcher Kittredge]

Another big problem for Linux/Unix-based routers of this size/cost is
upgrade-ability.   If you need to add cards, you are going to have to bring
the router down for extended periods.   Likewise, a software upgrade can be
a bigger deal than on a purpose designed router.   If a router is mission
critical, Linux/Unixed-based has issues over extended periods.

regards,
Fletcher

On Sun, Sep 26, 2010 at 4:35 PM, William Herrin  wrote:

> On Sun, Sep 26, 2010 at 6:15 AM, Nathanael C. Cariaga
>  wrote:
> > Thank you for the prompt response.  Just to clarify my previous
> > post, I was actually referring to Linux/Unix-based routers.
> > We've been considering this solution because presently we
> > don't have any budget for equipment acquisition this year.
>
> What's your time worth?
>
> Quagga on Linux is a fine software, but messing with the
> idiosyncrasies is far more time consuming than buying a Cisco 2811,
> adding enough RAM to handle BGP, configuring it once and forgetting
> about it.
>
> Also bear in mind that while your ISP's engineers can help you
> configure your Cisco router, Quagga is a mystery to them. You can
> still get help... but not from someone who also knows how the ISP's
> network is configured.
>
> This is not a problem if you have lots of experience with BGP routing. Do
> you?
>
> Regards,
> Bill Herrin
>
>
>
> --
> William D. Herrin  her...@dirtside.com  b...@herrin.us
> 3005 Crane Dr. .. Web: 
> Falls Church, VA 22042-3004
>
>


-- 
Fletcher Kittredge
GWI
8 Pomerleau Street
Biddeford, ME 04005-9457
207-602-1134

Re: Software-based Border Router

[Ingo Flaschberger]

Another big problem for Linux/Unix-based routers of this size/cost is
upgrade-ability.   If you need to add cards, you are going to have to bring
the router down for extended periods.   Likewise, a software upgrade can be
a bigger deal than on a purpose designed router.   If a router is mission
critical, Linux/Unixed-based has issues over extended periods.


depends on knowledge, as mentioned in previous post.

I have 2 software based border routers - no problem bringing one down.
700kpps for 1200eur that can handle a full view.

and changing line-cards - could be really funny at c6500.

kind regards,
Ingo Flaschberger

Re: Software-based Border Router

[khatfield]

I do agree here. If you are not moving a lot of data then something like BSD or 
Vyatta may be a good alternative.  You do still have possible reboots required 
and things you would not see as often with a hardware-appliance model. However, 
for cheaper than the cost of 1 appliance you could build in redundancy. I guess 
the question is how many PPS you plan to push, whether you have regularly 
scheduled maintenance windows that you could bring it down for a reboot, and 
whether the additional maintenance involved still keeps you in the black? 

I am a big proponent of open source every thing. Although, I am a bigger 
proponent of stability and less maintenance. If you could prove out a 
software-based solution against the cost of a hardware solution then I don't 
see any reason not to go that route.
-Original Message-
From: Fletcher Kittredge 
Date: Sun, 26 Sep 2010 17:21:57 
To: William Herrin
Cc: 
Subject: Re: Software-based Border Router

Another big problem for Linux/Unix-based routers of this size/cost is
upgrade-ability.   If you need to add cards, you are going to have to bring
the router down for extended periods.   Likewise, a software upgrade can be
a bigger deal than on a purpose designed router.   If a router is mission
critical, Linux/Unixed-based has issues over extended periods.

regards,
Fletcher

On Sun, Sep 26, 2010 at 4:35 PM, William Herrin  wrote:

> On Sun, Sep 26, 2010 at 6:15 AM, Nathanael C. Cariaga
>  wrote:
> > Thank you for the prompt response.  Just to clarify my previous
> > post, I was actually referring to Linux/Unix-based routers.
> > We've been considering this solution because presently we
> > don't have any budget for equipment acquisition this year.
>
> What's your time worth?
>
> Quagga on Linux is a fine software, but messing with the
> idiosyncrasies is far more time consuming than buying a Cisco 2811,
> adding enough RAM to handle BGP, configuring it once and forgetting
> about it.
>
> Also bear in mind that while your ISP's engineers can help you
> configure your Cisco router, Quagga is a mystery to them. You can
> still get help... but not from someone who also knows how the ISP's
> network is configured.
>
> This is not a problem if you have lots of experience with BGP routing. Do
> you?
>
> Regards,
> Bill Herrin
>
>
>
> --
> William D. Herrin  her...@dirtside.com  b...@herrin.us
> 3005 Crane Dr. .. Web: 
> Falls Church, VA 22042-3004
>
>


-- 
Fletcher Kittredge
GWI
8 Pomerleau Street
Biddeford, ME 04005-9457
207-602-1134

Re: Routers in Data Centers

[Adam Armstrong]

 On 24/09/2010 11:22, Venkatesh Sriram wrote:

Hi,

Can somebody educate me on (or pass some pointers) what differentiates
a router operating and optimized for data centers versus, say a router
work in the metro ethernet space? What is it thats required for
routers operating in data centers? High throughput, what else?


Depending upon the specific requirements of the scenario at each type of 
site, the optimal devices could be either identical, or completely 
different.


:)

adam.

Re: Routers in Data Centers

[ym1r . jr]

As far as I know open source solutions doesn't have support for fabric or high 
speed asics. So the throughput will always be a big difference. Unless you are 
comparing a pure packet software interrupt platform.
--Original Message--
From: Adam Armstrong
To: Venkatesh Sriram
To: nanog@nanog.org
Subject: Re: Routers in Data Centers
Sent: Sep 25, 2010 7:18 PM

  On 24/09/2010 11:22, Venkatesh Sriram wrote:
> Hi,
>
> Can somebody educate me on (or pass some pointers) what differentiates
> a router operating and optimized for data centers versus, say a router
> work in the metro ethernet space? What is it thats required for
> routers operating in data centers? High throughput, what else?

Depending upon the specific requirements of the scenario at each type of 
site, the optimal devices could be either identical, or completely 
different.

:)

adam.



Sent via my BlackBerry® device from Claro

Re: Routers in Data Centers

[Adrian Chadd]

On Sun, Sep 26, 2010, ym1r...@gmail.com wrote:
> As far as I know open source solutions doesn't have support for fabric or 
> high speed asics. So the throughput will always be a big difference. Unless 
> you are comparing a pure packet software interrupt platform.

Hasn't there been a post about this to the contrary?

Isn't someone from Google presenting at NANOG about this?



Adrian

Re: Routers in Data Centers

[Rubens Kuhl]

On Sun, Sep 26, 2010 at 8:54 PM,   wrote:
> As far as I know open source solutions doesn't have support for fabric or 
> high speed asics. So the throughput will always be a big difference. Unless 
> you are comparing a pure packet software interrupt platform.

Not high speed ASICs, but there are hardware-forwarding open-source(in
a broad definition) solutions:
http://netfpga.org

There are 3 related presentations on NANOG 50, which suggests these
solutions are reaching "real ops" quality.

Rubens

Re: Routers in Data Centers

[Adrian Chadd]

On Sun, Sep 26, 2010, Rubens Kuhl wrote:

> Not high speed ASICs, but there are hardware-forwarding open-source(in
> a broad definition) solutions:
> http://netfpga.org
> 
> There are 3 related presentations on NANOG 50, which suggests these
> solutions are reaching "real ops" quality.

I hate to sound (more) like a broken record but if people want
to see open source hardware forwarding platforms succeeding
(and the software platforms get better), then look at trying to be
involved in their development.

Too many companies seem to think open source equates to "free stuff
that I can use and not pay for"; rather than thinking of it as
a normal product (with development cycles, resources, etc that any
commercial development requires)  that gives them the ability to
choose their own direction rather than be beholden to the whims
of a vendor.

One of the fun divides in open source at times is the big gap between
"works" and "works in practice". The only way to get "ops ready" stuff
is to work with open source people to make it actually work in your
environment rather than "what works for them". :-)

(Or you could wait for Google - but doesn't that make you beholden
to them as your vendor? :)


Adrian

-- 
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $24/pm+GST entry-level VPSes w/ capped bandwidth charges available in WA -

Re: Routers in Data Centers

[Heath Jones]

I'm more than interested in developing a much cheaper, hardware
forwarding router..
I think there is a lot of room for innovation - especially at the
target market in this thread.
If anyone wants to work with me on this, just let me know!
I've got a tonne of ideas and a bit of free time..

NetFPGA is a good platform, im saving my pennies to buy one and do
some development.
Its only a 4 port device, so not a device you would really use in
production however.


> I hate to sound (more) like a broken record but if people want
> to see open source hardware forwarding platforms succeeding
> (and the software platforms get better), then look at trying to be
> involved in their development.

RE: Routers in Data Centers

[Alex Rubenstein]

> 
> I'm not saying the problems are the same, but I am saying that a
> backplane making cooling "hard" is not a good excuse, especially when
> the small empty chassis costs $10K+.


And, not to mention that some vendors do it sometimes.

"The 9-slot Cisco Catalyst 6509 Enhanced Vertical Switch (6509-V-E) provides 
[stuff]. It also provides front-to-back airflow that is optimized for hot and 
cold aisle designs in colocated data center and service provider deployments 
and is compliant with Network Equipment Building Standards (NEBS) deployments."

It only took 298 years from the inception of the 6509 to get a front-to-back 
version. If you can do it with that oversized thing, it certainly can be done 
on a 7200, XMR, juniper whatever, or whatever else you fancy.

There is no good excuse. The datacenter of today (and yesterday) really needs 
front to back cooling; the datacenter of tomorrow requires and demands it.

If vendors cared, they'd do it. Problem is, there is a disconnect between 
datacenter designer, datacenter builder, datacenter operator, IT operator, and 
IT manufacturer. No one is smart enough, yet, to say, "if you want to put that 
hunk of crap in my datacenter, it needs to suck in the front and put out in the 
back, otherwise my PUE will be 1.3 instead of 1.2 and you will be to blame for 
my oversized utility bills."

Perhaps when a bean-counter paying the power bill sees the difference, it will 
matter. I dunno.

I'll crawl back under my rock now.

Re: Routers in Data Centers

[Ingo Flaschberger]

I'm more than interested in developing a much cheaper, hardware
forwarding router..
I think there is a lot of room for innovation - especially at the
target market in this thread.
If anyone wants to work with me on this, just let me know!
I've got a tonne of ideas and a bit of free time..

NetFPGA is a good platform, im saving my pennies to buy one and do
some development.
Its only a 4 port device, so not a device you would really use in
production however.


But it seems, that NetFPGA has not enough memory to hold a full view 
(current 340k routes).

Re: Routers in Data Centers

[Richard A Steenbergen]

On Sun, Sep 26, 2010 at 09:24:54PM -0400, Alex Rubenstein wrote:
> 
> And, not to mention that some vendors do it sometimes.
> 
> "The 9-slot Cisco Catalyst 6509 Enhanced Vertical Switch (6509-V-E) 
> provides [stuff]. It also provides front-to-back airflow that is 
> optimized for hot and cold aisle designs in colocated data center and 
> service provider deployments and is compliant with Network Equipment 
> Building Standards (NEBS) deployments."

A classic 6509 is under 15U, a 6509-V-E is 21U. Anyone can do front to 
back airflow if they're willing to bloat the size of the chassis (in 
this case by 40%) to do all the fans and baffling, but then you'd have 
people whining about the size of the box. :)

> It only took 298 years from the inception of the 6509 to get a 
> front-to-back version. If you can do it with that oversized thing, it 
> certainly can be done on a 7200, XMR, juniper whatever, or whatever 
> else you fancy.

Well, a lot of people who buy 7200s, baby XMRs, etc, are doing it for 
the size. Lord knows I certainly bought enough 7606s instead of 6509s 
over the years for that very reason. I'm sure the vendors prefer to 
optimize the size footprint on the smaller boxes, and only do front to 
back airflow on the boxes with large thermal loads (like all the modern 
16+ slot chassis that are rapidly approaching 800W/card). Also, remember 
the 6509 has been around since its 9 slots were lucky to see 100W/card, 
which is a far cry from a box loaded with 6716s at 400W/card or other 
power hungry configs.

Remember the original XMR 32 chassis, which had side to side airflow? 
They quickly disappeared that sucker and replaced it with the much 
larger version they have today, I can only imagine how bad that was. :)

-- 
Richard A Steenbergenhttp://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Re: Software-based Border Router

[Chris Adams]

Once upon a time, William Herrin  said:
> Quagga on Linux is a fine software, but messing with the
> idiosyncrasies is far more time consuming than buying a Cisco 2811,
> adding enough RAM to handle BGP, configuring it once and forgetting
> about it.

Yeah, because IOS and JUNOS don't have idiosyncrasies. :-)
-- 
Chris Adams 
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: Routers in Data Centers

[Christian Martin]

On Sep 26, 2010, at 10:29 PM, Richard A Steenbergen  wrote:

> On Sun, Sep 26, 2010 at 09:24:54PM -0400, Alex Rubenstein wrote:
>> 
>> And, not to mention that some vendors do it sometimes.
>> 
>> "The 9-slot Cisco Catalyst 6509 Enhanced Vertical Switch (6509-V-E) 
>> provides [stuff]. It also provides front-to-back airflow that is 
>> optimized for hot and cold aisle designs in colocated data center and 
>> service provider deployments and is compliant with Network Equipment 
>> Building Standards (NEBS) deployments."
> 
> A classic 6509 is under 15U, a 6509-V-E is 21U. Anyone can do front to 
> back airflow if they're willing to bloat the size of the chassis (in 
> this case by 40%) to do all the fans and baffling, but then you'd have 
> people whining about the size of the box. :)

I would point out that it is quite possible to build a compact (say 4-5 U) 
front-back airflow platform if vendors were willing to pay to engineer a small 
midplane and leverage modular I/O cards in a single vertical arrangement.  As 
an example, envisage an M10i with 8 single height PIC slots, rear mounted 
RE/PFE combos, and a top/bottom impeller.  Same for a 72/73xx, or whatever 
platform you fancy.  But would it make business sense?  You'd lose rack space 
in favor of thermal efficiency.  

I think the push toward cloud computing and the re-emergence of big datacenters 
with far more stringent power and heat restrictions may actually drive such a 
move.   I guess we'll see...


C

> 
>> It only took 298 years from the inception of the 6509 to get a 
>> front-to-back version. If you can do it with that oversized thing, it 
>> certainly can be done on a 7200, XMR, juniper whatever, or whatever 
>> else you fancy.
> 
> Well, a lot of people who buy 7200s, baby XMRs, etc, are doing it for 
> the size. Lord knows I certainly bought enough 7606s instead of 6509s 
> over the years for that very reason. I'm sure the vendors prefer to 
> optimize the size footprint on the smaller boxes, and only do front to 
> back airflow on the boxes with large thermal loads (like all the modern 
> 16+ slot chassis that are rapidly approaching 800W/card). Also, remember 
> the 6509 has been around since its 9 slots were lucky to see 100W/card, 
> which is a far cry from a box loaded with 6716s at 400W/card or other 
> power hungry configs.
> 
> Remember the original XMR 32 chassis, which had side to side airflow? 
> They quickly disappeared that sucker and replaced it with the much 
> larger version they have today, I can only imagine how bad that was. :)
> 
> -- 
> Richard A Steenbergenhttp://www.e-gerbil.net/ras
> GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
> 

Re: Routers in Data Centers

[Heath Jones]

> But it seems, that NetFPGA has not enough memory to hold a full view
> (current 340k routes).

It's just a development platform for prototyping designs, not
something you would use in production...
I want to use it to implement and test ideas that I have, and play
with some different forwarding architectures, not use it as a final
product :)

Re: Routers in Data Centers

[Christopher Morrow]

On Sun, Sep 26, 2010 at 11:02 PM, Heath Jones  wrote:
>> But it seems, that NetFPGA has not enough memory to hold a full view
>> (current 340k routes).
>
> It's just a development platform for prototyping designs, not
> something you would use in production...
> I want to use it to implement and test ideas that I have, and play
> with some different forwarding architectures, not use it as a final
> product :)

also, does a datacenter router/switch need a full table? isn't that
the job of the peering/transit routers in your scheme?

Re: Routers in Data Centers

[James P. Ashton]

- Original Message -
On Sun, Sep 26, 2010 at 11:02 PM, Heath Jones  wrote:
>> But it seems, that NetFPGA has not enough memory to hold a full view
>> (current 340k routes).
>
> It's just a development platform for prototyping designs, not
> something you would use in production...
> I want to use it to implement and test ideas that I have, and play
> with some different forwarding architectures, not use it as a final
> product :)

also, does a datacenter router/switch need a full table? isn't that
the job of the peering/transit routers in your scheme?


Sometimes, but often you get odd results when internal gateway routers only see 
a pair of default gateways via OSPF or IS-IS. Sometimes the only real fix is to 
have a full table on these routers as well as your border/peering routers. 

James

RE: Routers in Data Centers

[Simon Lyall]

A few Blog posts on Datacentre network equipment that people may find 
interesting and relivant:



http://perspectives.mvdirona.com/2009/12/19/NetworkingTheLastBastionOfMainframeComputing.aspx

http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_CleanSlateCTO2009.pdf

http://perspectives.mvdirona.com/2010/08/01/EnergyProportionalDatacenterNetworks.aspx




--
Simon Lyall  |  Very Busy  |  Web: http://www.darkmere.gen.nz/
"To stay awake all night adds a day to your life" - Stilgar | eMT.