Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Andy Davidson]

On 23 Jan 2008, at 17:24, Paul Vixie wrote:


[EMAIL PROTECTED] (Andy Davidson) writes:

People pay the RIRs.
The RIRs spend money on parties for network operators.
...
according to   
for 2007 and


for 2006 and 2005, ARIN's expenses are mostly unrelated to partying.



After fielding plenty of offlist mail - I'm sorry that my tongue in  
cheek comments caused some raised eyebrows; it was a non-serious  
suggestion for how the RIRs could spend an imaginary windfall, not a  
comment that they spend too much on socials.  I recognise the value of  
conferencing, and the importance of injecting some socials at  
conference events since it's really good opportunity for delegates to  
spend time talking to others in the industry.


Andy

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Paul Vixie]

[EMAIL PROTECTED] (Andy Davidson) writes:

> People pay the RIRs.
> 
> The RIRs spend money on parties for network operators.
> ...

according to  for 2007 and

for 2006 and 2005, ARIN's expenses are mostly unrelated to partying.
-- 
Paul Vixie

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Mikael Abrahamsson]

On Wed, 23 Jan 2008, Andy Davidson wrote:

I think that charging for deaggregation of PA is hard to imagine.  I 
think charging for PI as a model may have been worthy of consideration 
several years ago, but since we're only months away from entire product 
lines of deployed edge kit nolonger accepting a full table, the battle 
is over (and operators lost).


As far as I can see, the only way to solve de-aggregation and PI is to 
create some kind of cryptographic signing of aggregate routes sent out to 
DFZ.


RIPE/ARIN and other equal instances need to sign the combination of AS 
and prefix, and this is then used (somehow) to authenticate the prefix 
when received. This would also have the added benefit of stopping people 
from sending more specifics with other ISPs IP space (or even their own, 
as only the actual aggregate prefix would be signed, not more specifics 
that people use for "TE").


So this "certificate" or alike needs to be time limited and coupled to 
payment if we're going to charge for PI/PA yearly.


Yes, this increases complexity in the DFZ enormously, and I don't know if 
the benefit outweighs the complexity and added risks for failures.


--
Mikael Abrahamssonemail: [EMAIL PROTECTED]

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Andy Davidson]

On 22 Jan 2008, at 17:30, Michael K. Smith - Adhost wrote:

Hmm, who gets paid?  It sounds like your hinting around a telco-type  
reciprocal payment model (correct me if I'm wrong).  Do I pay my  
upstreams who in turn pay there upstreams and so on and so on?  Or,  
is there some central, uber-authority that gets paid by all of us?   
It seems to me that there are many billing models that accommodate  
point-to-point relationships, but I'm having a hard time coming up  
with a mental model of payment in the many-to-many environment in  
the DFZ.


People pay the RIRs.

The RIRs spend money on parties for network operators.


I think that charging for deaggregation of PA is hard to imagine.  I  
think charging for PI as a model may have been worthy of consideration  
several years ago, but since we're only months away from entire  
product lines of deployed edge kit nolonger accepting a full table,  
the battle is over (and operators lost).


Andy

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[William Herrin]

On Jan 22, 2008 1:58 PM, Jon Lewis <[EMAIL PROTECTED]> wrote:
> Giving absolutely anyone who wants it PI space would make things much
> worse...so I wouldn't call that artificial supression.  It's more like
> keeping the model sustainable.

Jon,

Its kinda like gas in the 70's. There wasn't enough to go around given
the price controls so the only way to keep the consumption model
sustainable was with rationing.

Except history tells us that was kinda dumb. Had we allowed prices to
adjust (as we're doing today) the market would have taken care of
itself. Gas would have tripled in price and more folks would have
taken the bus but you wouldn't have the entire nation waiting in line
at the pump.

Right now, announcing a prefix is close to free. If my numbers are
right, it actually costs around $8k/year. A sustainable model selling
an $8k product as a free loss-leader requires some pretty harsh
rationing. Which is precisely what ARIN does, at our insistence.


> I think you mean "get paid for accepting prefixes" or perhaps "pay into
> some global pool (for redistribution to the participants) to announce
> prefixes".

That's right.

The vendors are currently delivering routers that can handle 1M
prefixes and they promise that they can build routers that handle 10M
prefixes with today's technology if there's a demand. If folks want to
pay us more than it costs to dump another 700k prefixes into the
table, why shouldn't we take the profit? It's free money. Even at $8k
there are folks willing to pay it. All it requires is a market
structure that makes the transaction possible.

Let's engage our imaginations, roll with your "global pool" model for
a moment and see if anything interesting pops out of the box.

So, ARIN starts assigning addresses down to a /28 level. The only
requirement for a prefix longer than /20 is that they must remain in
continuous use on the Internet or they'll be revoked

Then, NRO sets up a Universal Service Fund for DFZ routing. Any
transit AS which agrees to accept and normally propagate exactly the
prefixes that are subscribed to the USF is entitled to receive monthly
payments from the USF based on some formula including that AS's
backbone speeds, number of routers and number of peers. At the other
side, anyone who wants their routes carried can make a fixed
contribution to the USF for each prefix that they want to announce,
all the way down to a /32. That only entitles them to announce exactly
that one prefix. If they want to disaggregate, they have to pay for
each of the deaggregates separately.

So now you have folks who can only justify a /28 but its worth
$8k/year to their business to have PI space so that there are no
renumbering costs. And the best part is that they're paying you for
the privilege, paying more than it costs, instead of you having to
blandly accept those prefixes for free.

But wait, it gets better. Now that there's a market structure in
place, its possible to envision different classes of service.

Maybe this market holds a niche for folks who don't want to pay
$8k/year into the USF. Suppose ARIN auctions off the right to announce
a "covering /8" for each of its IANA allocations. The winner can't use
any of the addresses for itself, but it has the right to sell tunnels
to the folks with more specific prefixes. So, if you're a small fry,
maybe you don't pay $8k/year into the USF. Instead, you pay $500/year
each to the two backbones closest to you and then you pay another
$1000/year to the tunnel provider whose /8 covers your prefix. Your
ISP gives you one address from their PA space to catch the endpoint of
that tunnel and for $2k/year you're in business with PI space. If you
picked your backbones right, there's even a decent chance that traffic
following the /8 usually wanders into one of them and redirects your
way before hitting the tunnel.

And suddenly, surprisingly, the Internet works better than ever
without everyone having to carry full routes, you get PAID for the
prefixes you do carry and everyone who wants PI or lots of TE can have
it! Its not free any more, but you can have anything you're willing to
pay for without having to justify yourself to the rationing board.


> Good luck on that one. In how many languages can you say "not gonna happen"?

Do programming languages count? $paiddfz=!$happen;

Seriously, the goal may seem unachievable but that doesn't mean it's
not worth striving for. Who knows what we may find on the way?


On Jan 22, 2008 11:35 AM, Bill Woodcock <[EMAIL PROTECTED]> wrote:
> instead of eating less, we GET PAID for eating tasty
> sammitches!

I would love to get paid for eating tasty sammitches. How cool a job
would that be!


Regards,
Bill Herrin


-- 
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: 
Falls Church, VA 22042-3004

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Jon Lewis]

On Tue, 22 Jan 2008, William Herrin wrote:


Right now we rely on ARIN and the RIRs to artificially suppress the
growth of the prefix count and with it the availability of PI space.


If by artificially suppress, you mean anyone who wants it can't just fill 
out a form and be handed a portable /24 or other small CIDR sure...but you 
don't need PI space to multihome or increase the size of the global table. 
Giving absolutely anyone who wants it PI space would make things much 
worse...so I wouldn't call that artificial supression.  It's more like 
keeping the model sustainable.



If we can determine the cost to announce a prefix then we could
develop a market-based solution to the problem... One where instead of
suppressing the prefix count and dealing with it as business overhead,
we GET PAID for announcing and propagating prefixes.


I think you mean "get paid for accepting prefixes" or perhaps "pay into 
some global pool (for redistribution to the participants) to announce 
prefixes".


Good luck on that one. In how many languages can you say "not gonna 
happen"?


--
 Jon Lewis   |  I route
 Senior Network Engineer |  therefore you are
 Atlantic Net|
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

RE: Cost per prefix [was: request for help w/ ATT and terminology]

[Michael K. Smith - Adhost]

Hello Bill:

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
> William Herrin
> Sent: Tuesday, January 22, 2008 7:55 AM
> To: nanog@merit.edu
> Subject: Re: Cost per prefix [was: request for help w/ ATT and
> terminology]
> 
> 
> On Jan 21, 2008 10:28 PM, Jon Lewis <[EMAIL PROTECTED]> wrote:
> > Is there really any point in trying to put a $ figure on each route?
> 
> Jon,
> 
> Emphatically Yes!
> 
> Right now we rely on ARIN and the RIRs to artificially suppress the
> growth of the prefix count and with it the availability of PI space.
> This is a Really Bad Thing on so many levels, but absent a viable
> market-based solution to the problem, authority-based rationing is
> really the only thing we can do.
> 
> If we can determine the cost to announce a prefix then we could
> develop a market-based solution to the problem... One where instead of
> suppressing the prefix count and dealing with it as business overhead,
> we GET PAID for announcing and propagating prefixes.
> 
> There are several market models that could work, but they all depend
> on having a reliable metric for the cost of announcing a prefix. So,
> if you think you'd like to get paid for announcing routes instead of
> continuing to give the service away for free then yes, there is a
> point to determining the cost of a prefix.
> 
Hmm, who gets paid?  It sounds like your hinting around a telco-type reciprocal 
payment model (correct me if I'm wrong).  Do I pay my upstreams who in turn pay 
there upstreams and so on and so on?  Or, is there some central, uber-authority 
that gets paid by all of us?  It seems to me that there are many billing models 
that accommodate point-to-point relationships, but I'm having a hard time 
coming up with a mental model of payment in the many-to-many environment in the 
DFZ.

Regards,

Michael Smith


PGP.sig
Description: PGP signature

RE: Cost per prefix [was: request for help w/ ATT and terminology]

[andrew2]

William Herrin wrote:
> Right now we rely on ARIN and the RIRs to artificially suppress the
> growth of the prefix count and with it the availability of PI space.
> This is a Really Bad Thing on so many levels, but absent a viable
> market-based solution to the problem, authority-based rationing is
> really the only thing we can do.
> 
> If we can determine the cost to announce a prefix then we could
> develop a market-based solution to the problem... One where instead of
> suppressing the prefix count and dealing with it as business overhead,
> we GET PAID for announcing and propagating prefixes.


Hi, I'm Google/Yahoo/Microsoft/AT&T/AOL/Sprint/etc. and I plan to annnounce
only /24's and I refuse to pay you to propagate those routes.  Are you
really going to drop those routes?   Bottom line here is you're going to
have trouble getting the big content providers to buy in, and you're going
to have an equally tough time convincing the major carriers that they should
essentially raise their rates for particular clients.  So who exactly is
going to pay and how are you going to convince them they should?  If
provider X tells me they're going to charge me $X per prefix I want them to
propagate, I'll just go with provider Y.  You're going to need 100% buy-in.

Your solution here is merely a band-aid designed to disguise the actual
problem.  Growing prefix count is largely a symptom of missing BGP
functionality.  Fix or replace BGP in such a way that we can better control
the flow of incoming traffic without needing hacks like announcing smaller
subnets and prepending and the problem goes away without introducing extra
fees and beauracracy like you're suggesting.

Andrew Cruse

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Joe Greco]

> The problem with William's calculation is that he is claiming the  
> _only_ difference between X & Y is "prefix count".  (He said this,  
> more than once.)

The only meaningful difference between X & Y for the purposes of this
discussion _is_ prefix count.

> He is dead wrong.

No, he's quite right.

> I cannot think of a pair of boxes where one can support a full table  
> and one can't where the _only_ difference is prefix count.  

That's missing the point.

Let's say that I am buying a gig Ethernet switch.  I need 24 ports. 
I'm going to limit vendors to Dell, just to keep the options somewhat
limited, just as they are in the world of routing.

If I want an unmanaged switch, I can't get it.  I am forced to buy a
Dell 2724 "web-managed" switch for $279.  That's fine, for me, it's
still essentially an unmanaged switch, as I won't be willing/able to 
run Internet Explorer to manage it.

Now, if I want to include the ability to ssh into it and do interactive
stuff, such as scripting, then I'm looking at a different device.  I'm
forced to buy a Dell 5424 "managed" switch for $912.

The 5424 can do a *LOT* more than the 2724.  However, the single reason
*I* needed to pick the 5424 was due to ssh, and so even if the 5424 can
be managed via SNMP and do other neat stuff, those features mean nothing
to me, and the cost to me to obtain ssh as a feature is a bit over $600.
It's not $100 for ssh and $100 for SNMP and $100 each for each of four
other features.

> (I am  
> excluding things like "Box X with 128MB RAM and Box X with 1 GB  
> RAM".)

We know that will work for certain networks.  It is fair to establish 
that as the far low-end for calculating the incremental cost.

> Even the Sup2 & 3blx have far more differences than just  
> prefix count.  If you do not understand why, you clearly aren't  
> competent to post to this thread.

Of course they have far more differences than just prefix count.  That's
the way feature sets work.  I can't /buy/ a sup2-3bxl.  However, if I'm 
running a network today on a sup2, and I'm fine, EXCEPT that I am 
running out of prefixes, what do I do?

What DO I do?

If I choose to upgrade to the 3bxl, the _only_ meaningful difference 
between before and after is the prefix count.  The 3bxl may have some 
nifty other features, but I was getting along fine without them, and
even once I've upgraded, I'm just as likely to not be using them, so
the entire cost of the 3bxl upgrade can be attributed to the reason I
did the upgrade: support for the larger table size.

> For every network, there is equipment that will work, and equipment  
> that will not.  At the top end for very large networks, buying less  
> than a CRS1 / T640 is simply not an option.  And the amount of  
> engineering going into those boxes to deal with a 1M BGP entry table  
> is essentially _zero_.  Many networks buying those boxes have 100s of  
> 1000s of prefixes in their _IGP_, so the FIB / processor / RAM / etc.  
> all have to deal.

Yup.  Those may be balanced in some fashion by:

> Near the bottom end, for networks that could get away with a 3750 if  
> they only supported a full table (which I submit is a pretty small  
> fraction of the total boxes sold), they can still buy the 3750 and  
> default to a pair of cheap 2/3/4-port boxes with a gig of RAM and use  
> those for external connectivity.  The problem is perfectly solvable  
> without jumping to the 7600/3blx.

It is possible to cobble together a solution of some sort.  However,
that is equally likely to be inconvenient or exceedingly difficult,
based on the specifics of the network.  You seem to be aware of that:

> In between there are more variations than everyone here combined could  
> possibly imagine.  And the requirements are _NOT_ all about prefix  
> count.  Which means you have no basis for comparison.  If you try to  
> force a general comparison, you will be wrong.
> 
> So stop asking "what box would you compare?"  The answer is "whatever  
> I need!", not what YOU think is correct, since you are invariably  
> wrong unless you know everything about MY network.

Sure.  SO, for the purposes of this thread, the reasonable thing to do
is to try to do as direct a comparison as possible.  We want to know how
much extra the full table capability is.  While specific answers may vary
from provider to provider, it should be obvious that there's a cost of
some sort, and that for a typical bit of hardware, such as the 6500/7600,
it's the cost of the upgrade to 3bxl, plus whatever other upgrades you
might be forced to eat.

That's not to say that someone, somewhere, won't be replacing a 6500 with
a bunch of Linux PC's and a cheap switch, or some other cobbled-together
solution, and yes, then the answers change.  Patrick, this thread isn't
about the canonical way for a provider to upgrade, it's not even about a
real provider.  It's about the costs to a hypothetical provider.  The 
design and facts of your network are not particularly relevan

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Bill Woodcock]

Right now we rely on ARIN and the RIRs to artificially suppress the
growth of the prefix count and with it the availability of PI space.
If we can determine the cost to announce a prefix then we could
develop a market-based solution to the problem... instead of
suppressing the prefix count, we GET PAID for announcing and
propagating prefixes.



Right now, we rely on our appetites to tell us when we're full, and  
should stop eating tasty sammitches.  If we can determine the cost of  
obesity, then we could develop a market-based solution to the  
problem...  instead of eating less, we GET PAID for eating tasty  
sammitches!


Not sure this one holds together, when viewed at the macro-level.   
Note that this is my first posting of the morning, is probably  
crustier than average, and should most likely be ignored.


-Bill






PGP.sig
Description: This is a digitally signed message part

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[William Herrin]

On Jan 21, 2008 10:28 PM, Jon Lewis <[EMAIL PROTECTED]> wrote:
> Is there really any point in trying to put a $ figure on each route?

Jon,

Emphatically Yes!

Right now we rely on ARIN and the RIRs to artificially suppress the
growth of the prefix count and with it the availability of PI space.
This is a Really Bad Thing on so many levels, but absent a viable
market-based solution to the problem, authority-based rationing is
really the only thing we can do.

If we can determine the cost to announce a prefix then we could
develop a market-based solution to the problem... One where instead of
suppressing the prefix count and dealing with it as business overhead,
we GET PAID for announcing and propagating prefixes.

There are several market models that could work, but they all depend
on having a reliable metric for the cost of announcing a prefix. So,
if you think you'd like to get paid for announcing routes instead of
continuing to give the service away for free then yes, there is a
point to determining the cost of a prefix.



On Jan 21, 2008 6:14 PM, David Barak <[EMAIL PROTECTED]> wrote:
> Wouldn't a reasonable approach be to take the sum of a 6500/msfc2 and a 2851,
> and assume that the routing computation could be offloaded?

David,

No, because the FIB can't be offloaded; it has to sit on the router's
forwarding plane and it has to keep up. Between the low single-digit
gbps and the low double-digit gbps, equipment which both keeps up and
supports a large prefix count is significantly more expensive than
equipment which keeps up but does not support a large prefix count.


> The difficulty I have with this discussion is that the cost per prefix is 
> zero until you need to
> change eigenstate, where there's a big cost, and then it goes back to zero 
> again.

This is one of the harder aspects of operations cost analysis to wrap
your head around. Not only isn't the operations cost attributable to a
particular feature set bound to the associated manufacturing cost,
it's different for different scenarios in which the equipment is used.
To accurately assess the cost, you have to identify the boundary
conditions that drive equipment selection.

Let me construct two scenarios which illustrate this:

Scenario A: You have 1U LAN switches serving 500 PCs at 100mbps and a
few dozen servers at 1gbps. You want to upgrade to 1gbps to the PCs
and 10gbps to the servers. You replace the 1U switches with
7600/sup720s.

The cost driver in this scenario is the 10gbps server links. You can
get 1U switches that stack and backbone at 10gbps but to hook up all
the servers at 10gbps (to support the 1gbps workstations) you have to
step up to the next class of equipment. This is the sole reason you
need a 7600 instead of a few 1U gig-e switches. Therefore the 100% of
difference in cost between a 7600 and a few 1U gig-e switches is
attributable to the forwarding capacity required by the servers.

In scenario A, the 7600's much larger prefix carrying capacity is not
a cost driver. It plays no role in the decision to purchase a 7600
instead of 1U switches. As a result, the operations cost attributable
to the 7600's larger prefix capacity is exactly zero.


Scenario B: You have a metro ethernet POP moving 2 gbps of traffic
with a couple 1U fiber switches. Use is gradually increasing; you
project that 12 months out it will be 2.5gbps. For business reasons
you want to extend the DFZ to this POP. You replace the 1U switches
with a 7600/sup720-3bxl.

The cost driver in this scenario is the DFZ extension, specifically
the prefix count in the DFZ. At 2gbps, your existing equipment is not
even close to tapped out, but it can't even think about carrying the
DFZ's prefixes in its FIB. The sole reason you need a 7600 instead of
a few 1U gig-e switches is the prefix count. Therefore the 100% of
difference in cost between a 7600 and a few 1U gig-e switches is
attributable to the prefix count required by the DFZ.


Almost the exact same upgrade, but in one scenario the cost is
attributable to the forwarding capacity and in the other its
attributable to the prefix count.

I think this is what's giving Patrick the most trouble as well. He
wants equipment cost at an operations level to map to the component
manufacturing cost as common sense says it should, but it doesn't work
that way.

Regards,
Bill Herrin



--
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: 
Falls Church, VA 22042-3004

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Patrick W. Gilmore]

On Jan 21, 2008, at 6:14 PM, David Barak wrote:

Wouldn't a reasonable approach be to take the sum of a 6500/ 
msfc2 and a 2851, and assume that the routing computation could be  
offloaded?


The difficulty I have with this discussion is that the cost per  
prefix is zero until you need to change eigenstate, where  
there's a big cost, and then it goes back to zero again.


Because this isn't really all that new a problem, most vendors  
try not to make devices which have no headroom at all - so kit in  
the lower category seems to be qualitatively different.


We have a winner.

The problem with William's calculation is that he is claiming the  
_only_ difference between X & Y is "prefix count".  (He said this,  
more than once.)


He is dead wrong.

I cannot think of a pair of boxes where one can support a full table  
and one can't where the _only_ difference is prefix count.  (I am  
excluding things like "Box X with 128MB RAM and Box X with 1 GB  
RAM".)  Even the Sup2 & 3blx have far more differences than just  
prefix count.  If you do not understand why, you clearly aren't  
competent to post to this thread.



For every network, there is equipment that will work, and equipment  
that will not.  At the top end for very large networks, buying less  
than a CRS1 / T640 is simply not an option.  And the amount of  
engineering going into those boxes to deal with a 1M BGP entry table  
is essentially _zero_.  Many networks buying those boxes have 100s of  
1000s of prefixes in their _IGP_, so the FIB / processor / RAM / etc.  
all have to deal.


Near the bottom end, for networks that could get away with a 3750 if  
they only supported a full table (which I submit is a pretty small  
fraction of the total boxes sold), they can still buy the 3750 and  
default to a pair of cheap 2/3/4-port boxes with a gig of RAM and use  
those for external connectivity.  The problem is perfectly solvable  
without jumping to the 7600/3blx.


In between there are more variations than everyone here combined could  
possibly imagine.  And the requirements are _NOT_ all about prefix  
count.  Which means you have no basis for comparison.  If you try to  
force a general comparison, you will be wrong.


So stop asking "what box would you compare?"  The answer is "whatever  
I need!", not what YOU think is correct, since you are invariably  
wrong unless you know everything about MY network.



Anyone who thinks differently is either confused or has an agenda they  
are pushing.  Or, possibly, trying to sell you a bigger box.



PLEASE, let the thread die.

--
TTFN,
patrick

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Jon Lewis]

On Mon, 21 Jan 2008, William Herrin wrote:


Hmm. Well, the secondary market is flooded with sup2's right now, with
the card at sub-$1k prices and with a 6500+sup2 in the $5k range.
There isn't really a comparable availability of the sup720-3bxl
although eBay does have a few listed in the $12k range. If we take


I started to get into this in the last message and decided not to...but 
another problem with these comparisons is the 'going rate' for Sup2s
is very likely depressed considerably due to their no longer being 
suitable for full BGP table applications.  Go back a couple years, and 
they were quite a bit more $...probably closer to $10k.


Another is that networks having to upgrade now already bought whatever 
they're upgrading from (i.e. Sup2s) some time ago at prices similar to 
what the 3bxls go for now.  So they're not just having to spend more or 
the difference...they're having to nearly double their investment in full 
table routers (some parts such as the chassis and line cards will likely 
remain in service).



I wouldn't want to stand behind those numbers, though. I'm not sure
what the error band is, but it has to be huge. The equivalence in the
secondary market just isn't there. Nor can we use $12k as the baseline
price for the sup720-3bxl. There isn't wide availability at that
price, just a few sketchy sellers from Hong Kong.


I wonder if those are being faked yet?

Is there really any point in trying to put a $ figure on each route? 
Common sense should tell us that polluting the DFZ will eventually cost 
every network wanting/needing to participate real money (thousands for 
smaller networks, hundreds of thousands or millions for larger 
networks/backbones)...so we really ought to be putting effort into 
education rather than crunching theoretical numbers to determine exactly 
how many french fries each route equates to.


I know from past experience, that ARIN can step in and 'use their 
influence' to get transit providers to not accept routes they don't think 
an ASN should be announcing (acquisition that went way south).  I don't 
know what sort of yearly cash flow surpluses the other RIRs have, but IIRC 
ARIN is doing quite well.  The stats I have from last September suggest 
the 'ARIN region' is the worst as far as longer than RIR minimum routes 
being announced.  Perhaps ARIN could task one or more people with 
examining the ARIN portion of the DFZ, and contacting the networks 
announcing unnecessary deaggregates and when necessary their transit 
providers, for the purpose of educating and when necessary, leaning on 
them to clean up their configs.  Actually, there are already people doing 
the first part of that job for free...so all ARIN would have to do is 
accept & check data that's already been researched and pluck the ARIN 
member networks from it.


I know, BGP Police is not part of ARIN's mission...but they have the $ to 
put people on the job and the influence to perhaps get people to pay 
attention.  In my limited experience trying that role, I found networks 
were totally uninterested in cleaning up and it wasn't even possible to 
get directly in touch with someone who'd understand the issue much less 
have access to do anything about it.


--
 Jon Lewis   |  I route
 Senior Network Engineer |  therefore you are
 Atlantic Net|
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[William Herrin]

On Jan 21, 2008 5:26 PM, Jon Lewis <[EMAIL PROTECTED]> wrote:
> If using the 7600/3bxl as the cost basis of "the upgrade", you might as
> well compare it to the 6500/7600/sup2 or sup3b.  Either of these would
> likely be what people buying the 3bxls are upgrading from, in some cases
> just because of DFZ growth/bloat, in others, to get additional features
> (IPv6).

Hi Jon,

Hmm. Well, the secondary market is flooded with sup2's right now, with
the card at sub-$1k prices and with a 6500+sup2 in the $5k range.
There isn't really a comparable availability of the sup720-3bxl
although eBay does have a few listed in the $12k range. If we take
$5k-$1k=$4k for the chassis/ps/etc and compare $16k versus $5k for a
6500/sup720-3bxl versus a 6500/sup2 we get just shy of 70%
attributable to the prefix carrying capacity. That's essentially the
same number I came up with before.

I wouldn't want to stand behind those numbers, though. I'm not sure
what the error band is, but it has to be huge. The equivalence in the
secondary market just isn't there. Nor can we use $12k as the baseline
price for the sup720-3bxl. There isn't wide availability at that
price, just a few sketchy sellers from Hong Kong.

Regards,
Bill Herrin


-- 
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: 
Falls Church, VA 22042-3004

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[David Barak]

Wouldn't a reasonable approach be to take the sum of a 6500/msfc2 and a 
2851, and assume that the routing computation could be offloaded?

The difficulty I have with this discussion is that the cost per prefix is zero 
until you need to change eigenstate, where there's a big cost, and then it 
goes back to zero again. 

Because this isn't really all that new a problem, most vendors try not to 
make devices which have no headroom at all - so kit in the lower category seems 
to be qualitatively different.
-David

Joe Greco wrote: 
>> On Mon, 21 Jan 2008, Joe Greco wrote:
>> > Given that the 3750 is not acceptable, then what exactly would you propose
>> > for a 48 port multigigabit router, capable of wirespeed, that does /not/
>> > hold a 300K+ prefix table?  All we need is a model number and a price, and
>> > then we can substitute it into the pricing questions previously posed.
>> >
>> > If you disagree that the 7600/3bxl is a good choice for the fully-capable
>> > router, feel free to change that too.  I don't really care, I just want to
>> > see the cost difference between DFZ-capable and non-DFZ-capable on stuff
>> > that have similar features in other ways.
>> 
>> If using the 7600/3bxl as the cost basis of "the upgrade", you might as 
>> well compare it to the 6500/7600/sup2 or sup3b.  Either of these would 
>> likely be what people buying the 3bxls are upgrading from, in some cases 
>> just because of DFZ growth/bloat, in others, to get additional features 
>> (IPv6).
> I see a minor problem with that in that if I don't actually need a chassis
> as large as the 6500/sup2, there's a bit of a hefty jump to get to that
> platform from potentially reasonable lesser platforms.  If you're upgrading,
> though, it's essentially a discard of the sup2 (because you lose access to
> the chassis), so it may be fair to count the entire cost of the sup720-3bxl.
> Punching in 720-3bxl to Froogle comes up with $29K.  Since there are other
> costs that may be associated with the upgrade (daughterboards, incompatible
> line cards, etc), let's just pretend $30K is a reasonable figure, unless
> someone else has Figures To Share.
> ... JG
> -- 
> Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
> "We call it the 'one bite at the apple' rule. Give me one chance [and] then I
> won't contact you again." - Direct Marketing Ass'n position on e-mail 
> spam(CNN)
> With 24 million small businesses in the US alone, that's way too many apples.



  

Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ 

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Joe Greco]

> On Mon, 21 Jan 2008, Joe Greco wrote:
> > Given that the 3750 is not acceptable, then what exactly would you propose
> > for a 48 port multigigabit router, capable of wirespeed, that does /not/
> > hold a 300K+ prefix table?  All we need is a model number and a price, and
> > then we can substitute it into the pricing questions previously posed.
> >
> > If you disagree that the 7600/3bxl is a good choice for the fully-capable
> > router, feel free to change that too.  I don't really care, I just want to
> > see the cost difference between DFZ-capable and non-DFZ-capable on stuff
> > that have similar features in other ways.
> 
> If using the 7600/3bxl as the cost basis of "the upgrade", you might as 
> well compare it to the 6500/7600/sup2 or sup3b.  Either of these would 
> likely be what people buying the 3bxls are upgrading from, in some cases 
> just because of DFZ growth/bloat, in others, to get additional features 
> (IPv6).

I see a minor problem with that in that if I don't actually need a chassis
as large as the 6500/sup2, there's a bit of a hefty jump to get to that
platform from potentially reasonable lesser platforms.  If you're upgrading,
though, it's essentially a discard of the sup2 (because you lose access to
the chassis), so it may be fair to count the entire cost of the sup720-3bxl.

Punching in 720-3bxl to Froogle comes up with $29K.  Since there are other
costs that may be associated with the upgrade (daughterboards, incompatible
line cards, etc), let's just pretend $30K is a reasonable figure, unless
someone else has Figures To Share.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Jon Lewis]

On Mon, 21 Jan 2008, Joe Greco wrote:


Given that the 3750 is not acceptable, then what exactly would you propose
for a 48 port multigigabit router, capable of wirespeed, that does /not/
hold a 300K+ prefix table?  All we need is a model number and a price, and
then we can substitute it into the pricing questions previously posed.

If you disagree that the 7600/3bxl is a good choice for the fully-capable
router, feel free to change that too.  I don't really care, I just want to
see the cost difference between DFZ-capable and non-DFZ-capable on stuff
that have similar features in other ways.


If using the 7600/3bxl as the cost basis of "the upgrade", you might as 
well compare it to the 6500/7600/sup2 or sup3b.  Either of these would 
likely be what people buying the 3bxls are upgrading from, in some cases 
just because of DFZ growth/bloat, in others, to get additional features 
(IPv6).


--
 Jon Lewis   |  I route
 Senior Network Engineer |  therefore you are
 Atlantic Net|
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Joe Greco]

> > For example, the Cisco 3750G has all of features except for the
> > ability to hold 300k+  prefixes. Per CDW, the 48-port version costs
> > $10k, so the difference (ergo cost attributable to prefix count) is
> > $40k-$10k=$30k, or 75%.
> 
> Unfortunately, I have to run real packets through a real router in the  
> real world, not design a network off CDW's website.
> 
> As a simple for-instance, taking just a few thousand routes on the  
> 3750 and trying to do multipath over, say 4xGigE, the 'router' will  
> fail and you will see up to 50% packet loss.  This is not something I  
> got off CDW's website, this is something we saw in production.
> 
> And that's without ACLs, NetFlow, 100s of peering sessions, etc.  None  
> of which the 3750 can do and still pass gigabits of traffic through a  
> layer 3 decision matrix.

Patrick,

Please excuse me for asking, but you seem to be arguing in a most unusual
manner.  You seem to be saying that the 3750 is not a workable device for
L3 routing (which may simply be a firmware issue, don't know, don't care).
>From the point of finding a 48-port device which could conceivably route
packets at wirespeed, even if it doesn't /actually/ do so, this device 
seems like a reasonable choice for purposes of cost comparisons to me.  
But okay, we'll go your way for a bit.

Given that the 3750 is not acceptable, then what exactly would you propose
for a 48 port multigigabit router, capable of wirespeed, that does /not/
hold a 300K+ prefix table?  All we need is a model number and a price, and
then we can substitute it into the pricing questions previously posed.

If you disagree that the 7600/3bxl is a good choice for the fully-capable
router, feel free to change that too.  I don't really care, I just want to
see the cost difference between DFZ-capable and non-DFZ-capable on stuff
that have similar features in other ways.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

RE: Cost per prefix [was: request for help w/ ATT and terminology]

[Neil J. McRae]

Joe
How many E1 customers can I plug into that device? You can take a cross section 
on this issue at various points.

Think of the huge (sub RSP8) 7500 estates still in production, GSR Engine <5, 
7609 
Sent: 21 January 2008 00:15
To: William Herrin <[EMAIL PROTECTED]>
Cc: Patrick W. Gilmore <[EMAIL PROTECTED]>; nanog@merit.edu
Subject: Re: Cost per prefix [was: request for help w/ ATT and terminology]



A new cisco 2851 can be found for under $10k and can take a gig of  
RAM. If your goal is to have fine-grained routing data, and not to  
carry gigs of traffic, that particular router is perfectly adequate.

If you're prepared to consider second-hand equipment (which seems  
fair, since it's not as though the real Internet has no eBay VXRs in  
it) you could get better performance, or lower cost, depending on  
which way you wanted to turn the dial.

Sometimes it's important to appreciate that the network edge is bigger  
than the network core. Just because this kind of equipment wouldn't  
come close to cutting it in a carrier network doesn't mean that they  
aren't perfectly appropriate for a large proportion of deployed  
routers which take a full table.


Joe

RE: Cost per prefix [was: request for help w/ ATT and terminology]

[Neil J. McRae]

I'm sure they will make it work but there a long list of issues using PCs as 
routers that still applies capex goes to opex.

-Original Message-
From: Adrian Chadd <[EMAIL PROTECTED]>
Sent: 21 January 2008 02:23
To: Jeff McAdams <[EMAIL PROTECTED]>
Cc: nanog@merit.edu
Subject: Re: Cost per prefix [was: request for help w/ ATT and terminology]


On Sun, Jan 20, 2008, Jeff McAdams wrote:

> A Linux box (*BSD, pick your poison) running Quagga or similar will do
> the job at an extremely low price point.
> 
> Yeah, again, not gonna want to pass gigs of traffic through it, but the
> same concept does still apply.

I dunno, the *NIXes seem suddenly interested in fixing up their internal
network code to try and handle 10GE stuff "better". You might find that
someone decides to drop in a crazy-looking intel e1000 NIC driver at some
point thats tailored towards forwarding 64 byte frames at gige rate.

The commercial guys hacking on FreeBSD routers/bridges keep telling me
they've done it w/ custom coding on intel NICs on decent-looking PCIe
hardware.  It doesn't seem out of the realm of possibility nowdays.




Adrian

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[William Herrin]

On Jan 20, 2008 9:46 PM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:
>
> On Jan 20, 2008, at 8:46 PM, William Herrin wrote:
>
> > So at this point, the part of my analysis you still dispute is where I
> > claimed that 75% of the $40k cost of an entry-level DFZ router was
> > attributable to its ability to carry the needed prefix count.
>
> As I said before, your calculation is in error.  I very clearly
> explained why, but you threw out my explanation below.

Patrick,

Just to clarify, your "explanation" (which I've clipped again) claims
an error in the source numbers, not an error in the calculation.
Essentially, you've said that when I determined the percentage of an
entry level DFZ router's cost attributable to the prefix count I chose
as my point of comparison a piece of equipment that is not otherwise
functionally equivalent for the DFZ router task. Because an equivalent
piece of equipment would be more expensive, the percentage I found to
be attributable to carrying and using the prefixes was too high.

I disagree, but I acknowledge that you've offered reasonable support
for the claim that 75% is not the correct percentage of the router's
cost attributable to the DFZ prefix count.

So, run with it. Take the analysis you just did and come up with a
justified estimate of the percentage of the cost of a representative
DFZ router which is attributable to its need to carry a full BGP
table. If you think 75% is too high, lets talk about the number you
think is correct and why. Perhaps you feel that only the cost of the
pfc3bxl and msfc3 daughterboards should be attributable to the prefix
count? Whatever. Pick your numbers and justify them as I have. Then
lets plug your number in to the formula and see what we get.

>  Apparently I
> assumed you had knowledge you did not.  Please forgive me for not
> assuming you were ignorant.  I shall not repeat my mistake.

Or, you could just respond with another ad-hominem attack. It won't
advance anyone's understanding of the cost of carrying prefixes in the
DFZ, but it might make you feel superior.


> I'm certain there are networks who would (do?) use 3750s if the v4
> table were the size of the v6 table.  But they tend to be smaller
> networks, with few or no BGP customers, and not much traffic.  No
> 'tier one' network would, and most networks their size would not.
> Most networks half their size would not.

I don't believe that a tier 1's choice of DFZ routers is
representative of the average DFZ router. Their requirements are much
higher than the norm. If you'd like to argue the opposite position,
I'll be very interested to see what numbers you propose for the
representative router cost and the percentage attributable to handling
the prefix count.


> > For cost analysis purposes, you need only consider a true/false
> > condition here:
> > The device supports the required prefix count.
> > The device does not support the required prefix count.
>
> Would that the world were so simple.

In cost analysis as in software development, all complex problems
reduce to a sequence of simple steps.

Regards,
Bill Herrin


-- 
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: 
Falls Church, VA 22042-3004

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Patrick W. Gilmore]

On Jan 20, 2008, at 8:46 PM, William Herrin wrote:


So at this point, the part of my analysis you still dispute is where I
claimed that 75% of the $40k cost of an entry-level DFZ router was
attributable to its ability to carry the needed prefix count.

I didn't ask you to justify what you thought made my assessment of the
attributable cost was wrong, although I'm glad you agree that you
haven't done so. You also haven't adequately explained why the
justification I used to arrive at those numbers is in error.


As I said before, your calculation is in error.  I very clearly  
explained why, but you threw out my explanation below.  Apparently I  
assumed you had knowledge you did not.  Please forgive me for not  
assuming you were ignorant.  I shall not repeat my mistake.




For example, the Cisco 3750G has all of features except for the
ability to hold 300k+  prefixes. Per CDW, the 48-port version costs
$10k, so the difference (ergo cost attributable to prefix count) is
$40k-$10k=$30k, or 75%.


Unfortunately, I have to run real packets through a real router in the  
real world, not design a network off CDW's website.


As a simple for-instance, taking just a few thousand routes on the  
3750 and trying to do multipath over, say 4xGigE, the 'router' will  
fail and you will see up to 50% packet loss.  This is not something I  
got off CDW's website, this is something we saw in production.


And that's without ACLs, NetFlow, 100s of peering sessions, etc.  None  
of which the 3750 can do and still pass gigabits of traffic through a  
layer 3 decision matrix.


Have you ever configured a 3750 to do BGP with a few gigabits running  
through it?  A 7600?  I submit that you probably have not.  And you  
certainly have not done it in any but the most basic of configurations  
or you would not have posted this silliness.




Or you can keep swimming in that river in Egypt. Its up to you.


William, if that's the best flame you can come up with, perhaps I  
should stop reading your posts.


Then again, given you honestly believe the only difference between a  
3750 & a 7600 is the # of prefixes it can take, I'm not sure why I am  
still reading your posts.




On Jan 20, 2008 5:10 PM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:

On Jan 20, 2008, at 3:34 PM, William Herrin wrote:

( [entry level router's cost attributable to prefixes]/[expected
lifespan] ) / [DFZ prefix count]


I notice you cut out the next two sentences:


In short, if the table were 50K prefixes instead of 250K, would these
pieces of equipment be equivalent?  The answer is a blatant "no".



Yes, I did. Its irrelevant to the cost analysis.


We disagree.

What's more, this thing called "the real world" disagrees as well.



The dividing line between the two classes of equipment is in the
8k-16k prefix range. Thus your statement is like saying, "If you drop
the towing weight from 900 lbs to 200lbs, the 1000lb tow cable is
still not functionally equivalent to a 100lb tow cable." That has
something of a high "duh" factor. If you dropped the prefix count to
8k instead of 250k, the two pieces of equipment (virtual chassis
stack, entry level DFZ router) would be equivalent for most DFZ router
scenarios in which an entry-level DFZ router is used.


Drop it to 8K prefixes, then see example above.

Drop it to 2 prefixes (local LAN + default), the two routers are still  
not equivalent.


I'm certain there are networks who would (do?) use 3750s if the v4  
table were the size of the v6 table.  But they tend to be smaller  
networks, with few or no BGP customers, and not much traffic.  No  
'tier one' network would, and most networks their size would not.   
Most networks half their size would not.


Have you ever seen the requirements put on a peering or customer  
aggregation router at a large network?


I really do not have time to explain why if you do not already know.   
I've spent too much time trying to explain it to you already, yet I  
seem to be failing at explanations, and you just send me (pathetic)  
flames back anyway.  Perhaps some other poster can explain it to you.



For cost analysis purposes, you need only consider a true/false  
condition here:

The device supports the required prefix count.
The device does not support the required prefix count.


Would that the world were so simple.

--
TTFN,
patrick

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Adrian Chadd]

On Sun, Jan 20, 2008, Jeff McAdams wrote:

> A Linux box (*BSD, pick your poison) running Quagga or similar will do
> the job at an extremely low price point.
> 
> Yeah, again, not gonna want to pass gigs of traffic through it, but the
> same concept does still apply.

I dunno, the *NIXes seem suddenly interested in fixing up their internal
network code to try and handle 10GE stuff "better". You might find that
someone decides to drop in a crazy-looking intel e1000 NIC driver at some
point thats tailored towards forwarding 64 byte frames at gige rate.

The commercial guys hacking on FreeBSD routers/bridges keep telling me
they've done it w/ custom coding on intel NICs on decent-looking PCIe
hardware.  It doesn't seem out of the realm of possibility nowdays.




Adrian

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Matt Palmer]

On Sun, Jan 20, 2008 at 08:20:36PM -0500, Jeff McAdams wrote:
> Joe Abley wrote:
> > On 20-Jan-2008, at 15:34, William Herrin wrote:
> 
> >> Perhaps your definition of "entry level DFZ router" differs from mine.
> >> I selected a Cisco 7600 w/ sup720-3bxl or rsp720-3xcl as my baseline
> >> for an entry level DFZ router.
> 
> > A new cisco 2851 can be found for under $10k and can take a gig of RAM.
> > If your goal is to have fine-grained routing data, and not to carry gigs
> > of traffic, that particular router is perfectly adequate.
> 
> And to take that concept to its logical extreme.
> 
> A Linux box (*BSD, pick your poison) running Quagga or similar will do
> the job at an extremely low price point.

So if we plug in, say, $2k for the cost of the Linux box, and compare it to
the L3 switch mentioned earlier, each extra prefix saves the Internet around
50c?  

- Matt

-- 
"Ah, the beauty of OSS. Hundreds of volunteers worldwide volunteering their
time inventing and implementing new, exciting ways for software to suck."
-- Toni Lassila, in the Monastery

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[William Herrin]

On Jan 20, 2008 5:10 PM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:
> If we take out the "proper attribution for the cost delta" out of the
> equation and the equipment is still not considered equal, I submit
> your idea of "proper attribution" is, well, not proper.

Patrick,

So at this point, the part of my analysis you still dispute is where I
claimed that 75% of the $40k cost of an entry-level DFZ router was
attributable to its ability to carry the needed prefix count.

I didn't ask you to justify what you thought made my assessment of the
attributable cost was wrong, although I'm glad you agree that you
haven't done so. You also haven't adequately explained why the
justification I used to arrive at those numbers is in error.

I am, however, asking you to propose and justify an alternate pair of numbers:

(A) The router model and price that you believe qualifies as an
entry-level DFZ router which can reasonably be expected to have a
3-year service life in the DFZ if deployed today, and
(B) The percentage of the router's cost which for the typical DFZ
router task is attributable to the prefix count.

(A) is simple: you find a middling-cheap piece of equipment that meets
all the requirements for an entry level DFZ router and look up its
price. You then explain what the key features of that piece of
equipment are that qualify it as an entry-level DFZ router.

For example, with my choice of a Cisco 7600 w/ a sup720-3bxl card, the
features are: multigigabit layer-3 forwarding, BGP, supports the 300k+
prefixes likely to be needed within a 3-year deployment cycle. This
equipment along with an 48-port gig-e card costs around $40k according
to CDW.

(B) is also simple: you find a middling-cheap piece of equipment that
has all of the required features except for support for the large
prefix count. Then you subtract its price from the price you got in A.
The difference is the cost attributable to the prefix count for the
router in (A). The reason that's the attributable cost is that the
presence or absence of that one capability makes the difference
between the cheaper or more expensive device being usable in the given
application, namely as a DFZ router.

For example, the Cisco 3750G has all of features except for the
ability to hold 300k+  prefixes. Per CDW, the 48-port version costs
$10k, so the difference (ergo cost attributable to prefix count) is
$40k-$10k=$30k, or 75%.

This is not the only way to arrive at the cost attributable to the
prefix count for an entry level DFZ router, but it is by far the
easiest to justify.

I put it to you again: if you disagree with my numbers, propose and
justify your own so that we can have a rational discussion about which
justifications make more sense and thus which set of numbers is more
likely to be correct.

Or you can keep swimming in that river in Egypt. Its up to you.


On Jan 20, 2008 5:10 PM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:
> On Jan 20, 2008, at 3:34 PM, William Herrin wrote:
> > ( [entry level router's cost attributable to prefixes]/[expected
> > lifespan] ) / [DFZ prefix count]
>
> I notice you cut out the next two sentences:
>
> 
> In short, if the table were 50K prefixes instead of 250K, would these
> pieces of equipment be equivalent?  The answer is a blatant "no".
> 

Yes, I did. Its irrelevant to the cost analysis.

The dividing line between the two classes of equipment is in the
8k-16k prefix range. Thus your statement is like saying, "If you drop
the towing weight from 900 lbs to 200lbs, the 1000lb tow cable is
still not functionally equivalent to a 100lb tow cable." That has
something of a high "duh" factor. If you dropped the prefix count to
8k instead of 250k, the two pieces of equipment (virtual chassis
stack, entry level DFZ router) would be equivalent for most DFZ router
scenarios in which an entry-level DFZ router is used.

For cost analysis purposes, you need only consider a true/false condition here:
The device supports the required prefix count.
The device does not support the required prefix count.

There is no gray area between the two and its appropriate to pick
middling-low cost members of each group so long as the one from the
"supports" group is a device that actually sees (or is expected to
see) significant use in the DFZ application.


On Jan 20, 2008 7:15 PM, Joe Abley <[EMAIL PROTECTED]> wrote:
> A new cisco 2851 can be found for under $10k and can take a gig of
> RAM. If your goal is to have fine-grained routing data, and not to
> carry gigs of traffic, that particular router is perfectly adequate.

Joe,

For sub-gigabit applications you could also use Linux+Quagga on a $3k
server. However, neither of these observations provide a valid data
point for the given cost analysis.

The question was: What does announcing a prefix into the DFZ cost
"everybody else?" There are a number of constraints on the analysis
which are implicit in that question. The relevant constraint here is
that the equipment chosen for the various

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Jeff McAdams]

Joe Abley wrote:
> On 20-Jan-2008, at 15:34, William Herrin wrote:

>> Perhaps your definition of "entry level DFZ router" differs from mine.
>> I selected a Cisco 7600 w/ sup720-3bxl or rsp720-3xcl as my baseline
>> for an entry level DFZ router.

> A new cisco 2851 can be found for under $10k and can take a gig of RAM.
> If your goal is to have fine-grained routing data, and not to carry gigs
> of traffic, that particular router is perfectly adequate.

And to take that concept to its logical extreme.

A Linux box (*BSD, pick your poison) running Quagga or similar will do
the job at an extremely low price point.

Yeah, again, not gonna want to pass gigs of traffic through it, but the
same concept does still apply.
-- 
Jeff McAdams
"They that can give up essential liberty to obtain a
little temporary safety deserve neither liberty nor safety."
   -- Benjamin Franklin



signature.asc
Description: OpenPGP digital signature

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Joe Abley]

On 20-Jan-2008, at 15:34, William Herrin wrote:


Perhaps your definition of "entry level DFZ router" differs from mine.
I selected a Cisco 7600 w/ sup720-3bxl or rsp720-3xcl as my baseline
for an entry level DFZ router.


A new cisco 2851 can be found for under $10k and can take a gig of  
RAM. If your goal is to have fine-grained routing data, and not to  
carry gigs of traffic, that particular router is perfectly adequate.


If you're prepared to consider second-hand equipment (which seems  
fair, since it's not as though the real Internet has no eBay VXRs in  
it) you could get better performance, or lower cost, depending on  
which way you wanted to turn the dial.


Sometimes it's important to appreciate that the network edge is bigger  
than the network core. Just because this kind of equipment wouldn't  
come close to cutting it in a carrier network doesn't mean that they  
aren't perfectly appropriate for a large proportion of deployed  
routers which take a full table.



Joe

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Patrick W. Gilmore]

On Jan 20, 2008, at 3:34 PM, William Herrin wrote:


The difference is much, much, much greater than that.  Can the switch
do ACLs?  Policy routing?  SFlow with the same sampling rate?  Same
number of BGP session?


Is there some alternate piece of cheap hardware that supports the DFZ
prefix count at high data rates but doesn't have those features? If
the answer is no (and I'm pretty sure the answer is no), then the
prefix count remains the proper attribution for the cost delta.


We still disagree.

I notice you cut out the next two sentences:


In short, if the table were 50K prefixes instead of 250K, would these  
pieces of equipment be equivalent?  The answer is a blatant "no".



If we take out the "proper attribution for the cost delta" out of the  
equation and the equipment is still not considered equal, I submit  
your idea of "proper attribution" is, well, not proper.


To be clear, of course there are some people who could use either if  
the table were 50K prefixes.  But the majority of routers in the DFZ  
cannot be replaced by cheap 1U (or whatever) switches which can do a  
few 10s of 1000s of prefixes.  (Besides, the people who _can_ use them  
can use them today with properly configured filters, perhaps on the  
upstream router's side.  Which, of course, means the upstream router  
cannot be one of those cheap switches. :)


Perhaps you should justify numbers with nine zeros a little better  
before asking me to justify why they are wrong.


--
TTFN,
patrick

RE: Cost per prefix [was: request for help w/ ATT and terminology]

[Neil J. McRae]

Delivering BGP solutions for end users is starting to get very expensive 
particularly for those networks with lots of smaller pops, I think some effort 
to look at how this might be better delivered without access boxes needing to 
know the entire routing table esp in light of IPV6

Regards,
Neil
(missed the end of the last email!)
 

-Original Message-
From: William Herrin <[EMAIL PROTECTED]>
Sent: 20 January 2008 17:22
To: Patrick W. Gilmore <[EMAIL PROTECTED]>
Cc: nanog@merit.edu
Subject: Re: Cost per prefix [was: request for help w/ ATT and terminology]


On Jan 20, 2008 9:46 AM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:
> On Jan 20, 2008, at 6:06 AM, William Herrin wrote:
> > On Jan 19, 2008 11:43 PM, Patrick W. Gilmore <[EMAIL PROTECTED]>
> > wrote:
> >> On Jan 19, 2008, at 12:55 PM, William Herrin wrote:
> >>> There was some related work on ARIN PPML last year. The rough
> >>> numbers
> >>> suggested that the attributable economic cost of one IPv4 prefix in
> >>> the DFZ (whether PI, PA or TE) was then in the neighborhood of $8000
> >>> USD per year.
> >>
> >> I haven't seen that work, but I am guessing this number is an
> >> aggregate (i.e. every cost to everyone on the 'Net combined), not
> >> per-
> >> network? See, I'm just looking at that TWO BILLION DOLLARS PER YEAR
> >> number and thinking to myself, "um, yeah, right". :)
> >
> > Patrick,
> >
> > That was a worldwide total, yes. The cost per prefix per router is
> > obviously only measured in cents per year.
>
> I think you mean in tiny fractions of a single cent per router per
> year

No, I don't. The lower bound for that particular portion of the cost
analysis is easy to calculate:

Entry level DFZ router: $40,000
Stacked 1U layer-3 switches with similar switching capacity and port
density: $10,000
Difference between the two: The switch stack can't handle the DFZ prefix count.
Cost delta (attributable to the DFZ prefix count): $30,000
Expected lifespan in the DFZ of an entry-level router: 3 years
Prefixes in the table: 245,000

Calculation: The LOWER BOUND for the cost per prefix per router can be
calculated as:
( [entry level router's cost attributable to prefixes]/[expected
lifespan] ) / [DFZ prefix count]
($30,000/3)/245,000 = $0.04 per router per year, i.e. 4 cents.

Bear in mind that 4 cents per year is a LOWER BOUND. It costs AT LEAST
4 cents per router per year to carry one prefix in one DFZ router.
There are also routers in the DFZ which cost $500,000 where much more
than $30,000 is attributable to the prefix count.


>.  While there are 27K ASes ($0.30/year/AS, remember?), there are
> many more routers which carry a full table.

Yes, there are many more routers than ASes. In my original analysis on
ARIN, I estimated that there were some 200,000 routers carrying a full
table in the DFZ. The consensus at the time was that the number was
closer to 150,000 than 200,000. 150,000 times 4 cents yields a LOWER
BOUND economic impact of $6,000 per prefix per year, $1.5B overall.

Again, that's a lower bound on the estimate. The upper bound is
perhaps twice that with the highest probability cost around $8,000 per
prefix per year.


> Comparing cisco & Juniper's annual revenue to the cost of a prefix is
> like comparing Ford & GM's revenue to the cost of bulbs in
> headlights.  Hell, most of cisco's revenue is not even related to
> routers doing a full table.

Of course not. However, it makes a good sanity check on the cost
estimate: If the costs attributable to prefix count sums to more than
their revenues then there must be an error in the math. My point was
that the $8000/prefix/year estimate passes the sanity check with
plenty of room to spare.



> > The thread started here:
> > http://lists.arin.net/pipermail/ppml/2007-September/008927.html
> > It was originally an argument of about the cost of doing PI for IPv6,
> > which according to Cisco product literature consumes twice the amount
> > of space in the FIB as routes for IPv4.
>
> Anyway, thanks for the link.  I must be missing something seriously
> important to the calculation.  Perhaps it includes things like human
> time to upgrade equipment or filters or something?  I'll see how the
> calculation 


[The entire original message is not included]

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[William Herrin]

On Jan 20, 2008 1:11 PM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:
> On Jan 20, 2008, at 12:22 PM, William Herrin wrote:
> >> I think you mean in tiny fractions of a single cent per router per
> >> year
> >
> > No, I don't. The lower bound for that particular portion of the cost
> > analysis is easy to calculate:
> > ( [entry level router's cost attributable to prefixes]/[expected
> > lifespan] ) / [DFZ prefix count]
>
> Your calculation is in error.

Patrick,

Feel free to correct it. Substitute any numbers that you can
*justify*, add any factors that you can justify and recalculate. If
your justification is sensible, I'll adjust my own numbers
accordingly. I'd prefer that the numbers be lower so I can find a way
to justify IPv6 PI space. :)

However, please try not to disagree merely because you don't want to
accept the results. Denial is not productive.


> > Entry level DFZ router: $40,000
> > Stacked 1U layer-3 switches with similar switching capacity and port
> > density: $10,000
>
> One of us is very confused.  Given that I order "entry level DFZ
> routers" all the time far less than $40K every month, I am going to
> guess it is not me.

Perhaps your definition of "entry level DFZ router" differs from mine.
I selected a Cisco 7600 w/ sup720-3bxl or rsp720-3xcl as my baseline
for an entry level DFZ router. This seemed to be the minimum Cisco kit
which could reasonably be expected to remain in service as a full
table DFZ router through 1/2011. What would you select instead?

Please note that neither a Cisco 7200 nor a sup/rsp720 prior to the
3bxl/3cxl can be reasonably expected to have 3-year lifespan in the
DFZ. They simply can't keep up with the expected growth in the routing
table.

I'll confess to not knowing the Juniper line well enough to comment on
their equivalent system. Do they cost radically less than Cisco gear?


> The difference is much, much, much greater than that.  Can the switch
> do ACLs?  Policy routing?  SFlow with the same sampling rate?  Same
> number of BGP session?

Is there some alternate piece of cheap hardware that supports the DFZ
prefix count at high data rates but doesn't have those features? If
the answer is no (and I'm pretty sure the answer is no), then the
prefix count remains the proper attribution for the cost delta.

And yes, today's "virtual chassis" 1U switch stacks are pretty slick.
I can't speak to sflow or policy routing but many if not most support
both BGP and ACLs. Most also have 128M on the control plane and 8k or
16k entries in the TCAMs which obviously does not support a DFZ table.

Regards,
Bill Herrin


-- 
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: 
Falls Church, VA 22042-3004

RE: Cost per prefix [was: request for help w/ ATT and terminology]

[Neil J. McRae]

Bill
I think your approach has some merits as a measuring stick but more stufy on 
this is absolutely needed. 

Given what I've experienced recently I think the cost per prefix is greatly 
undervalued. It is actually a capacity/step change model that almost equates to 
the same model used for core bandwidth upgrades but the core bandwidth upgrades 
and the cost to refresh those has made the cost per prefix a non issue for the 
most part. 

Unfortunately thats changing especially in the access and distribution layers 
of the network. Where for the most part there are devices that can currently 
cope with the bandwidth demand but can't cope with the route table size demand. 
I suspect any AS older than 5 years has a ton of kit that can't take the entire 
routing table and is causing major headaches and requiring rehoming of BGP4 
customers to newer kit for no additional revenues!

Delivering BGP solutions for end users is starting to get very expensive 
particularly for those networks with lots of smaller pops, I think some effort 
to look at how this might be be

Regards,
Neil
 

-Original Message-
From: William Herrin <[EMAIL PROTECTED]>
Sent: 20 January 2008 17:22
To: Patrick W. Gilmore <[EMAIL PROTECTED]>
Cc: nanog@merit.edu
Subject: Re: Cost per prefix [was: request for help w/ ATT and terminology]


On Jan 20, 2008 9:46 AM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:
> On Jan 20, 2008, at 6:06 AM, William Herrin wrote:
> > On Jan 19, 2008 11:43 PM, Patrick W. Gilmore <[EMAIL PROTECTED]>
> > wrote:
> >> On Jan 19, 2008, at 12:55 PM, William Herrin wrote:
> >>> There was some related work on ARIN PPML last year. The rough
> >>> numbers
> >>> suggested that the attributable economic cost of one IPv4 prefix in
> >>> the DFZ (whether PI, PA or TE) was then in the neighborhood of $8000
> >>> USD per year.
> >>
> >> I haven't seen that work, but I am guessing this number is an
> >> aggregate (i.e. every cost to everyone on the 'Net combined), not
> >> per-
> >> network? See, I'm just looking at that TWO BILLION DOLLARS PER YEAR
> >> number and thinking to myself, "um, yeah, right". :)
> >
> > Patrick,
> >
> > That was a worldwide total, yes. The cost per prefix per router is
> > obviously only measured in cents per year.
>
> I think you mean in tiny fractions of a single cent per router per
> year

No, I don't. The lower bound for that particular portion of the cost
analysis is easy to calculate:

Entry level DFZ router: $40,000
Stacked 1U layer-3 switches with similar switching capacity and port
density: $10,000
Difference between the two: The switch stack can't handle the DFZ prefix count.
Cost delta (attributable to the DFZ prefix count): $30,000
Expected lifespan in the DFZ of an entry-level router: 3 years
Prefixes in the table: 245,000

Calculation: The LOWER BOUND for the cost per prefix per router can be
calculated as:
( [entry level router's cost attributable to prefixes]/[expected
lifespan] ) / [DFZ prefix count]
($30,000/3)/245,000 = $0.04 per router per year, i.e. 4 cents.

Bear in mind that 4 cents per year is a LOWER BOUND. It costs AT LEAST
4 cents per router per year to carry one prefix in one DFZ router.
There are also routers in the DFZ which cost $500,000 where much more
than $30,000 is attributable to the prefix count.


>.  While there are 27K ASes ($0.30/year/AS, remember?), there are
> many more routers which carry a full table.

Yes, there are many more routers than ASes. In my original analysis on
ARIN, I estimated that there were some 200,000 routers carrying a full
table in the DFZ. The consensus at the time was that the number was
closer to 150,000 than 200,000. 150,000 times 4 cents yields a LOWER
BOUND economic impact of $6,000 per prefix per year, $1.5B overall.

Again, that's a lower bound on the estimate. The upper bound is
perhaps twice that with the highest probability cost around $8,000 per
prefix per year.


> Comparing cisco & Juniper's annual revenue to the cost of a prefix is
> like comparing Ford & GM's revenue to the cost of bulbs in
> headlights.  Hell, most of cisco's revenue is not even related to
> routers doing a full table.

Of course not. However, it makes a good sanity check on the cost
estimate: If the costs attributable to prefix count sums to more than
their revenues then there must be an error in the math. My point was
that the $8000/prefix/year estimate passes the sanity check with
plenty of room to spare.



> > The thread started here:
> > http://lists.arin.net/pipermail/ppml/2007-September/008927.html
> > It was originally an argument of about the cost of doing PI for IPv6,
> > which according to Cisco product litera

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Joe Greco]

> But before we go too far down this road, everyone here should realize  
> that new PI space and PA deaggregation WILL CONTINUE TO HAPPEN.
> 
> Many corporations paying for Internet access will NOT be tied to a  
> single provider.  Period.  Trying to tell them "you are too small, you  
> should only let us big networks have our own space" is a silly  
> argument which won't fly.
> 
> The Internet is a business tool.  Trying to make that tool less  
> flexible, trying to tie the fate of a customer to the fate of a single  
> provider, or trying force them to jump through more hoops than you  
> have to jump through for the same redundancy / reliability is simply  
> not realistic.  And telling them it will cost some random network in  
> some random other place a dollar a year for their additional  
> flexibility / reliability / performance is not going to convince them  
> not to do it.
> 
> The number of these coAt least not while the Internet is still driven  
> by commercial realities.  (Which I personally think is a Very Good  
> Thing - much better than the alternative.)  Someone will take the  
> customer's check, so the prefix will be in the table.  And since you  
> want to take your customers' checks to provide access to that ISP's  
> customer, you will have to carry the prefix.
> 
> Of course, that doesn't mean we shouldn't be thrifty with table  
> space.  We just have to stop thinking that only the largest providers  
> should be allowed to add a prefix to the table.  At least if we are  
> going to continue making money on the Internet.

While I agree with this to some extent, it is clear that there are some
problems.  The obvious problem is where the line is drawn; it is not
currently reasonable for each business class DSL line to be issued PI
space, but it is currently reasonable for the largest 100 companies in
the world to have PI space.  (I've deliberately drawn the boundary lines
well outside what most would argue as a reasonable range; the boundaries
I've drawn are not open to debate, since they're for the purposes of
contemplating a problem.)

I don't think that simply writing a check to an ISP is going to be
sufficiently compelling to cause networks of the world to accept a 
prefix in the table.  If I happen to be close to running out of table
entries, then I may not see any particular value in accepting a prefix
that serves no good purpose.  For example, PA deaggregated space and
prefixes from far away might be among the first victims, with the former
being filtered (hope you have a covering route!) and the latter being
filtered with a local covering route installed to default a bunch of
APNIC routes out a reasonable pipe.

For the overall good of the Internet, that's not particularly desirable,
but it will be a reality for providers who can't keep justifying
installing lots of routers with larger table sizes every few years.

There is, therefore, some commercial interest above and beyond "hey, 
look, some guy paid me."  We'd like the Internet to work _well_, and
that means that self-interest exclusive of all else is not going to be
a good way to contemplate commercial realities.

So, what can reasonably be done?  Given what I've seen over the years,
I keep coming back to the idea that PI space allocations are not all
that far out of control, but the PA deaggregation situation is fairly
rough.  There would also seem to be some things that smaller sites could
do to "fix" the PA deagg situation.  Is this the way people see things
going, if we're going to be realistic?

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Patrick W. Gilmore]

On Jan 20, 2008, at 12:22 PM, William Herrin wrote:


I think you mean in tiny fractions of a single cent per router per
year


No, I don't. The lower bound for that particular portion of the cost
analysis is easy to calculate:


Your calculation is in error.



Entry level DFZ router: $40,000
Stacked 1U layer-3 switches with similar switching capacity and port
density: $10,000


One of us is very confused.  Given that I order "entry level DFZ  
routers" all the time far less than $40K every month, I am going to  
guess it is not me.



Difference between the two: The switch stack can't handle the DFZ  
prefix count.


The difference is much, much, much greater than that.  Can the switch  
do ACLs?  Policy routing?  SFlow with the same sampling rate?  Same  
number of BGP session?  Etc.  In short, if the table were 50K prefixes  
instead of 250K, would these pieces of equipment be equivalent?  The  
answer is a blatant "no".


I can cook stats as well as the next guy, but I try to be painfully  
obvious about my bias.


Given that you say the thread doesn't include more than the number  
here, I don't even think I need to read the thread.  The cost  
(billions of dollars per year) is clearly in error.


--
TTFN,
patrick



Cost delta (attributable to the DFZ prefix count): $30,000
Expected lifespan in the DFZ of an entry-level router: 3 years
Prefixes in the table: 245,000

Calculation: The LOWER BOUND for the cost per prefix per router can be
calculated as:
( [entry level router's cost attributable to prefixes]/[expected
lifespan] ) / [DFZ prefix count]
($30,000/3)/245,000 = $0.04 per router per year, i.e. 4 cents.

Bear in mind that 4 cents per year is a LOWER BOUND. It costs AT LEAST
4 cents per router per year to carry one prefix in one DFZ router.
There are also routers in the DFZ which cost $500,000 where much more
than $30,000 is attributable to the prefix count.



.  While there are 27K ASes ($0.30/year/AS, remember?), there are
many more routers which carry a full table.


Yes, there are many more routers than ASes. In my original analysis on
ARIN, I estimated that there were some 200,000 routers carrying a full
table in the DFZ. The consensus at the time was that the number was
closer to 150,000 than 200,000. 150,000 times 4 cents yields a LOWER
BOUND economic impact of $6,000 per prefix per year, $1.5B overall.

Again, that's a lower bound on the estimate. The upper bound is
perhaps twice that with the highest probability cost around $8,000 per
prefix per year.



Comparing cisco & Juniper's annual revenue to the cost of a prefix is
like comparing Ford & GM's revenue to the cost of bulbs in
headlights.  Hell, most of cisco's revenue is not even related to
routers doing a full table.


Of course not. However, it makes a good sanity check on the cost
estimate: If the costs attributable to prefix count sums to more than
their revenues then there must be an error in the math. My point was
that the $8000/prefix/year estimate passes the sanity check with
plenty of room to spare.




The thread started here:
http://lists.arin.net/pipermail/ppml/2007-September/008927.html
It was originally an argument of about the cost of doing PI for  
IPv6,
which according to Cisco product literature consumes twice the  
amount

of space in the FIB as routes for IPv4.


Anyway, thanks for the link.  I must be missing something seriously
important to the calculation.  Perhaps it includes things like human
time to upgrade equipment or filters or something?  I'll see how the
calculation was put together.


Nope, just the numbers you see in the link. I didn't calculate cost
increases due to manpower,  cost reductions due to equipment reuse
after its DFZ lifespan, or other factors for which data is sketchy and
the likely impact to the analysis is small.

Like I said: read the thread, critique the numbers and then add them
up for yourself. A prefix is surprisingly expensive. If it wasn't,
folks wouldn't be so concerned about the rising prefix count.

Regards,
Bill Herrin

--
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: 
Falls Church, VA 22042-3004

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[William Herrin]

On Jan 20, 2008 9:46 AM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:
> On Jan 20, 2008, at 6:06 AM, William Herrin wrote:
> > On Jan 19, 2008 11:43 PM, Patrick W. Gilmore <[EMAIL PROTECTED]>
> > wrote:
> >> On Jan 19, 2008, at 12:55 PM, William Herrin wrote:
> >>> There was some related work on ARIN PPML last year. The rough
> >>> numbers
> >>> suggested that the attributable economic cost of one IPv4 prefix in
> >>> the DFZ (whether PI, PA or TE) was then in the neighborhood of $8000
> >>> USD per year.
> >>
> >> I haven't seen that work, but I am guessing this number is an
> >> aggregate (i.e. every cost to everyone on the 'Net combined), not
> >> per-
> >> network? See, I'm just looking at that TWO BILLION DOLLARS PER YEAR
> >> number and thinking to myself, "um, yeah, right". :)
> >
> > Patrick,
> >
> > That was a worldwide total, yes. The cost per prefix per router is
> > obviously only measured in cents per year.
>
> I think you mean in tiny fractions of a single cent per router per
> year

No, I don't. The lower bound for that particular portion of the cost
analysis is easy to calculate:

Entry level DFZ router: $40,000
Stacked 1U layer-3 switches with similar switching capacity and port
density: $10,000
Difference between the two: The switch stack can't handle the DFZ prefix count.
Cost delta (attributable to the DFZ prefix count): $30,000
Expected lifespan in the DFZ of an entry-level router: 3 years
Prefixes in the table: 245,000

Calculation: The LOWER BOUND for the cost per prefix per router can be
calculated as:
( [entry level router's cost attributable to prefixes]/[expected
lifespan] ) / [DFZ prefix count]
($30,000/3)/245,000 = $0.04 per router per year, i.e. 4 cents.

Bear in mind that 4 cents per year is a LOWER BOUND. It costs AT LEAST
4 cents per router per year to carry one prefix in one DFZ router.
There are also routers in the DFZ which cost $500,000 where much more
than $30,000 is attributable to the prefix count.


>.  While there are 27K ASes ($0.30/year/AS, remember?), there are
> many more routers which carry a full table.

Yes, there are many more routers than ASes. In my original analysis on
ARIN, I estimated that there were some 200,000 routers carrying a full
table in the DFZ. The consensus at the time was that the number was
closer to 150,000 than 200,000. 150,000 times 4 cents yields a LOWER
BOUND economic impact of $6,000 per prefix per year, $1.5B overall.

Again, that's a lower bound on the estimate. The upper bound is
perhaps twice that with the highest probability cost around $8,000 per
prefix per year.


> Comparing cisco & Juniper's annual revenue to the cost of a prefix is
> like comparing Ford & GM's revenue to the cost of bulbs in
> headlights.  Hell, most of cisco's revenue is not even related to
> routers doing a full table.

Of course not. However, it makes a good sanity check on the cost
estimate: If the costs attributable to prefix count sums to more than
their revenues then there must be an error in the math. My point was
that the $8000/prefix/year estimate passes the sanity check with
plenty of room to spare.



> > The thread started here:
> > http://lists.arin.net/pipermail/ppml/2007-September/008927.html
> > It was originally an argument of about the cost of doing PI for IPv6,
> > which according to Cisco product literature consumes twice the amount
> > of space in the FIB as routes for IPv4.
>
> Anyway, thanks for the link.  I must be missing something seriously
> important to the calculation.  Perhaps it includes things like human
> time to upgrade equipment or filters or something?  I'll see how the
> calculation was put together.

Nope, just the numbers you see in the link. I didn't calculate cost
increases due to manpower,  cost reductions due to equipment reuse
after its DFZ lifespan, or other factors for which data is sketchy and
the likely impact to the analysis is small.

Like I said: read the thread, critique the numbers and then add them
up for yourself. A prefix is surprisingly expensive. If it wasn't,
folks wouldn't be so concerned about the rising prefix count.

Regards,
Bill Herrin

-- 
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: 
Falls Church, VA 22042-3004

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Roland Dobbins]

On Jan 21, 2008, at 12:22 AM, Ben Butler wrote:

Or maybe... we will run out of corporates first!  Which would have  
to be

the best of outcomes, everyone multihomed how wants/needs plus a
manageable route table without having run out of IPs or AS numbers.


As Internet connectivity becomes more and more vital to the mechanics  
of everyday life, it probably won't just be corporations wanting to  
multihome, but individuals (and their associated spime-clouds) which  
want to multihome, too - probably via a combination of wireline and  
wireless methods.


---
Roland Dobbins <[EMAIL PROTECTED]> // 408.527.6376 voice

Culture eats strategy for breakfast.

   -- Ford Motor Company

RE: Cost per prefix [was: request for help w/ ATT and terminology]

[Ben Butler]

Hi Patrick,

I agree, if anything I am advocating a spanking of those in the clueless
category, thus reducing the table size so that up to 110K more PI space
and corporate multi homing can occur without increasing the table
further than today.  Seems like a quick gain in flexibility and
functionality for the corporate customers while curtailing the littering
of those that should know better.

Of course it begs the question when do we run out of AS numbers / PI
space - maybe one of those events would now actually happen before
routers collapses under the weight of route table from corporate
multihoming once we have tided up the unnecessary PA deag.

Or maybe... we will run out of corporates first!  Which would have to be
the best of outcomes, everyone multihomed how wants/needs plus a
manageable route table without having run out of IPs or AS numbers.  Or
am I just living in fairy land?

I guess without knowing the size of the wave of corporate foaming out
the mouth to multihome it is difficult to know whether pushing back the
flood of deagregated PA by 110K is enough to make a serious dent in the
corporate wave or just a ripple to a problem that is too big to ever
solve and enormous route table are inevitable and 100K extra routes from
PA is neither here nor there at the end of the day?

Has anyone worked out any numbers / projections on this going forwards
into the future?

Ben

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
Patrick W. Gilmore
Sent: 20 January 2008 15:12
To: nanog@merit.edu
Cc: Patrick W. Gilmore
Subject: Re: Cost per prefix [was: request for help w/ ATT and
terminology]


Ben,

I believe you are correct that PA deaggregation is a huge problem, but
some of that could be corporate multi-homing.  (I don't know for certain
whether it is greater or less than providers just being
ninnies.)  Lots of companies get a /24 from one upstream and announce it
to two or more upstreams.  That is, IMHO, a legitimate deaggregation, as
opposed to a provider who is just too clueless aggregate.


But before we go too far down this road, everyone here should realize
that new PI space and PA deaggregation WILL CONTINUE TO HAPPEN.

Many corporations paying for Internet access will NOT be tied to a
single provider.  Period.  Trying to tell them "you are too small, you
should only let us big networks have our own space" is a silly argument
which won't fly.

The Internet is a business tool.  Trying to make that tool less
flexible, trying to tie the fate of a customer to the fate of a single
provider, or trying force them to jump through more hoops than you have
to jump through for the same redundancy / reliability is simply not
realistic.  And telling them it will cost some random network in some
random other place a dollar a year for their additional flexibility /
reliability / performance is not going to convince them not to do it.

The number of these coAt least not while the Internet is still driven by
commercial realities.  (Which I personally think is a Very Good Thing -
much better than the alternative.)  Someone will take the customer's
check, so the prefix will be in the table.  And since you want to take
your customers' checks to provide access to that ISP's customer, you
will have to carry the prefix.


Of course, that doesn't mean we shouldn't be thrifty with table space.
We just have to stop thinking that only the largest providers should be
allowed to add a prefix to the table.  At least if we are going to
continue making money on the Internet.

--
TTFN,
patrick



On Jan 20, 2008, at 7:08 AM, Ben Butler wrote:

>
> Hi,
>
> Out of curiosity was the reasoning also to charge the PA who are
> deagregating?
>
> To restate there are 113,220 extra routes smaller than RIR minimums  
> out
> of the /24:126,450 in the table.  The today reality seems to be that
> 113K of that 126K is probably being caused by existing networks
> de-aggregating PA.
>
> While I would I would agree that corporate multihoming with PI has a
> huge potential problem on the table in terms of number of prefixes.
> Further more as BGP skills are becoming more common place and
> Linux/Quagga skills the barrier to entry for a corporate is reducing  
> at
> the same time their commercial reliance on and use of the Internet is
> increasing.
>
> Corporate multihoming - if permitted - has the inevitable  
> consequence of
> an extra prefix.
>
> PA deagreagtion - has the avoidable consequence of lots of extra
> prefixes.
>
> I know who I would be charging first and maybe it would give the much
> need incentive for them to clean house.  We could then have some  
> number
> up to 113K new multihomed corporate before we got back to where we are
> today in terms of route table size.  An interesting question to gauge
> the size of the corporate multihomi

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Patrick W. Gilmore]

Ben,

I believe you are correct that PA deaggregation is a huge problem, but  
some of that could be corporate multi-homing.  (I don't know for  
certain whether it is greater or less than providers just being  
ninnies.)  Lots of companies get a /24 from one upstream and announce  
it to two or more upstreams.  That is, IMHO, a legitimate  
deaggregation, as opposed to a provider who is just too clueless  
aggregate.



But before we go too far down this road, everyone here should realize  
that new PI space and PA deaggregation WILL CONTINUE TO HAPPEN.


Many corporations paying for Internet access will NOT be tied to a  
single provider.  Period.  Trying to tell them "you are too small, you  
should only let us big networks have our own space" is a silly  
argument which won't fly.


The Internet is a business tool.  Trying to make that tool less  
flexible, trying to tie the fate of a customer to the fate of a single  
provider, or trying force them to jump through more hoops than you  
have to jump through for the same redundancy / reliability is simply  
not realistic.  And telling them it will cost some random network in  
some random other place a dollar a year for their additional  
flexibility / reliability / performance is not going to convince them  
not to do it.


The number of these coAt least not while the Internet is still driven  
by commercial realities.  (Which I personally think is a Very Good  
Thing - much better than the alternative.)  Someone will take the  
customer's check, so the prefix will be in the table.  And since you  
want to take your customers' checks to provide access to that ISP's  
customer, you will have to carry the prefix.



Of course, that doesn't mean we shouldn't be thrifty with table  
space.  We just have to stop thinking that only the largest providers  
should be allowed to add a prefix to the table.  At least if we are  
going to continue making money on the Internet.


--
TTFN,
patrick



On Jan 20, 2008, at 7:08 AM, Ben Butler wrote:



Hi,

Out of curiosity was the reasoning also to charge the PA who are
deagregating?

To restate there are 113,220 extra routes smaller than RIR minimums  
out

of the /24:126,450 in the table.  The today reality seems to be that
113K of that 126K is probably being caused by existing networks
de-aggregating PA.

While I would I would agree that corporate multihoming with PI has a
huge potential problem on the table in terms of number of prefixes.
Further more as BGP skills are becoming more common place and
Linux/Quagga skills the barrier to entry for a corporate is reducing  
at

the same time their commercial reliance on and use of the Internet is
increasing.

Corporate multihoming - if permitted - has the inevitable  
consequence of

an extra prefix.

PA deagreagtion - has the avoidable consequence of lots of extra
prefixes.

I know who I would be charging first and maybe it would give the much
need incentive for them to clean house.  We could then have some  
number

up to 113K new multihomed corporate before we got back to where we are
today in terms of route table size.  An interesting question to gauge
the size of the corporate multihoming potential problem is to guess  
how

many there may be worldwide that would bother / try - I have no idea.

I believe (possibly wrongly) that IPv6 doesn't really have a solution
for multihoming corporate with multiple allocations and weird shims  
and

NAT configurations to get it to work - or have the RIRs decided on a
policy change yet and issuing PI blocks of IPv6 as well?

Am I correct on my interpretation of the numbers for PA:PI smaller
prefix origins?

Kind Regards

Ben

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf  
Of

William Herrin
Sent: 20 January 2008 11:06
To: Patrick W. Gilmore
Cc: nanog@merit.edu
Subject: Re: Cost per prefix [was: request for help w/ ATT and
terminology]


On Jan 19, 2008 11:43 PM, Patrick W. Gilmore <[EMAIL PROTECTED]>  
wrote:

On Jan 19, 2008, at 12:55 PM, William Herrin wrote:

There was some related work on ARIN PPML last year. The rough
numbers suggested that the attributable economic cost of one IPv4
prefix in the DFZ (whether PI, PA or TE) was then in the
neighborhood of $8000 USD per year.


I haven't seen that work, but I am guessing this number is an
aggregate (i.e. every cost to everyone on the 'Net combined), not  
per-



network? See, I'm just looking at that TWO BILLION DOLLARS PER YEAR
number and thinking to myself, "um, yeah, right". :)


Patrick,

That was a worldwide total, yes. The cost per prefix per router is
obviously only measured in cents per year.

You do know that Cisco's sales are north of $20B per year, right?
Juniper, which sells few products that aren't DFZ routers, also posts
annual revenues well north of $1B.



Feel free to explain how confused I am.  (But be warned, I am not
go

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[Patrick W. Gilmore]

On Jan 20, 2008, at 6:06 AM, William Herrin wrote:
On Jan 19, 2008 11:43 PM, Patrick W. Gilmore <[EMAIL PROTECTED]>  
wrote:

On Jan 19, 2008, at 12:55 PM, William Herrin wrote:
There was some related work on ARIN PPML last year. The rough  
numbers

suggested that the attributable economic cost of one IPv4 prefix in
the DFZ (whether PI, PA or TE) was then in the neighborhood of $8000
USD per year.


I haven't seen that work, but I am guessing this number is an
aggregate (i.e. every cost to everyone on the 'Net combined), not  
per-

network? See, I'm just looking at that TWO BILLION DOLLARS PER YEAR
number and thinking to myself, "um, yeah, right". :)


Patrick,

That was a worldwide total, yes. The cost per prefix per router is
obviously only measured in cents per year.


I think you mean in tiny fractions of a single cent per router per  
year.  While there are 27K ASes ($0.30/year/AS, remember?), there are  
many more routers which carry a full table.




You do know that Cisco's sales are north of $20B per year, right?
Juniper, which sells few products that aren't DFZ routers, also posts
annual revenues well north of $1B.


Comparing cisco & Juniper's annual revenue to the cost of a prefix is  
like comparing Ford & GM's revenue to the cost of bulbs in  
headlights.  Hell, most of cisco's revenue is not even related to  
routers doing a full table.


Interesting thought experiment.  Let's assume _ALL_ $21B of revenue  
you quote above is routers which can do a full table.  The numbers you  
quote say 10% of that revenue is because of DFZ table size.  I was  
unaware so much cost in a router was just table size.  And since we  
all know that revenue is not all DFZ-capable routers (for instance,  
how much of that $20B is Linksys?), the %-age is much higher.


Wow, router member & CPU must be very expensive - and optics must be  
damned cheap.


Besides the obvious absurdity in this, it contradicts the point I made  
to your last paragraph.


I guess I'm thinking again: "Um, yeah, right". :)



Feel free to explain how confused I am.  (But be warned, I am not
going to believe it costs $2B/year to run a multi-homed network with
two full feeds. :)


The thread started here:
http://lists.arin.net/pipermail/ppml/2007-September/008927.html
It was originally an argument of about the cost of doing PI for IPv6,
which according to Cisco product literature consumes twice the amount
of space in the FIB as routes for IPv4.


Heaven forbid it costs each ASN an average of TWO DOLLARS per year.   
Eventually, since it will be quite a while before the v6 table is 250K  
prefixes.  Hrmm, I take that back.  By the time there are 250K v6  
prefixes, there will be far, far more ASNs, so the average cost will  
be less.


Anyway, thanks for the link.  I must be missing something seriously  
important to the calculation.  Perhaps it includes things like human  
time to upgrade equipment or filters or something?  I'll see how the  
calculation was put together.


--
TTFN,
patrick

RE: Cost per prefix [was: request for help w/ ATT and terminology]

[Ben Butler]

Hi,

Out of curiosity was the reasoning also to charge the PA who are
deagregating?

To restate there are 113,220 extra routes smaller than RIR minimums out
of the /24:126,450 in the table.  The today reality seems to be that
113K of that 126K is probably being caused by existing networks
de-aggregating PA.

While I would I would agree that corporate multihoming with PI has a
huge potential problem on the table in terms of number of prefixes.
Further more as BGP skills are becoming more common place and
Linux/Quagga skills the barrier to entry for a corporate is reducing at
the same time their commercial reliance on and use of the Internet is
increasing.

Corporate multihoming - if permitted - has the inevitable consequence of
an extra prefix.

PA deagreagtion - has the avoidable consequence of lots of extra
prefixes.

I know who I would be charging first and maybe it would give the much
need incentive for them to clean house.  We could then have some number
up to 113K new multihomed corporate before we got back to where we are
today in terms of route table size.  An interesting question to gauge
the size of the corporate multihoming potential problem is to guess how
many there may be worldwide that would bother / try - I have no idea.

I believe (possibly wrongly) that IPv6 doesn't really have a solution
for multihoming corporate with multiple allocations and weird shims and
NAT configurations to get it to work - or have the RIRs decided on a
policy change yet and issuing PI blocks of IPv6 as well?

Am I correct on my interpretation of the numbers for PA:PI smaller
prefix origins?

Kind Regards

Ben

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
William Herrin
Sent: 20 January 2008 11:06
To: Patrick W. Gilmore
Cc: nanog@merit.edu
Subject: Re: Cost per prefix [was: request for help w/ ATT and
terminology]


On Jan 19, 2008 11:43 PM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:
> On Jan 19, 2008, at 12:55 PM, William Herrin wrote:
> > There was some related work on ARIN PPML last year. The rough 
> > numbers suggested that the attributable economic cost of one IPv4 
> > prefix in the DFZ (whether PI, PA or TE) was then in the 
> > neighborhood of $8000 USD per year.
>
> I haven't seen that work, but I am guessing this number is an 
> aggregate (i.e. every cost to everyone on the 'Net combined), not per-

> network? See, I'm just looking at that TWO BILLION DOLLARS PER YEAR 
> number and thinking to myself, "um, yeah, right". :)

Patrick,

That was a worldwide total, yes. The cost per prefix per router is
obviously only measured in cents per year.

You do know that Cisco's sales are north of $20B per year, right?
Juniper, which sells few products that aren't DFZ routers, also posts
annual revenues well north of $1B.


> Feel free to explain how confused I am.  (But be warned, I am not 
> going to believe it costs $2B/year to run a multi-homed network with 
> two full feeds. :)

The thread started here:
http://lists.arin.net/pipermail/ppml/2007-September/008927.html
It was originally an argument of about the cost of doing PI for IPv6,
which according to Cisco product literature consumes twice the amount of
space in the FIB as routes for IPv4.

I encourage you to critique the numbers and then add them up for
yourself.

Regards,
Bill Herrin


-- 
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

Re: Cost per prefix [was: request for help w/ ATT and terminology]

[William Herrin]

On Jan 19, 2008 11:43 PM, Patrick W. Gilmore <[EMAIL PROTECTED]> wrote:
> On Jan 19, 2008, at 12:55 PM, William Herrin wrote:
> > There was some related work on ARIN PPML last year. The rough numbers
> > suggested that the attributable economic cost of one IPv4 prefix in
> > the DFZ (whether PI, PA or TE) was then in the neighborhood of $8000
> > USD per year.
>
> I haven't seen that work, but I am guessing this number is an
> aggregate (i.e. every cost to everyone on the 'Net combined), not per-
> network? See, I'm just looking at that TWO BILLION DOLLARS PER YEAR
> number and thinking to myself, "um, yeah, right". :)

Patrick,

That was a worldwide total, yes. The cost per prefix per router is
obviously only measured in cents per year.

You do know that Cisco's sales are north of $20B per year, right?
Juniper, which sells few products that aren't DFZ routers, also posts
annual revenues well north of $1B.


> Feel free to explain how confused I am.  (But be warned, I am not
> going to believe it costs $2B/year to run a multi-homed network with
> two full feeds. :)

The thread started here:
http://lists.arin.net/pipermail/ppml/2007-September/008927.html
It was originally an argument of about the cost of doing PI for IPv6,
which according to Cisco product literature consumes twice the amount
of space in the FIB as routes for IPv4.

I encourage you to critique the numbers and then add them up for yourself.

Regards,
Bill Herrin


-- 
William D. Herrin  [EMAIL PROTECTED]  [EMAIL PROTECTED]
3005 Crane Dr.Web: 
Falls Church, VA 22042-3004