Re: transit and peering costs projections

[Matthew Petach]

On Sun, Oct 15, 2023 at 9:47 AM Dave Taht  wrote:

> [...]
> The three forms of traffic I care most about are voip, gaming, and
> videoconferencing, which are rewarding to have at lower latencies.
> When I was a kid, we had switched phone networks, and while the sound
> quality was poorer than today, the voice latency cross-town was just
> like "being there". Nowadays we see 500+ms latencies for this kind of
> traffic.
>

When you were a kid, the cost of voice calls across town were completely
dwarfed by the cost of long distance calls, which were insane by today's
standards.  But let's take the $10/month local-only dialtone fee from 1980;
a typical household would spend less than 600 minutes a month on local
calls,
for a per-minute cost for local calls of about 1.6 cents/minute.
(data from https://babel.hathitrust.org/cgi/pt?id=umn.319510029171372&seq=75
)

Each call would use up a single trunk line--today, we would think of that
as an
ISDN BRI at 64Kbits.  Doing the math, that meant on average you were using
64Kbit/sec*600minutes*60sec/min or 2304000Kbit per month (2.3 Gbit/month).

A 1Mbit/sec circuit, running constantly, has a capacity to transfer
2592Gbit/month.
So, a typical household used about 1/1000th of a 1Mbit/sec circuit, on
average,
but paid about $10/month for that.  That works out to a comparative cost of
$10,000/Mbit/month in revenue from those local voice calls.

You can afford to put in a *LOT* of "just like "being there""
infrastructure when
 you're charging your customers the equivalent of $10,000/month per Mbit to
talk across town.  Remember, this isn't adding in any long-distance charges,
this is *just* for you to ring up Aunt Maude on the other side of town to
ask when
the bake sale starts on Saturday.  So, that revenue is going into covering
the costs of backhaul to the local IXP, and to your ports on the local IXP,
to put it into modern terms.


> As to how to make calls across town work that well again, cost-wise, I
> do not know, but the volume of traffic that would be better served by
> these interconnects quite low, respective to the overall gains in
> lower latency experiences for them.
>

If you can figure out how to charge your customers equivalent pricing
again today, you'll have no trouble getting those calls across town to
work that well again.
Unfortunately, the consumers have gotten used to much lower
prices, and it's really, really hard to stuff the cat back into the
genie bottle again, to bludgeon a dead metaphor.
Not to mention customers have gotten much more used to the
smaller world we live in today, where everything IP is considered "local",
and you won't find many willing customers to pay a higher price for
communicating with far-away websites.  Good luck getting customers
to sign up for split contracts, with one price for talking to the local IXP
in town, and a different, more expensive price to send traffic outside
the city to some far-away place like Prineville, OR!  ;)

 I think we often forget just how much of a massive inversion the
communications industry has undergone; back in the 80s, when
I started working in networking, everything was DS0 voice channels,
and data was just a strange side business that nobody in the telcos
really understood or wanted to sell to.  At the time, the volume of money
being raked in from those DS0/VGE channels was mammoth compared
to the data networking side; we weren't even a rounding error.  But as the
roles reversed and the pyramid inverted, the data networking costs didn't
rise to meet the voice costs (no matter how hard the telcos tried to push
VGE-mileage-based pricing models!
-- see https://transition.fcc.gov/form477/FVS/definitions_fvs.pdf)
Instead, once VoIP became possible, the high-revenue voice circuits
got pillaged, with more and more of the traffic being pulled off over to
the cheaper data side, until even internally the telcos saw the writing
on the wall, and started to move their trunked voice traffic over to IP
as well.
But as we moved away from the SS7-based signalling, with explicit
information about the locality of the destination exchange giving way
to more generic IP datagrams, the distinction of "local" versus
"long-distance"
became less meaningful, outside the regulatory tariff domain.
When everything is IP datagrams, making a call from you to a person on
the other side of town may just as easily be exchanged at an exchange point
1,000 miles away as it would be locally in town, depending upon where your
carrier and your friend's carriers happen to be network co-incident.  So,
for
the consumer, the prices go drastically down, but in return, we accept
potentially higher latencies to exchange traffic that in earlier days would
have been kept strictly local.

Long-winded way of saying "yes, you can go back to how it was when
you were a kid--but can you get all your customers to agree to go back
to those pricing models as well?"   ^_^;

Thanks!

Matt


> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-m

Re: Add communities on direct routes in Juniper

[Owen DeLong via NANOG]

I believe you need to add the communities either on the import policy which 
pulls in the direct route or the export policy to the neighbor(s) you want to 
feed the communities to. 

Owen


> On Oct 15, 2023, at 05:51, Jason R. Rokeach via NANOG  wrote:
> 
> Hi Stanislav,
> I believe this is what you are looking for:
> 
> [edit]
> jcluser@Lothlorien-MX1# show | compare 
> [edit interfaces lo0 unit 0 family inet]
>address 10.0.0.0/32 { ... }
> +   address 5.5.5.5/32;
> [edit protocols bgp]
> -   export IPV4-STATIC;
> +   export [ IPV4-STATIC TAG-DIRECT ];
> [edit policy-options]
> +   policy-statement TAG-DIRECT {
> +   from {
> +   protocol direct;
> +   route-filter 5.5.5.5/32 exact;
> +   }
> +   then {
> +   community set MYCOMMUNITY;
> +   accept;
> +   }
> +   }
> [edit policy-options]
> +   community MYCOMMUNITY members 5:5;
> 
> [edit]
> jcluser@Lothlorien-MX1# commit 
> commit complete
> 
> [edit]
> jcluser@Lothlorien-MX1# run show route advertising-protocol bgp 172.19.0.2 
> detail | find 5.5.5.5 
> * 5.5.5.5/32 (1 entry, 1 announced)
> BGP group RR-LOADBALANCER type External
> Nexthop: Self
> AS path: [65000] I 
> Communities: 5:5
> 
> Regards,
> Jason R. Rokeach
> 
> 
> --- Original Message ---
>> On Sunday, October 15th, 2023 at 8:29 AM, Saku Ytti - saku at ytti.fi 
>>  wrote:
>> 
>> 
>> Unfortunately not yet, as far as I know. Long time ago I gave this to
>> my account team
>> 
>> Title: Direct routes must support tag and or community
>> Platform: Trio, priority MX80, MPC2
>> JunOS: 12.4Rx
>> Command: 'set interfaxe ge-4/2.0 family inet address 10.42.42.1/24
>> tag|community X'
>> JTAC: n/a
>> ER:
>> - Router must be able to add tags communities to direct routes directly, like
>> it does for static routes
>> 
>> Usage Case:
>> Trivial way to signal route information to BGP. Often tag/community is used
>> by service providers to singal 'this is PI/PA prefix, leak it to internet' or
>> 'this is backup route, reduce its MED'. However for some reason it is only
>> supported for static routes, while usage scenario and benefits are exactly 
>> the
>> same for direct routes.
>> 
>> On Sun, 15 Oct 2023 at 15:27, Stanislav Datskevych via NANOG
>> nanog@nanog.org wrote:
>> 
>>> Dear all,
>>> 
>>> Is there a way to add BGP communities on direct (interface) routes in 
>>> Junipers? The task looks to be simple but the solution eludes me.
>>> In Cisco/Arista, for example, I could use "network 192.0.2.0/24 route-map 
>>> ".
>>> 
>>> In Juniper it seems to be impossible. I even tried putting interface-routes 
>>> into rib-group with an import policy.
>>> But it seems the import policy only works on importing routes into 
>>> Secondary routing tables (e.g. inet.50), and not into the Primary one 
>>> (inet.0).
>>> 
>>> I know it's possible to add communities on later stage while announcing 
>>> networks to peers, in [protocols bgp group  export]. But I'd better 
>>> slap the community on the routes right when they're imported into RIB, not 
>>> when they announced to peers.
>>> 
>>> Thanks in advance.
>> 
>> 
>> 
>> --
>> ++ytti
> 

Re: transit and peering costs projections

[Tim Burke]

I agree, but there are fortunately several large content networks that have had 
the forethought to put their stuff in Houston - Meta, Fastly, Akamai, AWS just 
to name a few… There is enough of a need to warrant those other networks having 
a presence, so hopefully it’s just a matter of time before other content 
networks jump in too.

Those 4 (plus Google cache fills) make up a huge majority of our transit usage, 
so at least we’ll get a majority of it peered off after we get these PNI’s 
stood up. And yes, I will continue to push for Google to light something up in 
Houston. 🤣

On Oct 15, 2023, at 11:33, Tom Beecher  wrote:


So for now, we'll keep paying for transit to get to the others (since it’s 
about as much as transporting IXP from Dallas), and hoping someone at Google 
finally sees Houston as more than a third rate city hanging off of Dallas. Or… 
someone finally brings a worthwhile IX to Houston that gets us more than 
peering to Kansas City. Yeah, I think the former is more likely. 😊

There is often a chicken/egg scenario here with the economics. As an eyeball 
network, your costs to build out and connect to Dallas are greater than your 
transit cost, so you do that. Totally fair.

However think about it from the content side. Say I want to build into to 
Houston. I have to put routers in, and a bunch of cache servers, so I have 
capital outlay , plus opex for space, power, IX/backhaul/transit costs. That's 
not cheap, so there's a lot of calculations that go into it. Is there enough 
total eyeball traffic there to make it worth it? Is saving 8-10ms enough of a 
performance boost to justify the spend? What are the long term trends in that 
market? These answers are of course different for a company running their own 
CDN vs the commercial CDNs.

I don't work for Google and obviously don't speak for them, but I would suspect 
that they're happy to eat a 8-10ms performance hit to serve from Dallas , 
versus the amount of capital outlay to build out there right now.

On Sat, Oct 14, 2023 at 11:47 PM Tim Burke mailto:t...@mid.net>> 
wrote:
I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
costing as much as transit in many cases after you factor in loop costs.

For example, in the Houston market (one of the largest and fastest growing 
regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
several thousand for a 100g wave, plus several thousand for a 100g port on one 
of those major IXes. Or, a better option, we can get a 100g flat internet 
transit for just a little bit more.

Fortunately, for us as an eyeball network, there are a good number of major 
content networks that are allowing for private peering in markets like Houston 
for just the cost of a cross connect and a QSFP if you’re in the right DC, with 
Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s 
about as much as transporting IXP from Dallas), and hoping someone at Google 
finally sees Houston as more than a third rate city hanging off of Dallas. Or… 
someone finally brings a worthwhile IX to Houston that gets us more than 
peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht 
mailto:dave.t...@gmail.com>> wrote:
>
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos

Re: ARIN whois contact abuse from ipv4depot aka Silicon Desert International Inc

[Tim Burke]

I’d vote for whoever promises to perma ban Cogent and all of these other clowns 
from access WHOIS data. Someone get on that!

> On Oct 13, 2023, at 19:33, Randy Bush  wrote:
> 
> i received an arin board electioneering "vote for me" today.  i guess
> now i have to go vote against then.
> 
> randy

Re: transit and peering costs projections

[Dave Taht]

For starters I would like to apologize for cc-ing both nanog and my
new nn list. (I will add sender filters)

A bit more below.

On Sun, Oct 15, 2023 at 9:32 AM Tom Beecher  wrote:
>>
>> So for now, we'll keep paying for transit to get to the others (since it’s 
>> about as much as transporting IXP from Dallas), and hoping someone at Google 
>> finally sees Houston as more than a third rate city hanging off of Dallas. 
>> Or… someone finally brings a worthwhile IX to Houston that gets us more than 
>> peering to Kansas City. Yeah, I think the former is more likely. 😊
>
>
> There is often a chicken/egg scenario here with the economics. As an eyeball 
> network, your costs to build out and connect to Dallas are greater than your 
> transit cost, so you do that. Totally fair.
>
> However think about it from the content side. Say I want to build into to 
> Houston. I have to put routers in, and a bunch of cache servers, so I have 
> capital outlay , plus opex for space, power, IX/backhaul/transit costs. 
> That's not cheap, so there's a lot of calculations that go into it. Is there 
> enough total eyeball traffic there to make it worth it? Is saving 8-10ms 
> enough of a performance boost to justify the spend? What are the long term 
> trends in that market? These answers are of course different for a company 
> running their own CDN vs the commercial CDNs.
>
> I don't work for Google and obviously don't speak for them, but I would 
> suspect that they're happy to eat a 8-10ms performance hit to serve from 
> Dallas , versus the amount of capital outlay to build out there right now.

The three forms of traffic I care most about are voip, gaming, and
videoconferencing, which are rewarding to have at lower latencies.
When I was a kid, we had switched phone networks, and while the sound
quality was poorer than today, the voice latency cross-town was just
like "being there". Nowadays we see 500+ms latencies for this kind of
traffic.

As to how to make calls across town work that well again, cost-wise, I
do not know, but the volume of traffic that would be better served by
these interconnects quite low, respective to the overall gains in
lower latency experiences for them.



>
> On Sat, Oct 14, 2023 at 11:47 PM Tim Burke  wrote:
>>
>> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
>> good bit less than $900 on the wholesale market.
>>
>> Sadly, IXP’s are seemingly turning into a pay to play game, with rates 
>> almost costing as much as transit in many cases after you factor in loop 
>> costs.
>>
>> For example, in the Houston market (one of the largest and fastest growing 
>> regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
>> several thousand for a 100g wave, plus several thousand for a 100g port on 
>> one of those major IXes. Or, a better option, we can get a 100g flat 
>> internet transit for just a little bit more.
>>
>> Fortunately, for us as an eyeball network, there are a good number of major 
>> content networks that are allowing for private peering in markets like 
>> Houston for just the cost of a cross connect and a QSFP if you’re in the 
>> right DC, with Google and some others being the outliers.
>>
>> So for now, we'll keep paying for transit to get to the others (since it’s 
>> about as much as transporting IXP from Dallas), and hoping someone at Google 
>> finally sees Houston as more than a third rate city hanging off of Dallas. 
>> Or… someone finally brings a worthwhile IX to Houston that gets us more than 
>> peering to Kansas City. Yeah, I think the former is more likely. 😊
>>
>> See y’all in San Diego this week,
>> Tim
>>
>> On Oct 14, 2023, at 18:04, Dave Taht  wrote:
>> >
>> > This set of trendlines was very interesting. Unfortunately the data
>> > stops in 2015. Does anyone have more recent data?
>> >
>> > https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>> >
>> > I believe a gbit circuit that an ISP can resell still runs at about
>> > $900 - $1.4k (?) in the usa? How about elsewhere?
>> >
>> > ...
>> >
>> > I am under the impression that many IXPs remain very successful,
>> > states without them suffer, and I also find the concept of doing micro
>> > IXPs at the city level, appealing, and now achievable with cheap gear.
>> > Finer grained cross connects between telco and ISP and IXP would lower
>> > latencies across town quite hugely...
>> >
>> > PS I hear ARIN is planning on dropping the price for, and bundling 3
>> > BGP AS numbers at a time, as of the end of this year, also.
>> >
>> >
>> >
>> > --
>> > Oct 30: 
>> > https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> > Dave Täht CSO, LibreQos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: transit and peering costs projections

[Tom Beecher]

>
> So for now, we'll keep paying for transit to get to the others (since it’s
> about as much as transporting IXP from Dallas), and hoping someone at
> Google finally sees Houston as more than a third rate city hanging off of
> Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us
> more than peering to Kansas City. Yeah, I think the former is more
> likely. 😊
>

There is often a chicken/egg scenario here with the economics. As an
eyeball network, your costs to build out and connect to Dallas are greater
than your transit cost, so you do that. Totally fair.

However think about it from the content side. Say I want to build into to
Houston. I have to put routers in, and a bunch of cache servers, so I have
capital outlay , plus opex for space, power, IX/backhaul/transit costs.
That's not cheap, so there's a lot of calculations that go into it. Is
there enough total eyeball traffic there to make it worth it? Is saving
8-10ms enough of a performance boost to justify the spend? What are the
long term trends in that market? These answers are of course different for
a company running their own CDN vs the commercial CDNs.

I don't work for Google and obviously don't speak for them, but I would
suspect that they're happy to eat a 8-10ms performance hit to serve from
Dallas , versus the amount of capital outlay to build out there right now.

On Sat, Oct 14, 2023 at 11:47 PM Tim Burke  wrote:

> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for
> a good bit less than $900 on the wholesale market.
>
> Sadly, IXP’s are seemingly turning into a pay to play game, with rates
> almost costing as much as transit in many cases after you factor in loop
> costs.
>
> For example, in the Houston market (one of the largest and fastest growing
> regions in the US!), we do not have a major IX, so to get up to Dallas it’s
> several thousand for a 100g wave, plus several thousand for a 100g port on
> one of those major IXes. Or, a better option, we can get a 100g flat
> internet transit for just a little bit more.
>
> Fortunately, for us as an eyeball network, there are a good number of
> major content networks that are allowing for private peering in markets
> like Houston for just the cost of a cross connect and a QSFP if you’re in
> the right DC, with Google and some others being the outliers.
>
> So for now, we'll keep paying for transit to get to the others (since it’s
> about as much as transporting IXP from Dallas), and hoping someone at
> Google finally sees Houston as more than a third rate city hanging off of
> Dallas. Or… someone finally brings a worthwhile IX to Houston that gets us
> more than peering to Kansas City. Yeah, I think the former is more likely.
> 😊
>
> See y’all in San Diego this week,
> Tim
>
> On Oct 14, 2023, at 18:04, Dave Taht  wrote:
> >
> > This set of trendlines was very interesting. Unfortunately the data
> > stops in 2015. Does anyone have more recent data?
> >
> >
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
> >
> > I believe a gbit circuit that an ISP can resell still runs at about
> > $900 - $1.4k (?) in the usa? How about elsewhere?
> >
> > ...
> >
> > I am under the impression that many IXPs remain very successful,
> > states without them suffer, and I also find the concept of doing micro
> > IXPs at the city level, appealing, and now achievable with cheap gear.
> > Finer grained cross connects between telco and ISP and IXP would lower
> > latencies across town quite hugely...
> >
> > PS I hear ARIN is planning on dropping the price for, and bundling 3
> > BGP AS numbers at a time, as of the end of this year, also.
> >
> >
> >
> > --
> > Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
>

Re: transit and peering costs projections

[John Kristoff]

On Sat, 14 Oct 2023 16:01:54 -0700
Dave Taht  wrote:

> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?

This may be of interest:

  Peering Costs and Fees
  

John

Re: transit and peering costs projections

[Tim Burke]

I’ve found that most of the CDNs that matter are in one facility in Houston, 
the Databank West (formerly Cyrus One) campus. We are about to light up a POP 
there so we’ll at least be able to get PNIs to them. There is even an IX in the 
facility, but it’s relatively small (likely because the operator wants 
near-transit pricing to get on it) so we’ll just PNI what we can for now.

On Oct 15, 2023, at 08:50, Mike Hammett  wrote:


Houston is tricky as due to it's geographic scope, it's quite expensive to 
build an IX that goes into enough facilities to achieve meaningful scale. CDN 1 
is in facility A. CDN 2 in facility B. CDN 3 is in facility C. When I last 
looked, it was about 80 driving miles to have a dark fiber ring that 
encompassed all of the facilities one would need to be in.



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Tim Burke" 
To: "Dave Taht" 
Cc: "Network Neutrality is back! Let´s make the technical aspects heard this 
time!" , "libreqos" 
, "NANOG" 
Sent: Saturday, October 14, 2023 10:45:47 PM
Subject: Re: transit and peering costs projections

I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
costing as much as transit in many cases after you factor in loop costs.

For example, in the Houston market (one of the largest and fastest growing 
regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
several thousand for a 100g wave, plus several thousand for a 100g port on one 
of those major IXes. Or, a better option, we can get a 100g flat internet 
transit for just a little bit more.

Fortunately, for us as an eyeball network, there are a good number of major 
content networks that are allowing for private peering in markets like Houston 
for just the cost of a cross connect and a QSFP if you’re in the right DC, with 
Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s 
about as much as transporting IXP from Dallas), and hoping someone at Google 
finally sees Houston as more than a third rate city hanging off of Dallas. Or… 
someone finally brings a worthwhile IX to Houston that gets us more than 
peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht  wrote:
>
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos

Re: [LibreQoS] transit and peering costs projections

[Tim Burke]

Man, I wanna know where you’re getting 100g transit for $4500 a month! Even 
someone as fly by night as Cogent wants almost double that, unfortunately.

On Oct 15, 2023, at 07:43, Jim Troutman  wrote:


Transit 1G wholesale in the right DCs is below $500 per port.  10gigE full port 
can be had around $1k-1.5k month on long term deals from multiple sources.   
100g IP transit ports start around $4k.

The cost of transport (dark or wavelength) is generally at least as much as the 
IP transit cost, and usually more in underserved markets.  In the northeast it 
is very hard to get 10GigE wavelengths below $2k/month to any location, and is 
generally closer to $3k.  100g waves are starting around $4k and go up a lot.

Pricing has come down somewhat over time, but not as fast as transit prices.   
6 years ago a 10Gig wave to Boston from Maine would be about $5k/month. Today 
about $2800.

With the cost of XCs in data centers and transport costs, you generally don’t 
want to go beyond 2x10gigE before jumping to 100.

On Sat, Oct 14, 2023 at 19:02 Dave Taht via LibreQoS 
mailto:libre...@lists.bufferbloat.net>> wrote:
This set of trendlines was very interesting. Unfortunately the data
stops in 2015. Does anyone have more recent data?

https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php

I believe a gbit circuit that an ISP can resell still runs at about
$900 - $1.4k (?) in the usa? How about elsewhere?

...

I am under the impression that many IXPs remain very successful,
states without them suffer, and I also find the concept of doing micro
IXPs at the city level, appealing, and now achievable with cheap gear.
Finer grained cross connects between telco and ISP and IXP would lower
latencies across town quite hugely...

PS I hear ARIN is planning on dropping the price for, and bundling 3
BGP AS numbers at a time, as of the end of this year, also.



--
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos
___
LibreQoS mailing list
libre...@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos

Re: transit and peering costs projections

[Mike Hammett]

Houston is tricky as due to it's geographic scope, it's quite expensive to 
build an IX that goes into enough facilities to achieve meaningful scale. CDN 1 
is in facility A. CDN 2 in facility B. CDN 3 is in facility C. When I last 
looked, it was about 80 driving miles to have a dark fiber ring that 
encompassed all of the facilities one would need to be in. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tim Burke"  
To: "Dave Taht"  
Cc: "Network Neutrality is back! Let´s make the technical aspects heard this 
time!" , "libreqos" 
, "NANOG"  
Sent: Saturday, October 14, 2023 10:45:47 PM 
Subject: Re: transit and peering costs projections 

I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
good bit less than $900 on the wholesale market. 

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
costing as much as transit in many cases after you factor in loop costs. 

For example, in the Houston market (one of the largest and fastest growing 
regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
several thousand for a 100g wave, plus several thousand for a 100g port on one 
of those major IXes. Or, a better option, we can get a 100g flat internet 
transit for just a little bit more. 

Fortunately, for us as an eyeball network, there are a good number of major 
content networks that are allowing for private peering in markets like Houston 
for just the cost of a cross connect and a QSFP if you’re in the right DC, with 
Google and some others being the outliers. 

So for now, we'll keep paying for transit to get to the others (since it’s 
about as much as transporting IXP from Dallas), and hoping someone at Google 
finally sees Houston as more than a third rate city hanging off of Dallas. Or… 
someone finally brings a worthwhile IX to Houston that gets us more than 
peering to Kansas City. Yeah, I think the former is more likely. 😊 

See y’all in San Diego this week, 
Tim 

On Oct 14, 2023, at 18:04, Dave Taht  wrote: 
> 
> This set of trendlines was very interesting. Unfortunately the data 
> stops in 2015. Does anyone have more recent data? 
> 
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>  
> 
> I believe a gbit circuit that an ISP can resell still runs at about 
> $900 - $1.4k (?) in the usa? How about elsewhere? 
> 
> ... 
> 
> I am under the impression that many IXPs remain very successful, 
> states without them suffer, and I also find the concept of doing micro 
> IXPs at the city level, appealing, and now achievable with cheap gear. 
> Finer grained cross connects between telco and ISP and IXP would lower 
> latencies across town quite hugely... 
> 
> PS I hear ARIN is planning on dropping the price for, and bundling 3 
> BGP AS numbers at a time, as of the end of this year, also. 
> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html 
> Dave Täht CSO, LibreQos 

Re: transit and peering costs projections

[Mike Hammett]

I've seen some attempts to put an IX at every corner, but I don't think those 
efforts will be overly successful. 

It's still difficult to gain sufficient scale in NFL-sized cities. Big content 
won't join without big eyeballs (well, not the national-level guys because they 
almost never will). Big eyeballs just can't be bothered. Small guys don't move 
the needle enough. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Dave Taht"  
To: "Network Neutrality is back! Let´s make the technical aspects heard this 
time!" , "libreqos" 
, "NANOG"  
Sent: Saturday, October 14, 2023 6:01:54 PM 
Subject: transit and peering costs projections 

This set of trendlines was very interesting. Unfortunately the data 
stops in 2015. Does anyone have more recent data? 

https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
 

I believe a gbit circuit that an ISP can resell still runs at about 
$900 - $1.4k (?) in the usa? How about elsewhere? 

... 

I am under the impression that many IXPs remain very successful, 
states without them suffer, and I also find the concept of doing micro 
IXPs at the city level, appealing, and now achievable with cheap gear. 
Finer grained cross connects between telco and ISP and IXP would lower 
latencies across town quite hugely... 

PS I hear ARIN is planning on dropping the price for, and bundling 3 
BGP AS numbers at a time, as of the end of this year, also. 



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html 
Dave Täht CSO, LibreQos 

Re: Add communities on direct routes in Juniper

[Jason R. Rokeach via NANOG]

Hi Stanislav,
I believe this is what you are looking for:

[edit]
jcluser@Lothlorien-MX1# show | compare 
[edit interfaces lo0 unit 0 family inet]
address 10.0.0.0/32 { ... }
+   address 5.5.5.5/32;
[edit protocols bgp]
-   export IPV4-STATIC;
+   export [ IPV4-STATIC TAG-DIRECT ];
[edit policy-options]
+   policy-statement TAG-DIRECT {
+   from {
+   protocol direct;
+   route-filter 5.5.5.5/32 exact;
+   }
+   then {
+   community set MYCOMMUNITY;
+   accept;
+   }
+   }
[edit policy-options]
+   community MYCOMMUNITY members 5:5;

[edit]
jcluser@Lothlorien-MX1# commit 
commit complete

[edit]
jcluser@Lothlorien-MX1# run show route advertising-protocol bgp 172.19.0.2 
detail | find 5.5.5.5 
* 5.5.5.5/32 (1 entry, 1 announced)
 BGP group RR-LOADBALANCER type External
 Nexthop: Self
 AS path: [65000] I 
 Communities: 5:5

Regards,
Jason R. Rokeach


--- Original Message ---
On Sunday, October 15th, 2023 at 8:29 AM, Saku Ytti - saku at ytti.fi 
 wrote:


> Unfortunately not yet, as far as I know. Long time ago I gave this to
> my account team
> 
> Title: Direct routes must support tag and or community
> Platform: Trio, priority MX80, MPC2
> JunOS: 12.4Rx
> Command: 'set interfaxe ge-4/2.0 family inet address 10.42.42.1/24
> tag|community X'
> JTAC: n/a
> ER:
> - Router must be able to add tags communities to direct routes directly, like
> it does for static routes
> 
> Usage Case:
> Trivial way to signal route information to BGP. Often tag/community is used
> by service providers to singal 'this is PI/PA prefix, leak it to internet' or
> 'this is backup route, reduce its MED'. However for some reason it is only
> supported for static routes, while usage scenario and benefits are exactly the
> same for direct routes.
> 
> On Sun, 15 Oct 2023 at 15:27, Stanislav Datskevych via NANOG
> nanog@nanog.org wrote:
> 
> > Dear all,
> > 
> > Is there a way to add BGP communities on direct (interface) routes in 
> > Junipers? The task looks to be simple but the solution eludes me.
> > In Cisco/Arista, for example, I could use "network 192.0.2.0/24 route-map 
> > ".
> > 
> > In Juniper it seems to be impossible. I even tried putting interface-routes 
> > into rib-group with an import policy.
> > But it seems the import policy only works on importing routes into 
> > Secondary routing tables (e.g. inet.50), and not into the Primary one 
> > (inet.0).
> > 
> > I know it's possible to add communities on later stage while announcing 
> > networks to peers, in [protocols bgp group  export]. But I'd better 
> > slap the community on the routes right when they're imported into RIB, not 
> > when they announced to peers.
> > 
> > Thanks in advance.
> 
> 
> 
> --
> ++ytti

Re: [LibreQoS] transit and peering costs projections

[Jim Troutman]

Transit 1G wholesale in the right DCs is below $500 per port.  10gigE full
port can be had around $1k-1.5k month on long term deals from multiple
sources.   100g IP transit ports start around $4k.

The cost of transport (dark or wavelength) is generally at least as much as
the IP transit cost, and usually more in underserved markets.  In the
northeast it is very hard to get 10GigE wavelengths below $2k/month to any
location, and is generally closer to $3k.  100g waves are starting around
$4k and go up a lot.

Pricing has come down somewhat over time, but not as fast as transit
prices.   6 years ago a 10Gig wave to Boston from Maine would be about
$5k/month. Today about $2800.

With the cost of XCs in data centers and transport costs, you generally
don’t want to go beyond 2x10gigE before jumping to 100.

On Sat, Oct 14, 2023 at 19:02 Dave Taht via LibreQoS <
libre...@lists.bufferbloat.net> wrote:

> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
>
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> ___
> LibreQoS mailing list
> libre...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

Re: Add communities on direct routes in Juniper

[Saku Ytti]

Unfortunately not yet, as far as I know. Long time ago I gave this to
my account team

Title: Direct routes must support tag and or community
Platform:  Trio, priority MX80, MPC2
JunOS: 12.4Rx
Command:   'set interfaxe ge-4/2.0 family inet address 10.42.42.1/24
tag|community X'
JTAC:  n/a
ER:
  - Router must be able to add tags communities to direct routes directly, like
it does for static routes

Usage Case:
  Trivial way to signal route information to BGP. Often tag/community is used
  by service providers to singal 'this is PI/PA prefix, leak it to internet' or
  'this is backup route, reduce its MED'. However for some reason it is only
  supported for static routes, while usage scenario and benefits are exactly the
  same for direct routes.

On Sun, 15 Oct 2023 at 15:27, Stanislav Datskevych via NANOG
 wrote:
>
> Dear all,
>
> Is there a way to add BGP communities on direct (interface) routes in 
> Junipers? The task looks to be simple but the solution eludes me.
> In Cisco/Arista, for example, I could use "network 192.0.2.0/24 route-map 
> ".
>
> In Juniper it seems to be impossible. I even tried putting interface-routes 
> into rib-group with an import policy.
> But it seems the import policy only works on importing routes into Secondary 
> routing tables (e.g. inet.50), and not into the Primary one (inet.0).
>
> I know it's possible to add communities on later stage while announcing 
> networks to peers, in [protocols bgp group  export]. But I'd better 
> slap the community on the routes right when they're imported into RIB, not 
> when they announced to peers.
>
> Thanks in advance.
>


-- 
  ++ytti

Add communities on direct routes in Juniper

[Stanislav Datskevych via NANOG]

Dear all, 

Is there a way to add BGP communities on direct (interface) routes in Junipers? 
The task looks to be simple but the solution eludes me.
In Cisco/Arista, for example, I could use "network 192.0.2.0/24 route-map 
".

In Juniper it seems to be impossible. I even tried putting interface-routes 
into rib-group with an import policy.
But it seems the import policy only works on importing routes into Secondary 
routing tables (e.g. inet.50), and not into the Primary one (inet.0).

I know it's possible to add communities on later stage while announcing 
networks to peers, in [protocols bgp group  export]. But I'd better slap 
the community on the routes right when they're imported into RIB, not when they 
announced to peers.

Thanks in advance.

Re: transit and peering costs projections

[Bill Woodcock]

Exactly.  Speed x distance = cost.  This is _exactly_ why IXPs get set up.  To 
avoid backhauling bandwidth from Dallas, or wherever.  Loss, latency, 
out-of-order delivery, and jitter.  All lower when you source your bandwidth 
closer.

-Bill



> On Oct 15, 2023, at 06:12, Tim Burke  wrote:
> 
> It’s better for customer experience to keep it local instead of adding 200 
> miles to the route. All of the competition hauls all of their traffic up to 
> Dallas, so we easily have a nice 8-10ms latency advantage by keeping transit 
> and peering as close to the customer as possible.
> 
> Plus, you can’t forget to mention another ~$10k MRC per pair in DF costs to 
> get up to Dallas, not including colo, that we can spend on more transit or 
> better gear!
> 
>> On Oct 14, 2023, at 23:03, Ryan Hamel  wrote:
>> 
>>  Why not place the routers in Dallas, aggregate the transit, IXP, and PNI's 
>> there, and backhaul it over redundant dark fiber with DWDM waves or 400G 
>> OpenZR?
>> 
>> Ryan
>> 
>> From: NANOG  on behalf of Tim 
>> Burke 
>> Sent: Saturday, October 14, 2023 8:45 PM
>> To: Dave Taht 
>> Cc: Network Neutrality is back! Let´s make the technical aspects heard this 
>> time! ; libreqos 
>> ; NANOG 
>> Subject: Re: transit and peering costs projections   Caution: This is an 
>> external email and may be malicious. Please take care when clicking links or 
>> opening attachments.
>> 
>> 
>> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
>> good bit less than $900 on the wholesale market.
>> 
>> Sadly, IXP’s are seemingly turning into a pay to play game, with rates 
>> almost costing as much as transit in many cases after you factor in loop 
>> costs.
>> 
>> For example, in the Houston market (one of the largest and fastest growing 
>> regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
>> several thousand for a 100g wave, plus several thousand for a 100g port on 
>> one of those major IXes. Or, a better option, we can get a 100g flat 
>> internet transit for just a little bit more.
>> 
>> Fortunately, for us as an eyeball network, there are a good number of major 
>> content networks that are allowing for private peering in markets like 
>> Houston for just the cost of a cross connect and a QSFP if you’re in the 
>> right DC, with Google and some others being the outliers.
>> 
>> So for now, we'll keep paying for transit to get to the others (since it’s 
>> about as much as transporting IXP from Dallas), and hoping someone at Google 
>> finally sees Houston as more than a third rate city hanging off of Dallas. 
>> Or… someone finally brings a worthwhile IX to Houston that gets us more than 
>> peering to Kansas City. Yeah, I think the former is more likely. 😊
>> 
>> See y’all in San Diego this week,
>> Tim
>> 
>> On Oct 14, 2023, at 18:04, Dave Taht  wrote:
>> >
>> > This set of trendlines was very interesting. Unfortunately the data
>> > stops in 2015. Does anyone have more recent data?
>> >
>> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D&reserved=0
>> >
>> > I believe a gbit circuit that an ISP can resell still runs at about
>> > $900 - $1.4k (?) in the usa? How about elsewhere?
>> >
>> > ...
>> >
>> > I am under the impression that many IXPs remain very successful,
>> > states without them suffer, and I also find the concept of doing micro
>> > IXPs at the city level, appealing, and now achievable with cheap gear.
>> > Finer grained cross connects between telco and ISP and IXP would lower
>> > latencies across town quite hugely...
>> >
>> > PS I hear ARIN is planning on dropping the price for, and bundling 3
>> > BGP AS numbers at a time, as of the end of this year, also.
>> >
>> >
>> >
>> > --
>> > Oct 30: 
>> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=0
>> > Dave Täht CSO, LibreQos



signature.asc
Description: Message signed with OpenPGP

Re: transit and peering costs projections

[Bill Woodcock]

> On Oct 15, 2023, at 01:01, Dave Taht  wrote:
> I am under the impression that many IXPs remain very successful,

I know of 760 active IXPs, out of 1,148 total, so, over 31 years, two-thirds 
are still successful now.  Obviously they didn’t all start 31 years ago, they 
started on a gradually-accelerating curve.  I guess we could do the 
visualization to plot range of lifespans versus start dates, but we haven’t 
done that as yet.

> states without them suffer

Any populated area without one or more of them suffers by comparison with areas 
that do have them.  States, countries, cities, etc.  There are still a 
surprising number of whole countries that don’t yet have one.  We try to 
prioritize those in our work:

https://www.pch.net/ixp/summary

> I also find the concept of doing micro IXPs at the city level, appealing, and 
> now achievable with cheap gear.

This has always, by definition, been achievable, since it’s the only way any 
IXP has ever succeeded, really.  I mean, big sample set, bell curve, you can 
always find a few things out at the fringes to argue about, but the thing that 
allows an IXP to succeed is good APBDC, and the thing that most frequently 
kills IXPs is over-investment.  An expensive switch at the outset is a huge 
liability, and one of the things most likely to tank a startup IXP.  Notably, 
that doesn’t mean a switch that costs the IXP a lot of money: you can tank an 
IXP by donating an expensive switch for free.  Expensive switches have 
expensive maintenance, whether you’re paying for it or not.  Maintenance means 
down-time, and down-time raises APBDC, regardless of whether you’ve laid out 
cash in parallel with it.

> Finer grained cross connects between telco and ISP and IXP would lower 
> latencies across town quite hugely...

Of course, and that requires that they show up in the same building, ideally 
with an MMR.  The same places that work well for IXPs.  Interconnection 
basically just requires a lot of networks be present close to a population 
center.  Which always presents a little tension vis-a-vis datacenters, which 
profit immensely if there’s a successful IXP in them, but can never afford to 
locate themselves where IXPs would be most valuable, and don’t like to have to 
provide free backhaul to better IXP locations.

-Bill



signature.asc
Description: Message signed with OpenPGP