Re: transit and peering costs projections

[Dave Taht]

On Sat, Oct 14, 2023 at 9:12 PM Tim Burke  wrote:
>
> It’s better for customer experience to keep it local instead of adding 200 
> miles to the route. All of the competition hauls all of their traffic up to 
> Dallas, so we easily have a nice 8-10ms latency advantage by keeping transit 
> and peering as close to the customer as possible.
>
> Plus, you can’t forget to mention another ~$10k MRC per pair in DF costs to 
> get up to Dallas, not including colo, that we can spend on more transit or 
> better gear!

Texas's BEAD funding and broadband offices are looking for proposals
and seem to have dollars to spend. I have spent much of the past few
years attempting to convince these entities that what was often more
needed was better, more local IXPs. Have you reached out to them?


> On Oct 14, 2023, at 23:03, Ryan Hamel  wrote:
>
> 
> Why not place the routers in Dallas, aggregate the transit, IXP, and PNI's 
> there, and backhaul it over redundant dark fiber with DWDM waves or 400G 
> OpenZR?
>
> Ryan
>
> 
> From: NANOG  on behalf of Tim Burke 
> 
> Sent: Saturday, October 14, 2023 8:45 PM
> To: Dave Taht 
> Cc: Network Neutrality is back! Let´s make the technical aspects heard this 
> time! ; libreqos 
> ; NANOG 
> Subject: Re: transit and peering costs projections
>
> Caution: This is an external email and may be malicious. Please take care 
> when clicking links or opening attachments.
>
>
> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
> good bit less than $900 on the wholesale market.
>
> Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
> costing as much as transit in many cases after you factor in loop costs.
>
> For example, in the Houston market (one of the largest and fastest growing 
> regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
> several thousand for a 100g wave, plus several thousand for a 100g port on 
> one of those major IXes. Or, a better option, we can get a 100g flat internet 
> transit for just a little bit more.
>
> Fortunately, for us as an eyeball network, there are a good number of major 
> content networks that are allowing for private peering in markets like 
> Houston for just the cost of a cross connect and a QSFP if you’re in the 
> right DC, with Google and some others being the outliers.
>
> So for now, we'll keep paying for transit to get to the others (since it’s 
> about as much as transporting IXP from Dallas), and hoping someone at Google 
> finally sees Houston as more than a third rate city hanging off of Dallas. 
> Or… someone finally brings a worthwhile IX to Houston that gets us more than 
> peering to Kansas City. Yeah, I think the former is more likely. 😊
>
> See y’all in San Diego this week,
> Tim
>
> On Oct 14, 2023, at 18:04, Dave Taht  wrote:
> >
> > This set of trendlines was very interesting. Unfortunately the data
> > stops in 2015. Does anyone have more recent data?
> >
> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D&reserved=0
> >
> > I believe a gbit circuit that an ISP can resell still runs at about
> > $900 - $1.4k (?) in the usa? How about elsewhere?
> >
> > ...
> >
> > I am under the impression that many IXPs remain very successful,
> > states without them suffer, and I also find the concept of doing micro
> > IXPs at the city level, appealing, and now achievable with cheap gear.
> > Finer grained cross connects between telco and ISP and IXP would lower
> > latencies across town quite hugely...
> >
> > PS I hear ARIN is planning on dropping the price for, and bundling 3
> > BGP AS numbers at a time, as of the end of this year, also.
> >
> >
> >
> > --
> > Oct 30: 
> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=0
> > Dave Täht CSO, LibreQos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: transit and peering costs projections

[Tim Burke]

It’s better for customer experience to keep it local instead of adding 200 
miles to the route. All of the competition hauls all of their traffic up to 
Dallas, so we easily have a nice 8-10ms latency advantage by keeping transit 
and peering as close to the customer as possible.

Plus, you can’t forget to mention another ~$10k MRC per pair in DF costs to get 
up to Dallas, not including colo, that we can spend on more transit or better 
gear!

On Oct 14, 2023, at 23:03, Ryan Hamel  wrote:


Why not place the routers in Dallas, aggregate the transit, IXP, and PNI's 
there, and backhaul it over redundant dark fiber with DWDM waves or 400G OpenZR?

Ryan


From: NANOG  on behalf of Tim Burke 

Sent: Saturday, October 14, 2023 8:45 PM
To: Dave Taht 
Cc: Network Neutrality is back! Let´s make the technical aspects heard this 
time! ; libreqos 
; NANOG 
Subject: Re: transit and peering costs projections

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.


I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
costing as much as transit in many cases after you factor in loop costs.

For example, in the Houston market (one of the largest and fastest growing 
regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
several thousand for a 100g wave, plus several thousand for a 100g port on one 
of those major IXes. Or, a better option, we can get a 100g flat internet 
transit for just a little bit more.

Fortunately, for us as an eyeball network, there are a good number of major 
content networks that are allowing for private peering in markets like Houston 
for just the cost of a cross connect and a QSFP if you’re in the right DC, with 
Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s 
about as much as transporting IXP from Dallas), and hoping someone at Google 
finally sees Houston as more than a third rate city hanging off of Dallas. Or… 
someone finally brings a worthwhile IX to Houston that gets us more than 
peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht  wrote:
>
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D&reserved=0
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: 
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=0
> Dave Täht CSO, LibreQos

Re: transit and peering costs projections

[Ryan Hamel]

Why not place the routers in Dallas, aggregate the transit, IXP, and PNI's 
there, and backhaul it over redundant dark fiber with DWDM waves or 400G OpenZR?

Ryan


From: NANOG  on behalf of Tim Burke 

Sent: Saturday, October 14, 2023 8:45 PM
To: Dave Taht 
Cc: Network Neutrality is back! Let´s make the technical aspects heard this 
time! ; libreqos 
; NANOG 
Subject: Re: transit and peering costs projections

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.


I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
costing as much as transit in many cases after you factor in loop costs.

For example, in the Houston market (one of the largest and fastest growing 
regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
several thousand for a 100g wave, plus several thousand for a 100g port on one 
of those major IXes. Or, a better option, we can get a 100g flat internet 
transit for just a little bit more.

Fortunately, for us as an eyeball network, there are a good number of major 
content networks that are allowing for private peering in markets like Houston 
for just the cost of a cross connect and a QSFP if you’re in the right DC, with 
Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s 
about as much as transporting IXP from Dallas), and hoping someone at Google 
finally sees Houston as more than a third rate city hanging off of Dallas. Or… 
someone finally brings a worthwhile IX to Houston that gets us more than 
peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht  wrote:
>
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D&reserved=0
>
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
>
> ...
>
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
>
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
>
>
>
> --
> Oct 30: 
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html&data=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D&reserved=0
> Dave Täht CSO, LibreQos

Re: transit and peering costs projections

[Tim Burke]

I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
good bit less than $900 on the wholesale market.

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
costing as much as transit in many cases after you factor in loop costs. 

For example, in the Houston market (one of the largest and fastest growing 
regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
several thousand for a 100g wave, plus several thousand for a 100g port on one 
of those major IXes. Or, a better option, we can get a 100g flat internet 
transit for just a little bit more. 

Fortunately, for us as an eyeball network, there are a good number of major 
content networks that are allowing for private peering in markets like Houston 
for just the cost of a cross connect and a QSFP if you’re in the right DC, with 
Google and some others being the outliers.

So for now, we'll keep paying for transit to get to the others (since it’s 
about as much as transporting IXP from Dallas), and hoping someone at Google 
finally sees Houston as more than a third rate city hanging off of Dallas. Or… 
someone finally brings a worthwhile IX to Houston that gets us more than 
peering to Kansas City. Yeah, I think the former is more likely. 😊

See y’all in San Diego this week,
Tim

On Oct 14, 2023, at 18:04, Dave Taht  wrote:
> 
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
> 
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
> 
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
> 
> ...
> 
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
> 
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
> 
> 
> 
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos

Re: [NNagain] transit and peering costs projections

[Dave Cohen]

I’m a couple years removed from dealing with this on the provider side but the 
focus has shifted rapidly to adding core capacity and large capacity ports to 
the extent that smaller capacity ports like 1 Gbps aren’t going to see much 
more price compression. Cost per bit will come down at higher tiers but there 
simply isn’t enough focus at lower levels at the hardware providers to afford 
carriers more price compression at 1 Gbps, even 10 Gbps. I would expect further 
price compression in access costs but not really in transit costs below 10 
Gbps. 

In general I agree that IXs continue to proliferate relative to quantity, 
throughput and geographic reach, almost to the degree that mainland Europe has 
been covered for years. In my home market of Atlanta, I’m aware of at least 
four IXs that have been established here or entered the market in the last 
three years - there were only two major ones prior to that. This is a net 
positive for a wide variety of reasons but I don’t think it’s created much of 
an impact in terms of pulling down transit prices. There are a few reasons for 
this, but primarily because that growth hasn’t really displaced transit demand 
(at least in my view) and has really been more about a relatively stable set of 
IX participants creating more resiliency and driving other performance 
improvements in that leg of the peering ecosystem. 

Dave Cohen
craetd...@gmail.com

> On Oct 14, 2023, at 7:02 PM, Dave Taht via Nnagain 
>  wrote:
> 
> This set of trendlines was very interesting. Unfortunately the data
> stops in 2015. Does anyone have more recent data?
> 
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
> 
> I believe a gbit circuit that an ISP can resell still runs at about
> $900 - $1.4k (?) in the usa? How about elsewhere?
> 
> ...
> 
> I am under the impression that many IXPs remain very successful,
> states without them suffer, and I also find the concept of doing micro
> IXPs at the city level, appealing, and now achievable with cheap gear.
> Finer grained cross connects between telco and ISP and IXP would lower
> latencies across town quite hugely...
> 
> PS I hear ARIN is planning on dropping the price for, and bundling 3
> BGP AS numbers at a time, as of the end of this year, also.
> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> ___
> Nnagain mailing list
> nnag...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain

transit and peering costs projections

[Dave Taht]

This set of trendlines was very interesting. Unfortunately the data
stops in 2015. Does anyone have more recent data?

https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php

I believe a gbit circuit that an ISP can resell still runs at about
$900 - $1.4k (?) in the usa? How about elsewhere?

...

I am under the impression that many IXPs remain very successful,
states without them suffer, and I also find the concept of doing micro
IXPs at the city level, appealing, and now achievable with cheap gear.
Finer grained cross connects between telco and ISP and IXP would lower
latencies across town quite hugely...

PS I hear ARIN is planning on dropping the price for, and bundling 3
BGP AS numbers at a time, as of the end of this year, also.



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: maximum ipv4 bgp prefix length of /24 ?

[Matthew Petach]

On Sat, Oct 14, 2023 at 2:37 PM John Kristoff  wrote:

> On Sat, 14 Oct 2023 13:59:11 -0700
> Matthew Petach  wrote:
>
> > That last report shows that only half of the top 1000 websites on the
> > Alexa ranking support IPv6.
>
> The Alexa ranking is no longer maintained.  ISOC had a recent article
> talking about just this:
>
>   <
> https://pulse.internetsociety.org/blog/do-half-of-the-most-popular-websites-use-ipv6
> >
>
> John
>

Good to know, John, though it doesn't change the underlying issue; as the
Oct 12th Pulse report from your link says,
"If we look at Figure 2, we can see nearly half of the top 1,000 websites
are IPv6 capable."

which basically says things haven't moved much since the last
Alexa-rank-based measurements were taken.

Thank you for the pointer to more-up-to-date data!

Matt

Re: maximum ipv4 bgp prefix length of /24 ?

[Matthew Petach]

On Wed, Oct 11, 2023 at 1:53 PM Mark Andrews  wrote:

> > On 12 Oct 2023, at 06:51, Delong.com  wrote:
>
> > The point here is that at some point, even with translation, we run out
> of IPv4 addresses to use for this purpose. What then?
>
> You deliver the Internet over IPv6.  A really large functional Internet
> exists today if you only have IPv6.  It is only getting bigger.  Lots of
> (the majority?) of CDNs deliver content over IPv6.  Lots of companies
> outsource their SMTP to dual stacked service providers so that email still
> gets through.
> After 20 years there is no excuse for ISPs failing to deliver IPv6.  If
> you have to you, outsource your NAT64, DS-Lite transition service to
> someone that has IPv4.  I’m surprised that it isn’t common today.


While you claim "there is no excuse for ISPs failing to deliver IPv6", the
reality is that many ISPs don't support fully functional IPv6 deployments.
There are far too many networks that allocate a single /64 for a wireless
customer, and ignore DHCPv6-PD requests.  There's a reason that IPv6-relay
functionality in OpenWRT is so widespread--because even when ISPs "support"
IPv6, they often do so poorly, leading to awkward hacks like relaying the
same /64 downstream through intervening routers.
I'm all for the eventual success of IPv6, but at the moment, we're really
not there.

But the bigger point is that there's still big chunks of the content side
that aren't reachable via IPv6.
https://www.6connect.com/blog/ipv6-progress-report-top-sites-2019/
http://www.delong.com/ipv6_alexa500.html
https://whynoipv6.com/

That last report shows that only half of the top 1000 websites on the Alexa
ranking support IPv6.  So we're a long way away from being able to simply
say "You deliver the Internet over IPv6."

Your last two sentences are exactly what I stated as a business proposition
earlier.  You said (fixing the typos):
"If you have to, you outsource your NAT64, DS-Lite transition service to
someone that has IPv4.  I’m surprised that it isn’t common today."

In a world where only half the content sites are reachable via IPv6, and
IPv4 address space is exhausted, that requirement to outsource NAT64
functionality is becoming more and more a business reality going forward.

Can you list any company today that provides an outsourced NAT64
translation service?
The only one I'm aware of is the one Kasper Dupont is running, and he's got
a very clear warning that it's not suitable for high-volume use.

I can't help but see an up-and-coming demand for services to fill this
need, as it's clear Kasper's setup isn't going to handle the load for all
the IPv6-only networks that decide there's actually content they want to
get to in the other half of the Alexa top 1000 sites that don't support
IPv6.  I'm enjoying being retired, but seeing a future demand with nobody
stepping up to fulfill that demand is almost enticing enough to be worth
un-retiring for to build out...

Thanks!

Matt