Re: Shared cache servers on an island's IXP

[Jérôme Nicolle]

Hi Nick,

Thanks for your remarks. It's actually an ongoing discussion.

Le 18/01/2024 à 18:24, Nick Hilliard a écrit :
two issues here: the smaller issue is that CDNs sometimes want their own 
routable IP address blocks, especially if they're connecting directly to 
the IXP, which usually means /24 in practice. It doesn't always happen, 
and sometimes the CDN is happy to use provider address space (i.e. IXP), 
or smaller address blocks. But it's something to note.


I'd rather have CDN use some of their anycast /24 to peer with the IX, 
with a back-end connectivity for their control-plane and back-feeding.


The bigger issue is: who pays the transit costs for the CDN's cache-fill 
requirements? CDNs typically won't pay for cache-fill for installations 
like this, and if one local ISP is pulling disproportionate quantities 
of data compared to other ISPs at the IXP, then this can cause problems 
unless there's an shared billing mechanism built in.


We're willing to provide a dedicated LAN, with routed access, to fill 
caches and administer the machines. It would be fully dissociated from 
the IXP though, unless we could find a way to make it work and as to 
meet extra requirements upon redundancy.


Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Re: Shared cache servers on an island's IXP

[Nick Hilliard]

Jérôme Nicolle wrote on 18/01/2024 14:38:
Those I'm nearly sure I could get, if I can pool caches amongst ISPs. 
The current constraints are issues to any content provider, not just for 
local ISPs.


two issues here: the smaller issue is that CDNs sometimes want their own 
routable IP address blocks, especially if they're connecting directly to 
the IXP, which usually means /24 in practice. It doesn't always happen, 
and sometimes the CDN is happy to use provider address space (i.e. IXP), 
or smaller address blocks. But it's something to note.


The bigger issue is: who pays the transit costs for the CDN's cache-fill 
requirements? CDNs typically won't pay for cache-fill for installations 
like this, and if one local ISP is pulling disproportionate quantities 
of data compared to other ISPs at the IXP, then this can cause problems 
unless there's an shared billing mechanism built in.


Nick

Re: Shared cache servers on an island's IXP

[Gael Hernandez]

Hosting authoritative and recursive dns servers at the IXP would
drastically improve the experience of users most of the time.

Of course, Stephane considerations are correct and there’s no solution for
when global connectivity is lost and responses will stop being sent.

Gaël


On Thu 18 Jan 2024 at 14:42, Jérôme Nicolle  wrote:

> Hi Gael,
>
> Le 18/01/2024 à 13:48, Gael Hernandez a écrit :
> > Friends from PCH (www.pch.net ) operate backend
> > services for DNS authoritative ccTLDs and the Quad9 DNS resolver. They
> > would be very happy to help.
>
> I'm sure they would, I'm a big fan of their work BTW. Though hosting
> them in a densely connected area isn't the same as it will in remote
> locations, I guess there could be some work to be done to get it running
> properly, as Stephane wrote.
>
> How would you think we could work on that ? I mean, disconnected or
> extremely high latency scenarii should be on a research roadmap by
> SpaceX' standards, right ? ;-)
>
> Best regards,
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>

Re: Shared cache servers on an island's IXP

[Jérôme Nicolle]

Hi Tom,

Le 18/01/2024 à 15:20, Tom Beecher a écrit :
Many CDNs have hardware options for self hosted caches. I think there 
would likely be concerns about <20G of connectivity to those caches 
though. With an incorrect setup, you could end up maxing out those links 
just with cache fill traffic itself.


In a case where these servers are on a dedicated network peering with 
the ISPs, I think it would be safe to throttle the sync feeds to not 
saturate actual uplinks.


At least, that we can do, but throttling uncached content to customers 
is forbidden (net neutrality).


Though Netflix is supposedly sending pre-loaded servers, and I think 
that - in this location - it's gonna mean a lot already. The quastion is 
: how would the servers peer with local ISPs.


Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Re: Shared cache servers on an island's IXP

[Mike Hammett]

Some will work directly on the IX via BGP. Others have to go behind a member of 
the IX. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Jérôme Nicolle"  
To: "Mehmet"  
Cc: nanog@nanog.org 
Sent: Thursday, January 18, 2024 8:38:31 AM 
Subject: Re: Shared cache servers on an island's IXP 

Hello Mehmet, 

Le 18/01/2024 à 12:58, Mehmet a écrit : 
> VMs are no go for big content companies except Microsoft. You can run 
> Microsoft CDN on VM but rest of the content will need to be cached. You 
> can actually install this yourself 

I've already read most docs for caching servers provided by major 
actors. What I'm mostly concerned about is their ability to peer with 
multiple AS on the local IXP, as to not over-replicate them. 

Should I establish a dedicated network peering on the IXP ? Or will they 
come with their own ASNs ? The peering case is quite not documented on 
publicly available specs, if even possible. 

> Depending on how much traffic do you have , you may be able to get 
> facebook, youtube, netflix caches i think minimum bw requirement changes 
> per region 

Those I'm nearly sure I could get, if I can pool caches amongst ISPs. 
The current constraints are issues to any content provider, not just for 
local ISPs. 

Best regards, 

-- 
Jérôme Nicolle 
+33 6 19 31 27 14 

Re: Shared cache servers on an island's IXP

[Jérôme Nicolle]

Hi Gael,

Le 18/01/2024 à 13:48, Gael Hernandez a écrit :
Friends from PCH (www.pch.net ) operate backend 
services for DNS authoritative ccTLDs and the Quad9 DNS resolver. They 
would be very happy to help.


I'm sure they would, I'm a big fan of their work BTW. Though hosting 
them in a densely connected area isn't the same as it will in remote 
locations, I guess there could be some work to be done to get it running 
properly, as Stephane wrote.


How would you think we could work on that ? I mean, disconnected or 
extremely high latency scenarii should be on a research roadmap by 
SpaceX' standards, right ? ;-)


Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Re: Shared cache servers on an island's IXP

[Jérôme Nicolle]

Hello Mehmet,

Le 18/01/2024 à 12:58, Mehmet a écrit :
VMs are no go for big content companies except Microsoft. You can run 
Microsoft CDN on VM but rest of the content will need to be cached. You 
can actually install this yourself


I've already read most docs for caching servers provided by major 
actors. What I'm mostly concerned about is their ability to peer with 
multiple AS on the local IXP, as to not over-replicate them.


Should I establish a dedicated network peering on the IXP ? Or will they 
come with their own ASNs ? The peering case is quite not documented on 
publicly available specs, if even possible.


Depending on how much traffic do you have , you may be able to get 
facebook, youtube, netflix caches i think minimum bw requirement changes 
per region


Those I'm nearly sure I could get, if I can pool caches amongst ISPs. 
The current constraints are issues to any content provider, not just for 
local ISPs.


Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Re: Shared cache servers on an island's IXP

[Tom Beecher]

Many CDNs have hardware options for self hosted caches. I think there would
likely be concerns about <20G of connectivity to those caches though. With
an incorrect setup, you could end up maxing out those links just with cache
fill traffic itself.


On Thu, Jan 18, 2024 at 6:54 AM Jérôme Nicolle  wrote:

> Hello,
>
> I'm trying to find out the best way to consolidate connectivity on an
> island.
>
> The current issues are :
> - Low redundancy of old cables (2)
> - Low system capacity of said cables (<=20Gbps)
> - Total service loss when both cables are down because of congestion on
> satelite backups
> - Sheer price of bandwidth
>
> On the plus side, there are over 5 AS on the island, an IXP and
> small-ish collocation capacity (approx 10kW available, could be
> upgraded, second site available later this year).
>
> We'd like to host cache servers and/or VMs on the IXP, with an option to
> anycast many services - without hijacking them, that goes without saying
> - such as quad-whatever DNS resolvers, NTP servers and whatever else
> could be useful for weather-induced disaster-recovery and/or offload
> cables.
>
> Do you think most CDNs, stream services and CSPs could accommodate a
> scenario where we'd host their gear or provide VMs for them to announce
> on the local route-servers ? If not, what could be a reasonable
> technical arrangement ?
>
> Thanks !
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>

Re: Shared cache servers on an island's IXP

[Stephane Bortzmeyer]

On Thu, Jan 18, 2024 at 12:53:19PM +0100,
 Jérôme Nicolle  wrote 
 a message of 36 lines which said:

> - Low redundancy of old cables (2)
> - Total service loss when both cables are down because of congestion on
> satelite backups

A problem which is not often mentioned is that most (all?) "local
caches" (CDN, DNS resolvers, etc) do not have an "offline mode" (or
"disconnected-from-master mode"). During an outage, they continue to
work for some time then break suddenly, in a not-friendly way, serving
various error messages instead of old data and/or useful
messages. (For instance, the DNS resolver may not be able to serve
stale answers.)

The time during which they can continue to work when they are
disconnected from their master is typically undocumented (except for
the DNS), and discovered only when there is a long outage.

Making the Internet work better with sometimes-broken connectivity is
still an area of research.

Re: Shared cache servers on an island's IXP

[Mehmet]

Hi Jérôme

VMs are no go for big content companies except Microsoft. You can run
Microsoft CDN on VM but rest of the content will need to be cached. You can
actually install this yourself

Depending on how much traffic do you have , you may be able to get
facebook, youtube, netflix caches i think minimum bw requirement changes
per region

Good luck

On Thu, Jan 18, 2024 at 06:53 Jérôme Nicolle  wrote:

> Hello,
>
> I'm trying to find out the best way to consolidate connectivity on an
> island.
>
> The current issues are :
> - Low redundancy of old cables (2)
> - Low system capacity of said cables (<=20Gbps)
> - Total service loss when both cables are down because of congestion on
> satelite backups
> - Sheer price of bandwidth
>
> On the plus side, there are over 5 AS on the island, an IXP and
> small-ish collocation capacity (approx 10kW available, could be
> upgraded, second site available later this year).
>
> We'd like to host cache servers and/or VMs on the IXP, with an option to
> anycast many services - without hijacking them, that goes without saying
> - such as quad-whatever DNS resolvers, NTP servers and whatever else
> could be useful for weather-induced disaster-recovery and/or offload
> cables.
>
> Do you think most CDNs, stream services and CSPs could accommodate a
> scenario where we'd host their gear or provide VMs for them to announce
> on the local route-servers ? If not, what could be a reasonable
> technical arrangement ?
>
> Thanks !
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>