Re: DNS pulling BGP routes?

[Masataka Ohta]

Jared Mauch wrote:


This is quite common to tie an underlying service announcement to BGP
announcements in an Anycast or similar environment.


Yes, that is a commonly seen mistake with anycast.


I would say more like Application availability caused the BGP routes
to be withdrawn.


Considering a failure mode that routes are not withdrawn even
if application dies or routes are withdrawn even if application
is alive, active withdrawal of routes is unnecessary complication
with no improvement of redundancy.

DNS (and other protocol's) redundancy is to have multiple
unicast/anycast name server addresses. Just rely on it.

Masataka Ohta

Another Perspective - Kentik's View on the Facebook Outage

[Mark Tinka]

https://www.kentik.com/blog/facebooks-historic-outage-explained/?mkt_tok=ODY5LVBBRC04ODcAAAF_9diyPhC6WFsJNFpS4z2ggF-DWEc6FksyD3aW8B3am5tUvtTOJDYl2MIgMdsmmqOLTL-BpUugbQHAIXCT677LE0OxHM8Dy-mqRorJQXnAjg4

Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/7/21 00:22, Michael Thomas wrote:



But it wasn't just their DNS subnets that were pulled, I thought. I'm 
obviously really confused. Anycast to a DNS server makes sense that 
they'd pull out if they couldn't contact the backend. But I thought 
that almost all of their routes to the backend were pulled? That is, 
the DFZ was emptied of FB routes.


During the outage, we kept serving traffic to Facebook in various 
locations. So it would seem that while a large amount of their NLRI left 
the DFZ, it wasn't all of them.


However, what was left was not sufficient to actually keep typical 
services up, including their DNS.


Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/7/21 00:37, Michael Thomas wrote:

Maybe the problem here is that two things happened and the article 
conflated the two: the DNS infrastructure pulled its routes from the 
anycast address and something else pulled all of the other routes but 
wasn't mentioned in the article.


The origin problem was some "automation thingy" that went to check 
capacity status around the network ahead of some planned maintenance 
work, and that "automation thingy" decided checking was not enough, 
let's just turn the whole thing off.


Mark.

Re: DNS pulling BGP routes?

[Jon Lewis]

On Wed, 6 Oct 2021, Michael Thomas wrote:



On 10/6/21 3:33 PM, Jon Lewis wrote:

 On Wed, 6 Oct 2021, Michael Thomas wrote:


  People have been anycasting DNS server IPs for years (decades?). So,
 no.


 But it wasn't just their DNS subnets that were pulled, I thought. I'm
 obviously really confused. Anycast to a DNS server makes sense that
 they'd pull out if they couldn't contact the backend. But I thought that
 almost all of their routes to the backend were pulled? That is, the DFZ
 was emptied of FB routes.


 Well, as someone else said, DNS wasn't the problem...it was just one of
 the more noticeable casualties.  Whatever they did broke the network
 rather completely, and that took out all of their DNS, which broke lots of
 other things that depend on DNS.

Maybe the problem here is that two things happened and the article conflated 
the two: the DNS infrastructure pulled its routes from the anycast address 
and something else pulled all of the other routes but wasn't mentioned in the 
article.



From the engineering.fb.com article:


"This was the source of yesterday’s outage. During one of these routine 
maintenance jobs, a command was issued with the intention to assess the 
availability of global backbone capacity, which unintentionally took down 
all the connections in our backbone network, effectively disconnecting 
Facebook data centers globally."


If you kill the backbone, and every site determines "my connectivity is 
hosed, suppress anycast propagation.", then you simultaneously have no 
network, and no anycast (which might otherwise propagate to transit/peers 
at each or at least some subset of your sites). All of your internal data 
and communication systems that rely on both network and working DNS 
suddenly don't work, so internal communications likely degraded to 
engineers calling or texting each other.


From one of the earlier articles, it sounds like they don't have true out 
of band access to their routers/switches, which makes it kind of hard to 
fix the network, if it's no longer a network and you have no access to 
console or management ports.


--
 Jon Lewis, MCP :)   |  I route
 StackPath, Sr. Neteng   |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: DNS pulling BGP routes?

[Michael Thomas]

On 10/6/21 3:33 PM, Jon Lewis wrote:

On Wed, 6 Oct 2021, Michael Thomas wrote:

 People have been anycasting DNS server IPs for years (decades?). 
So, no.


But it wasn't just their DNS subnets that were pulled, I thought. I'm 
obviously really confused. Anycast to a DNS server makes sense that 
they'd pull out if they couldn't contact the backend. But I thought 
that almost all of their routes to the backend were pulled? That is, 
the DFZ was emptied of FB routes.


Well, as someone else said, DNS wasn't the problem...it was just one 
of the more noticeable casualties.  Whatever they did broke the 
network rather completely, and that took out all of their DNS, which 
broke lots of other things that depend on DNS.


Maybe the problem here is that two things happened and the article 
conflated the two: the DNS infrastructure pulled its routes from the 
anycast address and something else pulled all of the other routes but 
wasn't mentioned in the article.


Mike

Re: DNS pulling BGP routes?

[Jon Lewis]

On Wed, 6 Oct 2021, Michael Thomas wrote:


 People have been anycasting DNS server IPs for years (decades?). So, no.

But it wasn't just their DNS subnets that were pulled, I thought. I'm 
obviously really confused. Anycast to a DNS server makes sense that they'd 
pull out if they couldn't contact the backend. But I thought that almost all 
of their routes to the backend were pulled? That is, the DFZ was emptied of 
FB routes.


Well, as someone else said, DNS wasn't the problem...it was just one of 
the more noticeable casualties.  Whatever they did broke the network 
rather completely, and that took out all of their DNS, which broke lots of 
other things that depend on DNS.


--
 Jon Lewis, MCP :)   |  I route
 StackPath, Sr. Neteng   |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: DNS pulling BGP routes?

[Michael Thomas]

On 10/6/21 2:58 PM, Jon Lewis wrote:

On Wed, 6 Oct 2021, Michael Thomas wrote:



On 10/6/21 2:33 PM, William Herrin wrote:

 On Wed, Oct 6, 2021 at 10:43 AM Michael Thomas  wrote:

 So if I understand their post correctly, their DNS servers have the
 ability to withdraw routes if they determine are sub-optimal (fsvo).

 The servers' IP addresses are anycasted. When one data center
 determines itself to be malfunctioning, it withdraws the routes so
 that users will reach a different data center that is, in theory,
 still functioning.

Ah, I was wondering if the anycast part was the relevant bit. But 
doesn't it seem odd that it would be intertwined with the DNS 
infrastructure?


People have been anycasting DNS server IPs for years (decades?). So, no.

But it wasn't just their DNS subnets that were pulled, I thought. I'm 
obviously really confused. Anycast to a DNS server makes sense that 
they'd pull out if they couldn't contact the backend. But I thought that 
almost all of their routes to the backend were pulled? That is, the DFZ 
was emptied of FB routes.


Mike

Re: DNS pulling BGP routes?

[Jon Lewis]

On Wed, 6 Oct 2021, Michael Thomas wrote:



On 10/6/21 2:33 PM, William Herrin wrote:

 On Wed, Oct 6, 2021 at 10:43 AM Michael Thomas  wrote:

 So if I understand their post correctly, their DNS servers have the
 ability to withdraw routes if they determine are sub-optimal (fsvo).

 The servers' IP addresses are anycasted. When one data center
 determines itself to be malfunctioning, it withdraws the routes so
 that users will reach a different data center that is, in theory,
 still functioning.

Ah, I was wondering if the anycast part was the relevant bit. But doesn't it 
seem odd that it would be intertwined with the DNS infrastructure?


People have been anycasting DNS server IPs for years (decades?).  So, no.

--
 Jon Lewis, MCP :)   |  I route
 StackPath, Sr. Neteng   |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: Better description of what happened

[Hugo Slabbert]

>
> Do we actually know this wrt the tools referred to in "the total loss of
> DNS broke many of the tools we’d normally use to investigate and resolve
> outages like this."?  Those tools aren't necessarily located in any of
> the remote data centers, and some of them might even refer to resources
> outside the facebook network.


Yea; that's kinda the thinking here.  Specifics are scarce, but there were
notes re: the OOB for instance also being unusable.  The questions are how
much that was due to dependence of the OOB network on the production side,
and how much DNS being notionally available might have supported getting
things back off the ground (if it would just provide mgt addresses for key
devices, or if perhaps there was a AAA dependency that also rode on DNS).
This isn't to say there aren't other design considerations in play to make
that fly (e.g. if DNS lives in edge POPs, and such an edge POP gets
isolated from the FB network but still has public Internet peering, how do
we ensure that edge POP does not continue exporting the DNS prefix into the
DFZ and serving stale records?), but perhaps also still solvable

I'm sure they'll learn from this and in the future have some better things
> in place to account for such a scenario.


100%

I think we can say with some level of confidence that there is going to be
a *lot* of discussion and re-evaluation of inter-service dependencies.

-- 
Hugo Slabbert


On Wed, Oct 6, 2021 at 9:48 AM Tom Beecher  wrote:

> I mean, at the end of the day they likely designed these systems to be
> able to handle one or more datacenters being disconnected from the world,
> and considered a scenario of ALL their datacenters being disconnected from
> the world so unlikely they chose not to solve for it. Works great, until it
> doesn't.
>
> I'm sure they'll learn from this and in the future have some better
> things in place to account for such a scenario.
>
> On Wed, Oct 6, 2021 at 12:21 PM Bjørn Mork  wrote:
>
>> Tom Beecher  writes:
>>
>> >  Even if the external
>> > announcements were not withdrawn, and the edge DNS servers could provide
>> > stale answers, the IPs those answers provided wouldn't have actually
>> been
>> > reachable
>>
>> Do we actually know this wrt the tools referred to in "the total loss of
>> DNS broke many of the tools we’d normally use to investigate and resolve
>> outages like this."?  Those tools aren't necessarily located in any of
>> the remote data centers, and some of them might even refer to resources
>> outside the facebook network.
>>
>> Not to mention that keeping the DNS service up would have prevented
>> resolver overload in the rest of the world.
>>
>> Besides, the disconnected frontend servers are probably configured to
>> display a "we have a slight technical issue. will be right back" notice
>> in such situations.  This is a much better user experience that the
>> "facebook?  never heard of it" message we got on monday.
>>
>> yes, it makes sense to keep your domains alive even if your network
>> isn't.  That's why the best practice is name servers in more than one
>> AS.
>>
>>
>>
>>
>> Bjørn
>>
>

Re: DNS pulling BGP routes?

[Michael Thomas]

On 10/6/21 2:33 PM, William Herrin wrote:

On Wed, Oct 6, 2021 at 10:43 AM Michael Thomas  wrote:

So if I understand their post correctly, their DNS servers have the
ability to withdraw routes if they determine are sub-optimal (fsvo).

The servers' IP addresses are anycasted. When one data center
determines itself to be malfunctioning, it withdraws the routes so
that users will reach a different data center that is, in theory,
still functioning.

Ah, I was wondering if the anycast part was the relevant bit. But 
doesn't it seem odd that it would be intertwined with the DNS 
infrastructure?


Mike

Re: DNS pulling BGP routes?

[William Herrin]

On Wed, Oct 6, 2021 at 10:43 AM Michael Thomas  wrote:
>
> So if I understand their post correctly, their DNS servers have the
> ability to withdraw routes if they determine are sub-optimal (fsvo).

The servers' IP addresses are anycasted. When one data center
determines itself to be malfunctioning, it withdraws the routes so
that users will reach a different data center that is, in theory,
still functioning.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: DNS pulling BGP routes?

[Jon Lewis]

On Wed, 6 Oct 2021, Michael Thomas wrote:

So if I understand their post correctly, their DNS servers have the ability 
to withdraw routes if they determine are sub-optimal (fsvo). I can certainly 
understand for the DNS servers to not give answers they think are unreachable 
but there is always the problem that they may be partitioned and not the 
routes themselves. At a minimum, I would think they'd need some consensus 
protocol that says that it's broken across multiple servers.


But I just don't understand why this is a good idea at all. Network topology 
is not DNS's bailiwick so using it as a trigger to withdraw routes seems


Everything I've seen posted about this (whether from Facebook directly, or 
others) is so vague as to what happened, that I think everyone's just 
making assumptions based on their own experiences or best guesses as to 
what really happened.


In that vein, imagine you have dozens of small sites acting as anycast 
origins for DNS.  Each regularly does some network health tests to 
determine if its links to the rest of the (region|backbone|world|etc.) are 
working within defined paramters.  If the health test fails, the site 
needs to be removed from anycast until the network health issue is 
resolved.  You're big, like automating things, and feel the need for 
speed, so when the health test fails, rather than trigger an alarm which 
your NOC may or may not act on in a timely manner, the local anycast 
origin routes are automatically suppressed from propagating beyond the 
site.


Just suppose you pushed out a new network health test that was guaranteed 
to fail in every POP...and you pushed it out to every POP.  All of a 
sudden, your anycast routes aren't advertised anywhere.


Is this what happened?  I really have no clue.  It sounds like something 
like this might have happened.  Unless someone at Facebook shares an 
actual detailed account of what they broke, most of us will never know 
what really happened.


--
 Jon Lewis, MCP :)   |  I route
 StackPath, Sr. Neteng   |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

The bottom line problem is that we have allowed vertical integration to allow
the natural monopoly that exists in last mile infrastructure in most locations 
to
be leveraged into an effective full-stack monopoly for those same players.

Lack of competition in the last-mile/eyeball space has allowed larger monopoly
eyeball providers to leverage double-payment for the same traffic, extorting
eyeballs that have no alternative to pay them to deliver the content while
simultaneously turning around and extorting the content providers themselves
to be allowed to reach “their customers”.

While monopoly $CABLECO/$DSLTELCO/$GPONPROVIDER can generally
get away making access problematic for a handful of content providers for
short periods of time, no content provider can really absorb the losses 
associated
with allowing that situation to continue, thus giving the content providers
leverage.

Unfortunately for the content providers, newer laws and lower costs to provide
higher bandwidth are making WISPs a viable competitor in both rural and
metropolitan settings. Consumers are catching on to this and telling the more
traditional ISPs that they now have a choice and the ISPs will have to up their
game to keep their business, so finally progress is being made, which is
removing that leverage from the previously monopoly providers on both
the consumer and the content provider side of that equation.

Obviously, ISPs don’t like this and the monopoly ones being not in the
communications business, but rather operating as a law firm with some
switching infrastructure will attempt to use legislation, PUC, and courts
as weapons to try and preserve the status quo.

I’m hopeful that the Korean courts will see SK’s shakedown for what it is
and toss it in summary judgment (or whatever the Korean equivalent is)
with costs and attorneys’ fees awarded to Netflix. I’m hoping that this will
also send a clear message to other ISPs contemplating such extortive
behavior.

OTOH, I admit I will watch in amusement if SK’s customers trying to play
Netflix videos are only able to watch in 480p after viewing a brief video
explaining that the video they are about to watch is degraded courtesy
of SK’s business practices (obviously with appropriate details).

Owen


> On Oct 1, 2021, at 11:24 , Joshua Pool via NANOG  wrote:
> 
> I think instances where the end ISP is peered directly with Netflix and 
> demands more money is not valid at all.  That should be normal cost of doing 
> business to increase capacity as the consumer demand grows.
> The topic of interest is instances where the ISP is not directly peered with 
> Netflix and uses upstream providers and those providers are trying to make 
> content providers absorb the cost of increasing peering capacity for services 
> that traverse their infrastructure.
> One could make the argument that Tier1's should never be the choke point as 
> they should be keeping up with the times and be proactively increasing 
> capacity.
> One could also note that it's 2021 and Cogent and Hurricane Electric are 
> still not peered.
> 
> 
> 
> 
> On Fri, Oct 1, 2021 at 10:47 AM Blake Hudson  > wrote:
> 
> 
> On 10/1/2021 11:23 AM, Sean Donelan wrote:
> >
> >
> > In the old days, postal services used to charge the recipient of a 
> > letter to deliver the letter. Then stamps were invented, and postal 
> > services charged the sender of the letter, and the recipent got free 
> > delivery.
> >
> > Now there is free-shipping, and pre-paid return envelopes for DVDs.
> >
> > Of course, the shipping isn't really "Free."  Its built into the cost 
> > of goods sold.
> >
> > There is no universal, fixed, unchangable way of allocating business 
> > costs.
> >
> 
> True. But when the sender has already paid the stamp to their courier to 
> deliver the bits on their leg of the journey, and the recipient has 
> already paid a stamp to deliver the bits on their leg of the journey, 
> what case does the recipient's courier have to demand additional payment 
> from the sender (lest the package get lost)? The stamp has already been 
> paid. TWICE! Withholding service until additional payments are made just 
> smells like extortion.

Re: DNS pulling BGP routes?

[Sabri Berisha]

- On Oct 6, 2021, at 10:42 AM, Michael Thomas m...@mtcc.com wrote:

Hi,

> My guess is that their post while more clear that most doesn't go into
> enough detail, but is it me or does it seem like this is a really weird
> thing to do?

In large environments, it's not uncommon to have DNS servers announce
themselves on an anycast IP. This is also referred to as "host BGP". 
Basically, the host (or hypervisor) speaks BGP with the TOR. Your spines
or superspines will then pick a best route or ECMP across multiple DNS
servers. 

My guess is that Facebook took this concept a step further and anycasted
their public DNS servers through their datacenters to the internet. One
single config change made the DNS servers think that they were no longer
functioning properly which caused them to withdraw the routes. At least,
that's what I understand from the post-mortem.

Thanks,

Sabri

NANOG 83 Sneak Peak + Safety Concerns

[Nanog News]

Sneak Peak of NANOG 83Register Now for Incredible Programming, Network
Opportunities + More

*The anticipation is killing us!* We are only three short weeks away from
our next meeting and we cannot wait to see you! NANOG 83 will take place
online + in person in Minneapolis, MN, from Nov. 1 - 3.


*Topics to be featured:*

   - Famous Internet Outages Panel
   - Security
   - IPv6
   - Automation
   - Networking Sessions
   - Newcomers on Monday
   - Women in Tech on Tuesday
   - Community Meeting
   - Board Candidate Session

*Be sure not to miss* the NANOG Board of Directors candidate session on
Monday, 11/1. NANOG 83 Keynotes, given by Bert Hubert on Tuesday, 11/2, and
John Brzozowski on Wednesday, 11/3. We’ll also hold the NANOG Community
Meeting on Wednesday, 11/3 (all are welcome!), and the NANOG Members
Meeting on Tuesday, 11/02. If this is your first NANOG conference, we
encourage you to attend the Newcomers Networking Session on Monday.

*Also, don't miss our hybrid expo.* Daily games + prizes, and numerous
opportunities to connect with our community in real-time (online +
in-person).

*REGISTER NOW  *
*NANOG 83 Safety + Travel Concerns*
Addressing the COVID Elephant

We understand the changing course of the pandemic may concern the NANOG
community about traveling to NANOG 83 this fall.

Your health and safety are very important to us and we are continuing to
monitor the latest federal, state, and local government guidelines and
restrictions. Check out our Meeting Guidelines to learn more about the
precautions we are taking.

*MEETING GUIDELINES  *
*Register Now for Hackathon*
Fun Way to Network + Explore Passion

*NANOG 83 Hackathon Details*
Fri. Oct. 22 - intro/tutorial/team formation
Sat. + Sun. Oct. 30, 31 - hacking

*Enjoy in-Person + virtual: *NANOG Hackathons are hands-on, and educational
at their core — directly supporting the most critical aspects of our
mission. All levels are welcome to participate, and as always, registration
is free.

So, what is it actually like to participate? Q + A with a NANOG Hackathon

alumni.


*REGISTER NOW * 

Re: DNS pulling BGP routes?

[Matthew Petach]

On Wed, Oct 6, 2021 at 10:45 AM Michael Thomas  wrote:

> So if I understand their post correctly, their DNS servers have the
> ability to withdraw routes if they determine are sub-optimal (fsvo). I
> can certainly understand for the DNS servers to not give answers they
> think are unreachable but there is always the problem that they may be
> partitioned and not the routes themselves. At a minimum, I would think
> they'd need some consensus protocol that says that it's broken across
> multiple servers.
>
> But I just don't understand why this is a good idea at all. Network
> topology is not DNS's bailiwick so using it as a trigger to withdraw
> routes seems really strange and fraught with unintended consequences.
> Why is it a good idea to withdraw the route if it doesn't seem reachable
> from the DNS server? Give answers that are reachable, sure, but to
> actually make a topology decision? Yikes. And what happens to the cached
> answers that still point to the supposedly dead route? They're going to
> fail until the TTL expires anyway so why is it preferable withdraw the
> route too?
>
> My guess is that their post while more clear that most doesn't go into
> enough detail, but is it me or does it seem like this is a really weird
> thing to do?
>
> Mike
>


Hi Mike,

You're kinda thinking about this from the wrong angle.

It's not that the route is withdrawn if doesn't seem reachable
from the DNS server.

It's that your DNS server is geolocating requests to the nearest
content delivery cluster, where the CDN cluster is likely fetching
content from a core datacenter elsewhere.  You don't want that
remote/edge CDN node to give back A records for a CDN node
that is isolated from the rest of the network and can't reach the
datacenter to fetch the necessary content; otherwise, you'll have
clients that reach the page, can load the static elements on the
page, but all the dynamic elements hang, waiting for a fetch to
complete from the origin which won't ever complete.  Not a very
good end user experience.

So, the idea is that if the edge CDN node loses connectivity to
the core datacenters, the DNS servers should stop answering
queries for A records with the local CDN node's address, and
let a different site respond back to the client's DNS request.
In particular, you really don't want the client to even send the
request to the edge CDN node that's been isolated, you want
to allow anycast to find the next-best edge site; so, once the
DNS servers fail the "can-I-reach-my-datacenter" health check,
they stop announcing the Anycast service address to the local
routers; that way, they drop out of the Anycast pool, and normal
Internet routing will ensure the client DNS requests are now sent
to the next-nearest edge CDN cluster for resolution and retrieving
data.

This works fine for ensuring that one or two edge sites that get
isolated due to fiber cuts don't end up pulling client requests into
them, and subsequently leaving the users hanging, waiting for
data that will never arrive.

However, it fails big-time if *all* sites fail their
"can-I-reach-the-datacenter"
check simultaneously.  When I was involved in the decision making
on a design like this, a choice was made to have a set of "really core"
sites in the middle of the network always announce the anycast prefixes,
as a fallback, so even if the routing wasn't optimal to reach them, the
users would still get *some* level of reply back.

In this situation, that would have ensured that at least some DNS
servers were reachable; but it wouldn't have fixed the "oh crap we
pushed 'no router bgp' out to all the routers at the same time" type
problem.  But that isn't really the core of your question, so we'll
just quietly push that aside for now.   ^_^;

Point being--it's useful and normal for edge sites that may become
isolated from the rest of the network to be configured to stop announcing
the Anycast service address for DNS out to local peers and transit
providers at that site during the period in which they are isolated, to
prevent users from being directed to CDN servers which can't fetch
content from the origin servers in the datacenter.  It's just generally
assumed that not every site will become "isolated" at the same time
like that.   :)

I hope this helps clear up the confusion.

Thanks!

Matt

Re: DNS pulling BGP routes?

[Blake Dunlap]

Yes, it really is common to announce sink routes via bgp from destination
services / proxies and to have those announcements be dynamically based on
service viability.

On Wed, Oct 6, 2021, 12:56 Jared Mauch  wrote:

> This is quite common to tie an underlying service announcement to BGP
> announcements in an Anycast or similar environment.  It doesn’t have to be
> externally visible like this event for that to be the case.
>
> I would say more like Application availability caused the BGP routes to be
> withdrawn.
>
> I know several network operators that run DNS internally (even on
> raspberry pi devices)  and may have OSPF or BGP announcements internally to
> ensure things work well.  If the process dies (crash, etc) they want to
> route to the next nearest cluster.
>
> Of course if they all are down there’s negative outcomes.
>
> - Jared
>
>
> > On Oct 6, 2021, at 1:42 PM, Michael Thomas  wrote:
> >
> > So if I understand their post correctly, their DNS servers have the
> ability to withdraw routes if they determine are sub-optimal (fsvo). I can
> certainly understand for the DNS servers to not give answers they think are
> unreachable but there is always the problem that they may be partitioned
> and not the routes themselves. At a minimum, I would think they'd need some
> consensus protocol that says that it's broken across multiple servers.
> >
> > But I just don't understand why this is a good idea at all. Network
> topology is not DNS's bailiwick so using it as a trigger to withdraw routes
> seems really strange and fraught with unintended consequences. Why is it a
> good idea to withdraw the route if it doesn't seem reachable from the DNS
> server? Give answers that are reachable, sure, but to actually make a
> topology decision? Yikes. And what happens to the cached answers that still
> point to the supposedly dead route? They're going to fail until the TTL
> expires anyway so why is it preferable withdraw the route too?
> >
> > My guess is that their post while more clear that most doesn't go into
> enough detail, but is it me or does it seem like this is a really weird
> thing to do?
> >
> > Mike
> >
> >
> > On 10/5/21 11:56 PM, Bjørn Mork wrote:
> >> Masataka Ohta  writes:
> >>
> >>> As long as name servers with expired zone data won't serve
> >>> request from outside of facebook, whether BGP routes to the
> >>> name servers are announced or not is unimportant.
> >> I am not convinced this is true.  You'd normally serve some semi-static
> >> content, especially wrt stuff you need yourself to manage your network.
> >> Removing all DNS servers at the same time is never a good idea, even in
> >> the situation where you believe they are all failing.
> >>
> >> The problem is of course that you can't let the servers take the
> >> decision to withdraw from anycast if you want to prevent this
> >> catastrophe.  The servers have no knowledge of the rest of the network.
> >> They only know that they've lost contact with it.  So they all make the
> >> same stupid decision.
> >>
> >> But if the servers can't withdraw, then they will serve stale content if
> >> the data center loses backbone access. And with a large enough network
> >> then that is probably something which happens on a regular basis.
> >>
> >> This is a very hard problem to solve.
> >>
> >> Thanks a lot to facebook for making the detailed explanation available
> >> to the public.  I'm crossing my fingers hoping they follow up with
> >> details about the solutions they come up with.  The problem affects any
> >> critical anycast DNS service. And it doesn't have to be as big as
> >> facebook to be locally critical to an enterprise, ISP or whatever.
> >>
> >>
> >>
> >> Bjørn
>
>

Re: DNS pulling BGP routes?

[Jared Mauch]

This is quite common to tie an underlying service announcement to BGP 
announcements in an Anycast or similar environment.  It doesn’t have to be 
externally visible like this event for that to be the case.

I would say more like Application availability caused the BGP routes to be 
withdrawn.  

I know several network operators that run DNS internally (even on raspberry pi 
devices)  and may have OSPF or BGP announcements internally to ensure things 
work well.  If the process dies (crash, etc) they want to route to the next 
nearest cluster.

Of course if they all are down there’s negative outcomes.

- Jared


> On Oct 6, 2021, at 1:42 PM, Michael Thomas  wrote:
> 
> So if I understand their post correctly, their DNS servers have the ability 
> to withdraw routes if they determine are sub-optimal (fsvo). I can certainly 
> understand for the DNS servers to not give answers they think are unreachable 
> but there is always the problem that they may be partitioned and not the 
> routes themselves. At a minimum, I would think they'd need some consensus 
> protocol that says that it's broken across multiple servers.
> 
> But I just don't understand why this is a good idea at all. Network topology 
> is not DNS's bailiwick so using it as a trigger to withdraw routes seems 
> really strange and fraught with unintended consequences. Why is it a good 
> idea to withdraw the route if it doesn't seem reachable from the DNS server? 
> Give answers that are reachable, sure, but to actually make a topology 
> decision? Yikes. And what happens to the cached answers that still point to 
> the supposedly dead route? They're going to fail until the TTL expires anyway 
> so why is it preferable withdraw the route too?
> 
> My guess is that their post while more clear that most doesn't go into enough 
> detail, but is it me or does it seem like this is a really weird thing to do?
> 
> Mike
> 
> 
> On 10/5/21 11:56 PM, Bjørn Mork wrote:
>> Masataka Ohta  writes:
>> 
>>> As long as name servers with expired zone data won't serve
>>> request from outside of facebook, whether BGP routes to the
>>> name servers are announced or not is unimportant.
>> I am not convinced this is true.  You'd normally serve some semi-static
>> content, especially wrt stuff you need yourself to manage your network.
>> Removing all DNS servers at the same time is never a good idea, even in
>> the situation where you believe they are all failing.
>> 
>> The problem is of course that you can't let the servers take the
>> decision to withdraw from anycast if you want to prevent this
>> catastrophe.  The servers have no knowledge of the rest of the network.
>> They only know that they've lost contact with it.  So they all make the
>> same stupid decision.
>> 
>> But if the servers can't withdraw, then they will serve stale content if
>> the data center loses backbone access. And with a large enough network
>> then that is probably something which happens on a regular basis.
>> 
>> This is a very hard problem to solve.
>> 
>> Thanks a lot to facebook for making the detailed explanation available
>> to the public.  I'm crossing my fingers hoping they follow up with
>> details about the solutions they come up with.  The problem affects any
>> critical anycast DNS service. And it doesn't have to be as big as
>> facebook to be locally critical to an enterprise, ISP or whatever.
>> 
>> 
>> 
>> Bjørn

Re: DNS pulling BGP routes?

[J. Hellenthal via NANOG]

They most likely sent an update to the DNS servers for TLV DNSSEC and in 
oversight forgot they needed to null something's out of the workbook to not 
touch the BGP instances.

I'd hardly believe that would be triggered by the dns server itself.

-- 
 J. Hellenthal

The fact that there's a highway to Hell but only a stairway to Heaven says a 
lot about anticipated traffic volume.

> On Oct 6, 2021, at 12:45, Michael Thomas  wrote:
> 
> So if I understand their post correctly, their DNS servers have the ability 
> to withdraw routes if they determine are sub-optimal (fsvo). I can certainly 
> understand for the DNS servers to not give answers they think are unreachable 
> but there is always the problem that they may be partitioned and not the 
> routes themselves. At a minimum, I would think they'd need some consensus 
> protocol that says that it's broken across multiple servers.
> 
> But I just don't understand why this is a good idea at all. Network topology 
> is not DNS's bailiwick so using it as a trigger to withdraw routes seems 
> really strange and fraught with unintended consequences. Why is it a good 
> idea to withdraw the route if it doesn't seem reachable from the DNS server? 
> Give answers that are reachable, sure, but to actually make a topology 
> decision? Yikes. And what happens to the cached answers that still point to 
> the supposedly dead route? They're going to fail until the TTL expires anyway 
> so why is it preferable withdraw the route too?
> 
> My guess is that their post while more clear that most doesn't go into enough 
> detail, but is it me or does it seem like this is a really weird thing to do?
> 
> Mike
> 
> 
>> On 10/5/21 11:56 PM, Bjørn Mork wrote:
>> Masataka Ohta  writes:
>> 
>>> As long as name servers with expired zone data won't serve
>>> request from outside of facebook, whether BGP routes to the
>>> name servers are announced or not is unimportant.
>> I am not convinced this is true.  You'd normally serve some semi-static
>> content, especially wrt stuff you need yourself to manage your network.
>> Removing all DNS servers at the same time is never a good idea, even in
>> the situation where you believe they are all failing.
>> 
>> The problem is of course that you can't let the servers take the
>> decision to withdraw from anycast if you want to prevent this
>> catastrophe.  The servers have no knowledge of the rest of the network.
>> They only know that they've lost contact with it.  So they all make the
>> same stupid decision.
>> 
>> But if the servers can't withdraw, then they will serve stale content if
>> the data center loses backbone access. And with a large enough network
>> then that is probably something which happens on a regular basis.
>> 
>> This is a very hard problem to solve.
>> 
>> Thanks a lot to facebook for making the detailed explanation available
>> to the public.  I'm crossing my fingers hoping they follow up with
>> details about the solutions they come up with.  The problem affects any
>> critical anycast DNS service. And it doesn't have to be as big as
>> facebook to be locally critical to an enterprise, ISP or whatever.
>> 
>> 
>> 
>> Bjørn

DNS pulling BGP routes?

[Michael Thomas]

So if I understand their post correctly, their DNS servers have the 
ability to withdraw routes if they determine are sub-optimal (fsvo). I 
can certainly understand for the DNS servers to not give answers they 
think are unreachable but there is always the problem that they may be 
partitioned and not the routes themselves. At a minimum, I would think 
they'd need some consensus protocol that says that it's broken across 
multiple servers.


But I just don't understand why this is a good idea at all. Network 
topology is not DNS's bailiwick so using it as a trigger to withdraw 
routes seems really strange and fraught with unintended consequences. 
Why is it a good idea to withdraw the route if it doesn't seem reachable 
from the DNS server? Give answers that are reachable, sure, but to 
actually make a topology decision? Yikes. And what happens to the cached 
answers that still point to the supposedly dead route? They're going to 
fail until the TTL expires anyway so why is it preferable withdraw the 
route too?


My guess is that their post while more clear that most doesn't go into 
enough detail, but is it me or does it seem like this is a really weird 
thing to do?


Mike


On 10/5/21 11:56 PM, Bjørn Mork wrote:

Masataka Ohta  writes:


As long as name servers with expired zone data won't serve
request from outside of facebook, whether BGP routes to the
name servers are announced or not is unimportant.

I am not convinced this is true.  You'd normally serve some semi-static
content, especially wrt stuff you need yourself to manage your network.
Removing all DNS servers at the same time is never a good idea, even in
the situation where you believe they are all failing.

The problem is of course that you can't let the servers take the
decision to withdraw from anycast if you want to prevent this
catastrophe.  The servers have no knowledge of the rest of the network.
They only know that they've lost contact with it.  So they all make the
same stupid decision.

But if the servers can't withdraw, then they will serve stale content if
the data center loses backbone access. And with a large enough network
then that is probably something which happens on a regular basis.

This is a very hard problem to solve.

Thanks a lot to facebook for making the detailed explanation available
to the public.  I'm crossing my fingers hoping they follow up with
details about the solutions they come up with.  The problem affects any
critical anycast DNS service. And it doesn't have to be as big as
facebook to be locally critical to an enterprise, ISP or whatever.



Bjørn

Re: Better description of what happened

[Tom Beecher]

I mean, at the end of the day they likely designed these systems to be able
to handle one or more datacenters being disconnected from the world, and
considered a scenario of ALL their datacenters being disconnected from the
world so unlikely they chose not to solve for it. Works great, until it
doesn't.

I'm sure they'll learn from this and in the future have some better
things in place to account for such a scenario.

On Wed, Oct 6, 2021 at 12:21 PM Bjørn Mork  wrote:

> Tom Beecher  writes:
>
> >  Even if the external
> > announcements were not withdrawn, and the edge DNS servers could provide
> > stale answers, the IPs those answers provided wouldn't have actually been
> > reachable
>
> Do we actually know this wrt the tools referred to in "the total loss of
> DNS broke many of the tools we’d normally use to investigate and resolve
> outages like this."?  Those tools aren't necessarily located in any of
> the remote data centers, and some of them might even refer to resources
> outside the facebook network.
>
> Not to mention that keeping the DNS service up would have prevented
> resolver overload in the rest of the world.
>
> Besides, the disconnected frontend servers are probably configured to
> display a "we have a slight technical issue. will be right back" notice
> in such situations.  This is a much better user experience that the
> "facebook?  never heard of it" message we got on monday.
>
> yes, it makes sense to keep your domains alive even if your network
> isn't.  That's why the best practice is name servers in more than one
> AS.
>
>
>
>
> Bjørn
>

Cloudflare contact

[David Brain]

Hi,

Is there a contact/procedure with Cloudflare to address the issue of
users getting CAPCHAs when accessing sites behind CloudFlare.  We are
seeing this impacting some of our user base, and it has been
persisting for a number of days.

Thanks,

David.

-- 
David Brain - MCNC - 919.248.1998

Re: Better description of what happened

[Bjørn Mork]

Tom Beecher  writes:

>  Even if the external
> announcements were not withdrawn, and the edge DNS servers could provide
> stale answers, the IPs those answers provided wouldn't have actually been
> reachable

Do we actually know this wrt the tools referred to in "the total loss of
DNS broke many of the tools we’d normally use to investigate and resolve
outages like this."?  Those tools aren't necessarily located in any of
the remote data centers, and some of them might even refer to resources
outside the facebook network.

Not to mention that keeping the DNS service up would have prevented
resolver overload in the rest of the world.

Besides, the disconnected frontend servers are probably configured to
display a "we have a slight technical issue. will be right back" notice
in such situations.  This is a much better user experience that the
"facebook?  never heard of it" message we got on monday.

yes, it makes sense to keep your domains alive even if your network
isn't.  That's why the best practice is name servers in more than one
AS.




Bjørn

Re: Better description of what happened

[PJ Capelli via NANOG]

I probably still have my US Robotics 14.4 in the basement, but it's been awhile 
since I've had access to a POTS line it would work on ... :)

pj capelli
pjcape...@pm.me

"Never to get lost, is not living" - Rebecca Solnit

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐

On Wednesday, October 6th, 2021 at 10:41 AM, Curtis Maurand 
 wrote:

> On 10/5/21 5:51 AM, scott wrote:
> 

> > On 10/5/21 8:39 PM, Michael Thomas wrote:
> > 

> > > This bit posted by Randy might get lost in the other thread, but it 
> > > appears that their DNS withdraws BGP routes for prefixes that they can't 
> > > reach or are flaky it seems. Apparently that goes for the prefixes that 
> > > the name servers are on too. This caused internal outages too as it seems 
> > > they use their front facing DNS just like everybody else.
> > > 

> > > Sounds like they might consider having at least one split horizon server 
> > > internally. Lots of fodder here.
> 

> even a POTS line connected to a modem connected to a serial port on a 
> workstation in the data enter so that you can talk to whatever you need to 
> talk to.  I would go so far as to have other outgoing serial connections to 
> routers from that workstation. It's ugly, but it provides remote out of band 
> disaster management.  Just sayin'
> 

> > 
> > 

> > Move fast; break things? :)
> > 

> > scott
> > 

> > > >

signature.asc
Description: OpenPGP digital signature

Re: Better description of what happened

[Tom Beecher]

By what they have said publicly, the initial trigger point was that all of
their datacenters were disconnected from their internal backbone, thus
unreachable.

Once that occurs, nothing else really matters. Even if the external
announcements were not withdrawn, and the edge DNS servers could provide
stale answers, the IPs those answers provided wouldn't have actually been
reachable, and there wouldn't be 3 days of red herring conversations about
DNS design.

No DNS design exists that can help people reach resources not network
reachable. /shrug


On Tue, Oct 5, 2021 at 6:30 PM Hugo Slabbert  wrote:

> Had some chats with other folks:
> Arguably you could change the nameserver isolation check failure action to
> be "depref your exports" rather than "yank it all".  Basically, set up a
> tiered setup so the boxes passing those additional health checks and that
> should have correct entries would be your primary destination and failing
> nodes shouldn't receive query traffic since they're depref'd in your
> internal routing.  But in case all nodes fail that check simultaneously,
> those nodes failing the isolation check would attract traffic again as no
> better paths remain.  Better to serve stale data than none at all; CAP
> theorem trade-offs at work?
>
> --
> Hugo Slabbert
>
>
> On Tue, Oct 5, 2021 at 3:22 PM Michael Thomas  wrote:
>
>>
>> On 10/5/21 3:09 PM, Andy Brezinsky wrote:
>>
>> It's a few years old, but Facebook has talked a little bit about their
>> DNS infrastructure before.  Here's a little clip that talks about
>> Cartographer: https://youtu.be/bxhYNfFeVF4?t=2073
>>
>> From their outage report, it sounds like their authoritative DNS servers
>> withdraw their anycast announcements when they're unhealthy.  The health
>> check from those servers must have relied on something upstream.  Maybe
>> they couldn't talk to Cartographer for a few minutes so they thought they
>> might be isolated from the rest of the network and they decided to withdraw
>> their routes instead of serving stale data.  Makes sense when a single node
>> does it, not so much when the entire fleet thinks that they're out on their
>> own.
>>
>> A performance issue in Cartographer (or whatever manages this fleet these
>> days) could have been the ticking time bomb that set the whole thing in
>> motion.
>>
>> Rereading it is said that their internal (?) backbone went down so
>> pulling the routes was arguably the right thing to do. Or at least not flat
>> out wrong. Taking out their nameserver subnets was clearly a problem
>> though, though a fix is probably tricky since you clearly want to take down
>> errant nameservers too.
>>
>>
>> Mike
>>
>>
>>
>>
>>

Re: Better description of what happened

[Curtis Maurand]

On 10/5/21 5:51 AM, scott wrote:



On 10/5/21 8:39 PM, Michael Thomas wrote:


This bit posted by Randy might get lost in the other thread, but it 
appears that their DNS withdraws BGP routes for prefixes that they 
can't reach or are flaky it seems. Apparently that goes for the 
prefixes that the name servers are on too. This caused internal 
outages too as it seems they use their front facing DNS just like 
everybody else.


Sounds like they might consider having at least one split horizon 
server internally. Lots of fodder here.




even a POTS line connected to a modem connected to a serial port on a 
workstation in the data enter so that you can talk to whatever you need 
to talk to.  I would go so far as to have other outgoing serial 
connections to routers from that workstation. It's ugly, but it provides 
remote out of band disaster management.  Just sayin'







Move fast; break things? :)


scott

Re: Facebook post-mortems...

[Bjørn Mork]

Masataka Ohta  writes:
> Bjørn Mork wrote:
>
>> Removing all DNS servers at the same time is never a good idea, even in
>> the situation where you believe they are all failing.
>
> As I wrote:
>
> : That facebook use very short expiration period for zone
> : data is a separate issue.
>
> that is a separate issue.

Sorry, I don't understand what you're getting at.  The TTL is not an
issue. An infinite TTL won't save you if all authoritative servers are
unreachable.  It will just make things worse in almost every other error
scenario.

The only solution to the problem of unreachable authoritative DNS
servers is:  Don't do that.

>> This is a very hard problem to solve.
>
> If that is their policy, it is just a policy to enforce and not
> a problem to solve.

The policy is there to solve a real problem.

Serving stale data from a single disconnected anycast site is a problem.
A disconnected site is unmanaged and must make autonomous decisions.
That pre-programmed decision is "just policy".  Should you withdraw the
DNS routes or not?  Serve stale or risk meltdown?

I still don't think there is an easy and obviously correct answer.  But
they do of course need to add a safety net or two if they continue with
the "meltdown" policy.


Bjørn

Re: Facebook post-mortems...

[Masataka Ohta]

Bjørn Mork wrote:


Removing all DNS servers at the same time is never a good idea, even in
the situation where you believe they are all failing.


As I wrote:

: That facebook use very short expiration period for zone
: data is a separate issue.

that is a separate issue.

> This is a very hard problem to solve.

If that is their policy, it is just a policy to enforce and not
a problem to solve.

Masataka Ohta

Re: Facebook post-mortems...

[Masataka Ohta]

Hank Nussbacher wrote:


- "it was not possible to access our data centers through our normal
 means because their networks were down, and second, the total loss
of DNS broke many of the internal tools we'd normally use to
investigate and resolve outages like this.  Our primary and
out-of-band network access was down..."

Does this mean that FB acknowledges that the loss of DNS broke their
OOB access?


It means FB still do not yet understand what happened.

Lack of BGP announcement does not mean "total loss". Name
servers should still be accessible by internal tools.

But, withdrawing route (for BGP and, maybe, IGP) of failing anycast
server is a bad engineering seemingly derived from commonly seen
misunderstanding that anycast could provide redundancy.

Redundancy of DNS is maintained by multiple (unicast or anycast)
name servers with different addresses, for which, withdrawal of
failing route is unnecessary complication.

Masataka Ohta

Re: Disaster Recovery Process

[Wolfgang Tremmel]

And a layer 8 item from me:

- put a number (as in money) into the process up to that anything spend by 
anyone working on the recovery is covered.

Has to be a number because if you write "all cost are covered" it makes the 
recovery person 2nd-guess if the airplane ticket or spare part he just bought 
is really covered.

Additionally you should put an "approver" on the team who approves higher cost 
on very short notice.

Wolfgang

> On 5. Oct 2021, at 16:05, Karl Auer  wrote:
> 
> On Tue, 2021-10-05 at 08:50 -0400, Jared Mauch wrote:
>> A few reminders for people:
>> [excellent list snipped]
> 
> I'd add one "soft" list item:

-- 
Wolfgang Tremmel 

Phone +49 69 1730902 0  | wolfgang.trem...@de-cix.net
Executive Directors: Harald A. Summa and Sebastian Seifert | Trade Registry: AG 
Cologne, HRB 51135
DE-CIX Management GmbH | Lindleystrasse 12 | 60314 Frankfurt am Main | Germany 
| www.de-cix.net

Re: Facebook post-mortems...

[Bjørn Mork]

Masataka Ohta  writes:

> As long as name servers with expired zone data won't serve
> request from outside of facebook, whether BGP routes to the
> name servers are announced or not is unimportant.

I am not convinced this is true.  You'd normally serve some semi-static
content, especially wrt stuff you need yourself to manage your network.
Removing all DNS servers at the same time is never a good idea, even in
the situation where you believe they are all failing.

The problem is of course that you can't let the servers take the
decision to withdraw from anycast if you want to prevent this
catastrophe.  The servers have no knowledge of the rest of the network.
They only know that they've lost contact with it.  So they all make the
same stupid decision.

But if the servers can't withdraw, then they will serve stale content if
the data center loses backbone access. And with a large enough network
then that is probably something which happens on a regular basis.

This is a very hard problem to solve.

Thanks a lot to facebook for making the detailed explanation available
to the public.  I'm crossing my fingers hoping they follow up with
details about the solutions they come up with.  The problem affects any
critical anycast DNS service. And it doesn't have to be as big as
facebook to be locally critical to an enterprise, ISP or whatever.



Bjørn