"an engineer put in a config change on a BGP router that propagated to the
other BGP routers and cut off access"
Sounds like something I would do which is why I generally insist on
having a remote tech physically in the building with a console cable if I
don't have OOB console access when changing BGP on my edge.
On Tue, Oct 5, 2021 at 11:34 AM Robert <i...@avantwireless.com> wrote:
> FB has posted that an engineer put in a config change on a BGP router that
> propagated to the other BGP routers and cut off access. Sounds like it
> took a while to track down the config change. I do know someone who is
> intimate with that group but he's not talking... Drinking, but not
> talking...
>
> On 10/5/21 9:17 AM, Bill Prince wrote:
>
> I see it differently. In the beginning cloudflare thought something was
> wrong with their systems. It doesn't explain what happened inside facebook,
> probably because only facebook knows what went wrong. That's not
> cloudflare's problem. We may never know what happened at facebook probably
> because it's very embarrassing to facebook.
>
>
> bp
> <part15sbs{at}gmail{dot}com>
>
> On 10/5/2021 9:01 AM, Robert wrote:
>
> I really don't like that cloudflare posted this, as it doesn't do anything
> to tell why. It's not a root cause, it's an effect and cloudflare looks
> like they are just trying to say how smart they are.
>
> On 10/5/21 8:40 AM, Bill Prince wrote:
>
>
> https://blog.cloudflare.com/october-2021-facebook-outage/
>
>
> bp
> <part15sbs{at}gmail{dot}com>
>
> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
>
> Sign him up for a SNAP card.
>
> Sent from my iPhone
>
> On Oct 5, 2021, at 8:25 AM, Bill Prince <part15...@gmail.com>
> <part15...@gmail.com> wrote:
>
>
>
> Heard the Zuch got around $6 or $7 billion trimmed from his net worth
> yesterday. Now he's only worth around $121 billion.
>
>
> bp
> <part15sbs{at}gmail{dot}com>
>
> On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
>
> I am very curious what it was that took 6 hours to figure out or undo.
>
>
>
> …..but I wouldn’t judge too harshly. When any contraption gets big and
> complex it’s easy to overlook one seemingly small thing.
>
>
>
> Remember this XKCD?
>
>
> <image001.png>
>
>
>
>
>
>
>
> *From:* AF <af-boun...@af.afmug.com> <af-boun...@af.afmug.com> *On Behalf
> Of *Steve Jones
> *Sent:* Monday, October 04, 2021 6:51 PM
> *To:* AnimalFarm Microwave Users Group <af@af.afmug.com> <af@af.afmug.com>
> *Subject:* Re: [AFMUG] facebook outage
>
>
>
> so it turns out FB murdered the announcements for the DNS server prefixes,
> including the registrar.
>
>
> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>
>
>
> This really seems like an elementary design flaw. Even a moron like me
> knows not to poke the DNS subnets all at once, and we probably get way less
> than 1k daily lookups across all the domains we host DNS for. Ill bet
> nobody involved in this makes less than 80k a year. facebook lost 4% stock
> value today attributed to this on top of the almost percent attributed to
> the "whistleblower". I get nervous when I lose the company a number 10
> wrench.
>
>
>
> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller <par...@cyberbroadband.net>
> wrote:
>
>
>
> it is alive here again
>
>
>
> ----- Original Message -----
>
> *From:* CBB - Jay Fuller <par...@cyberbroadband.net>
>
> *To:* AnimalFarm Microwave Users Group <af@af.afmug.com>
>
> *Sent:* Monday, October 4, 2021 4:44 PM
>
> *Subject:* Re: [AFMUG] facebook outage
>
>
>
>
>
> yup. resolving here. but nothing coming up yet.
>
>
>
> ----- Original Message -----
>
> *From:* Mike Hammett <af...@ics-il.net>
>
> *To:* AnimalFarm Microwave Users Group <af@af.afmug.com>
>
> *Sent:* Monday, October 4, 2021 4:36 PM
>
> *Subject:* Re: [AFMUG] facebook outage
>
>
>
> They're back... ish.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
> <https://www.facebook.com/ICSIL>
> <~WRD0002.jpg> <https://www.facebook.com/ICSIL>
> <~WRD0002.jpg>
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
> <~WRD0002.jpg>
> <https://www.linkedin.com/company/intelligent-computing-solutions>
> <~WRD0002.jpg>
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
> <~WRD0002.jpg>
> <https://www.facebook.com/mdwestix>
> <~WRD0002.jpg>
> <https://www.linkedin.com/company/midwest-internet-exchange>
> <~WRD0002.jpg>
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
> <~WRD0002.jpg>
> <https://www.facebook.com/thebrotherswisp>
> <~WRD0002.jpg>
>
>
>
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> ------------------------------
>
> *From: *"CBB - Jay Fuller" <par...@cyberbroadband.net>
> *To: *af@af.afmug.com
> *Sent: *Monday, October 4, 2021 1:13:01 PM
> *Subject: *[AFMUG] facebook outage
>
>
>
> anyone else feeling alone and isolated without facebook? I need my safe
> space (no, not myspace) back.
>
>
>
> babaahahah.
>
>
>
>
>
>
>
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
>
> ------------------------------
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
> ------------------------------
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
>
>
>
>
>
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
--
Carl Peterson
*PORT NETWORKS*
401 E Pratt St, Ste 2553
Baltimore, MD 21202
(410) 637-3707
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
