ill bet whatever their process was, its not what it is and delayed
commits/fallbacks are instituted forthwith. Not by the guy who
committed this set of changes though, hes at the bottom of a river now
On Tue, Oct 5, 2021 at 12:13 PM Cassidy B. Larson <c...@infowest.com> wrote:
> I wonder if they did a "commit confirmed 30”? If not, would that have
> avoided the whole thing?
>
>
> On Oct 5, 2021, at 11:08 AM, Carl Peterson <cpeter...@portnetworks.com>
> wrote:
>
> "an engineer put in a config change on a BGP router that propagated to
> the other BGP routers and cut off access"
>
> Sounds like something I would do which is why I generally insist on
> having a remote tech physically in the building with a console cable if I
> don't have OOB console access when changing BGP on my edge.
>
> On Tue, Oct 5, 2021 at 11:34 AM Robert <i...@avantwireless.com> wrote:
>
>> FB has posted that an engineer put in a config change on a BGP router
>> that propagated to the other BGP routers and cut off access. Sounds like
>> it took a while to track down the config change. I do know someone who is
>> intimate with that group but he's not talking... Drinking, but not
>> talking...
>>
>> On 10/5/21 9:17 AM, Bill Prince wrote:
>>
>> I see it differently. In the beginning cloudflare thought something was
>> wrong with their systems. It doesn't explain what happened inside facebook,
>> probably because only facebook knows what went wrong. That's not
>> cloudflare's problem. We may never know what happened at facebook probably
>> because it's very embarrassing to facebook.
>>
>>
>> bp
>> <part15sbs{at}gmail{dot}com>
>>
>> On 10/5/2021 9:01 AM, Robert wrote:
>>
>> I really don't like that cloudflare posted this, as it doesn't do
>> anything to tell why. It's not a root cause, it's an effect and cloudflare
>> looks like they are just trying to say how smart they are.
>>
>> On 10/5/21 8:40 AM, Bill Prince wrote:
>>
>>
>> https://blog.cloudflare.com/october-2021-facebook-outage/
>>
>>
>> bp
>> <part15sbs{at}gmail{dot}com>
>>
>> On 10/5/2021 7:39 AM, Chuck McCown via AF wrote:
>>
>> Sign him up for a SNAP card.
>>
>> Sent from my iPhone
>>
>> On Oct 5, 2021, at 8:25 AM, Bill Prince <part15...@gmail.com>
>> <part15...@gmail.com> wrote:
>>
>>
>>
>> Heard the Zuch got around $6 or $7 billion trimmed from his net worth
>> yesterday. Now he's only worth around $121 billion.
>>
>>
>> bp
>> <part15sbs{at}gmail{dot}com>
>>
>> On 10/5/2021 6:52 AM, dmmoff...@gmail.com wrote:
>>
>> I am very curious what it was that took 6 hours to figure out or undo.
>>
>>
>>
>> …..but I wouldn’t judge too harshly. When any contraption gets big and
>> complex it’s easy to overlook one seemingly small thing.
>>
>>
>>
>> Remember this XKCD?
>>
>>
>> <image001.png>
>>
>>
>>
>>
>>
>>
>>
>> *From:* AF <af-boun...@af.afmug.com> <af-boun...@af.afmug.com> *On
>> Behalf Of *Steve Jones
>> *Sent:* Monday, October 04, 2021 6:51 PM
>> *To:* AnimalFarm Microwave Users Group <af@af.afmug.com>
>> <af@af.afmug.com>
>> *Subject:* Re: [AFMUG] facebook outage
>>
>>
>>
>> so it turns out FB murdered the announcements for the DNS server
>> prefixes, including the registrar.
>>
>>
>> https://blog.cloudflare.com/october-2021-facebook-outage/?fbclid=IwAR1NiZ1N3M-tYQzvTVum8bV5OGpi8nLZkb7lqANzioJ7lkG6mjsqTHvhHMQ
>>
>>
>>
>> This really seems like an elementary design flaw. Even a moron like me
>> knows not to poke the DNS subnets all at once, and we probably get way less
>> than 1k daily lookups across all the domains we host DNS for. Ill bet
>> nobody involved in this makes less than 80k a year. facebook lost 4% stock
>> value today attributed to this on top of the almost percent attributed to
>> the "whistleblower". I get nervous when I lose the company a number 10
>> wrench.
>>
>>
>>
>> On Mon, Oct 4, 2021 at 4:51 PM CBB - Jay Fuller <
>> par...@cyberbroadband.net> wrote:
>>
>>
>>
>> it is alive here again
>>
>>
>>
>> ----- Original Message -----
>>
>> *From:* CBB - Jay Fuller <par...@cyberbroadband.net>
>>
>> *To:* AnimalFarm Microwave Users Group <af@af.afmug.com>
>>
>> *Sent:* Monday, October 4, 2021 4:44 PM
>>
>> *Subject:* Re: [AFMUG] facebook outage
>>
>>
>>
>>
>>
>> yup. resolving here. but nothing coming up yet.
>>
>>
>>
>> ----- Original Message -----
>>
>> *From:* Mike Hammett <af...@ics-il.net>
>>
>> *To:* AnimalFarm Microwave Users Group <af@af.afmug.com>
>>
>> *Sent:* Monday, October 4, 2021 4:36 PM
>>
>> *Subject:* Re: [AFMUG] facebook outage
>>
>>
>>
>> They're back... ish.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions <http://www.ics-il.com/>
>> <https://www.facebook.com/ICSIL>
>> <~WRD0002.jpg> <https://www.facebook.com/ICSIL>
>> <~WRD0002.jpg>
>> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb>
>> <~WRD0002.jpg>
>> <https://www.linkedin.com/company/intelligent-computing-solutions>
>> <~WRD0002.jpg>
>> <https://twitter.com/ICSIL>
>> Midwest Internet Exchange <http://www.midwest-ix.com/>
>> <~WRD0002.jpg>
>> <https://www.facebook.com/mdwestix>
>> <~WRD0002.jpg>
>> <https://www.linkedin.com/company/midwest-internet-exchange>
>> <~WRD0002.jpg>
>> <https://twitter.com/mdwestix>
>> The Brothers WISP <http://www.thebrotherswisp.com/>
>> <~WRD0002.jpg>
>> <https://www.facebook.com/thebrotherswisp>
>> <~WRD0002.jpg>
>>
>>
>>
>> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
>> ------------------------------
>>
>> *From: *"CBB - Jay Fuller" <par...@cyberbroadband.net>
>> *To: *af@af.afmug.com
>> *Sent: *Monday, October 4, 2021 1:13:01 PM
>> *Subject: *[AFMUG] facebook outage
>>
>>
>>
>> anyone else feeling alone and isolated without facebook? I need my safe
>> space (no, not myspace) back.
>>
>>
>>
>> babaahahah.
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>
>>
>> ------------------------------
>>
>> --
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>
>> ------------------------------
>>
>> --
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>
>> --
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>
>>
>> --
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>
>>
>>
>>
>>
>>
>>
>> --
>> AF mailing list
>> AF@af.afmug.com
>> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>>
>
>
> --
>
> Carl Peterson
>
> *PORT NETWORKS*
>
> 401 E Pratt St, Ste 2553
>
> Baltimore, MD 21202
>
> (410) 637-3707
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
>
> --
> AF mailing list
> AF@af.afmug.com
> http://af.afmug.com/mailman/listinfo/af_af.afmug.com
>
--
AF mailing list
AF@af.afmug.com
http://af.afmug.com/mailman/listinfo/af_af.afmug.com
