Re: setting ntp with dhcp

2021-10-04 Thread Giovane C. M. Moura via NANOG
Thanks everyone for their answers. To summarize: 1. NTP from DHCP usage: * Not all clients accept DHCP configs for NTP (like android phones) * WiFi VoIP phones use ntp from dhcp (default) -- Google Fiber case showed that VOIP phones may fail to register if DHCP provides

FYI: NANOG and ICANN

2021-10-04 Thread Hank Nussbacher
https://www.icann.org/en/announcements/details/icann-signs-a-memorandum-of-understanding-with-nanog-27-9-2021-en Regards, Hank

Re: ntp with dhcp

2021-10-04 Thread Eliot Lear
On 03.10.21 08:25, Hal Murray wrote: I'm looking for statistics on setting NTP servers on clients using DHCP, in the wild. Does anyone know if there is any available somewhere? That brings up an interesting can of worms. If you run a NAT box with lots of clients, please don't point your NTP cl

Re: FYI: NANOG and ICANN

2021-10-04 Thread Patrick W. Gilmore
NANOG’s version: https://www.nanog.org/stories/nanog-signs-a-memorandum-of-understanding-with-internet-society-icann/ -- TTFN, patrick > On Oct 4, 2021, at 4:42 AM, Hank Nussbacher wrote: > > https://www.icann.org/en/announcements/details/icann-signs-a-memorandum-of-understanding-with-nanog-

massive facebook outage presently

2021-10-04 Thread Eric Kuhnke
https://downdetector.com/status/facebook/ Normally not worth mentioning random $service having an outage here, but this will undoubtedly generate a large volume of customer service calls. Appears to be failure in DNS resolution.

Re: FYI: NANOG and ICANN

2021-10-04 Thread J. Hellenthal via NANOG
You mean they could not come together enough to share even the same page yet even a domain name or central corporation to oversee these needs ? Personally I'm calling partially "bullshit" the Matthew McConaughey way! -- J. Hellenthal The fact that there's a highway to Hell but only a stairway

Re: massive facebook outage presently

2021-10-04 Thread chris
yeah we are seeing same, appears that none of their NS are responding to dns queries affecting all their properties, whatsapp, facebook, instagram etc chris On Mon, Oct 4, 2021 at 12:06 PM Eric Kuhnke wrote: > https://downdetector.com/status/facebook/ > > Normally not worth mentioning rando

Re: massive facebook outage presently

2021-10-04 Thread Darwin Costa
> On 4 Oct 2021, at 18:03, Eric Kuhnke wrote: > > https://downdetector.com/status/facebook/ > > > Normally not worth mentioning random $service having an outage here, but this > will undoubtedly generate a large volume of customer service calls. M

Re: massive facebook outage presently

2021-10-04 Thread George Metz
Also impacting Instagram and, apparently, WhatsApp. On Mon, Oct 4, 2021 at 12:05 PM Eric Kuhnke wrote: > > https://downdetector.com/status/facebook/ > > Normally not worth mentioning random $service having an outage here, but this > will undoubtedly generate a large volume of customer service ca

Re: massive facebook outage presently

2021-10-04 Thread Mel Beckman
Here’s a screenshot: [cid:3E071EF9-BBC5-44BF-865D-2EDC36E05C71-L0-001] -mel beckman On Oct 4, 2021, at 9:06 AM, Eric Kuhnke wrote:  https://downdetector.com/status/facebook/ Normally not worth mentioning random $service having an outage here, but this will undoubtedly generate a large volu

RE: massive facebook outage presently

2021-10-04 Thread Dmitry Sherman
same problem in Israel From: NANOG [mailto:nanog-bounces+dmitry=interhost@nanog.org] On Behalf Of Eric Kuhnke Sent: Monday, 4 October 2021 19:03 To: nanog@nanog.org list Subject: massive facebook outage presently https://downdetector.com/status/facebook/ Normally not worth mentioning rand

Re: FYI: NANOG and ICANN

2021-10-04 Thread Mehmet Akcin
Icann meetings used to be great. They are horrible now with few exceptions of several technical sessions. I hope this don’t ruin nanog meetings On Mon, Oct 4, 2021 at 12:08 J. Hellenthal via NANOG wrote: > You mean they could not come together enough to share even the same page > yet even a dom

Re: massive facebook outage presently

2021-10-04 Thread George Herbert
And WhatsApp and Instagram. Twitter users nationwide agree anecdotally. What I’m getting is DNS failure. -George Sent from my iPhone > On Oct 4, 2021, at 9:07 AM, Eric Kuhnke wrote: > >  > https://downdetector.com/status/facebook/ > > Normally not worth mentioning random $service having

Re: massive facebook outage presently

2021-10-04 Thread Eric Kuhnke
Considering the massive impact of this it would be interesting to see some traffic graphs from ISPs that have PNIs with Facebook, or high volume peering sessions across an IX, showing traffic to FB falling off a cliff. On Mon, Oct 4, 2021 at 12:16 PM George Herbert wrote: > And WhatsApp and Inst

Re: massive facebook outage presently

2021-10-04 Thread Hugo Slabbert
Looks like their auth DNS dropped out of the DFZ: https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20 https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20 -- Hugo Slabbert On Mon, Oct 4, 2021 at 9:21 AM George Metz wrote: > Also impacting Instagram and, apparently, Wh

Re: massive facebook outage presently

2021-10-04 Thread Matthew Petach
Not so much a DNS failure as an "oops, where did the routes to our auth servers go?" moment. or, in this case, several moments... ^_^;; Matt On Mon, Oct 4, 2021 at 9:07 AM Eric Kuhnke wrote: > https://downdetector.com/status/facebook/ > > Normally not worth mentioning random $service having

Re: massive facebook outage presently

2021-10-04 Thread Jason Kuehl
Looks like they run there own nameservers and I see the soa records are even missing. On Mon, Oct 4, 2021, 12:23 PM Mel Beckman wrote: > Here’s a screenshot: > > > > -mel beckman > > On Oct 4, 2021, at 9:06 AM, Eric Kuhnke wrote: > >  > https://downdetector.com/status/facebook/ > > Normally n

Re: FYI: NANOG and ICANN

2021-10-04 Thread Edward McNair
This partnership will have no ill effect on NANOG conferences. The focus of these partnerships are on outreach, technical education and underserved communities in North America. Edward McNair Executive Director emcn...@nanog.org | +1 866-902-1336 ext. 102 | www.nanog.org NANOG | 305 E. Eisenh

Re: massive facebook outage presently

2021-10-04 Thread Tom Beecher
Curious if there is a malicious angle after the 60 Minutes story last night. On Mon, Oct 4, 2021 at 12:26 PM Dmitry Sherman wrote: > same problem in Israel > > > > > > *From:* NANOG [mailto:nanog-bounces+dmitry=interhost@nanog.org] *On > Behalf Of *Eric Kuhnke > *Sent:* Monday, 4 October 202

RE: massive facebook outage presently

2021-10-04 Thread michael
Looks bigger than DNS. Some people are saying they’ve disappeared off the DFZ. From: NANOG On Behalf Of Dmitry Sherman Sent: Monday, October 4, 2021 12:10 PM To: Eric Kuhnke ; nanog@nanog.org list Subject: RE: massive facebook outage presently same problem in Israel From: NANOG [m

Re: massive facebook outage presently

2021-10-04 Thread Jason Kuehl
Yeah it looks like there dns servers are just dead. I can't get an response from them. On Mon, Oct 4, 2021, 12:26 PM Dmitry Sherman wrote: > same problem in Israel > > > > > > *From:* NANOG [mailto:nanog-bounces+dmitry=interhost@nanog.org] *On > Behalf Of *Eric Kuhnke > *Sent:* Monday, 4 Oct

Re: massive facebook outage presently

2021-10-04 Thread Adrian Minta
https://www.youtube.com/watch?v=gmk673M5BoA On 10/4/21 7:03 PM, Eric Kuhnke wrote: https://downdetector.com/status/facebook/ Normally not worth mentioning random $service having an outage here, but this will undoubtedly generate a large volume of cu

Re: massive facebook outage presently

2021-10-04 Thread Tom Beecher
I see the same. Still see those prefixes via direct peering, but DFZ doesn't have them. Their auths are not reachable even though I have routes for them. Some other weirdness I'm poking a bit on that doesn't seem like it could possibly be related to FB DNS though. On Mon, Oct 4, 2021 at 12:39 PM

IRR for IX peers

2021-10-04 Thread Randy Bush
so i have an AS (3130) which peers at the SIX (RSs and some direct). in the hope that leak detectors such as artemis would stop false positives when they see my prefixes announced customer cones of SIX peers, i want to add the SIX peers to my aut-num: policy. export: toAS-SEATTLEIX-RS-CLIENT

Re: massive facebook outage presently

2021-10-04 Thread Matthew Petach
On Mon, Oct 4, 2021 at 9:47 AM Jason Kuehl wrote: > Looks like they run there own nameservers and I see the soa records are > even missing. > > On Mon, Oct 4, 2021, 12:23 PM Mel Beckman wrote: > >> Here’s a screenshot: >> >> >> >> -mel beckman >> >> On Oct 4, 2021, at 9:06 AM, Eric Kuhnke wrot

Re: massive facebook outage presently

2021-10-04 Thread richey goldberg
In other news worker productivity is up 100% today. -richey From: NANOG on behalf of Jason Kuehl Date: Monday, October 4, 2021 at 12:45 PM To: Mel Beckman Cc: nanog@nanog.org list Subject: Re: massive facebook outage presently Looks like they run there own nameservers and I see the soa recor

Re: massive facebook outage presently

2021-10-04 Thread Jay Hennigan
On 10/4/21 09:27, Tom Beecher wrote: Curious if there is a malicious angle after the 60 Minutes story last night. Hanlon's razor. "Never attribute to malice that which is adequately explained by stupidity" Corollary - "But don't rule out malice." -- Jay Hennigan - j...@west.net Network Engi

Re: massive facebook outage presently

2021-10-04 Thread Tom Beecher
They haven't completely dropped off, but the big subnets certainly have. I'm only seeing 20-odd /24s from them via the DFZ, but everything larger still directly. On Mon, Oct 4, 2021 at 12:55 PM wrote: > Looks bigger than DNS. Some people are saying they’ve disappeared off the > DFZ. > > > > *Fro

Re: massive facebook outage presently

2021-10-04 Thread Anne P. Mitchell, Esq.
> On Oct 4, 2021, at 10:34 AM, Adrian Minta wrote: > > https://www.youtube.com/watch?v=gmk673M5BoA Because it's helpful to know what a link is about before clicking on it, Adrian's link goes to "The Onion Movie - Internet Down" On a related note, what do you think the scene is like in FB HQ

Re: massive facebook outage presently

2021-10-04 Thread Michael Spears
https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/?utm_medium=android_app&utm_source=share Sent from my Verizon, Samsung Galaxy smartphoneGet Outlook for Android

Re: massive facebook outage presently

2021-10-04 Thread Casey Russell via NANOG
>> In other news worker productivity is up 100% today. For everyone except IT workers. Although, I suppose if you're just counting the number of tickets they can quickly clear by sending out a "It's the internet, not us". You could count that as increased productivity. Sincerely, Casey Russell

Re: massive facebook outage presently

2021-10-04 Thread Niels Bakker
* m...@beckman.org (Mel Beckman) [Mon 04 Oct 2021, 18:23 CEST]: Here’s a screenshot: [cid:3E071EF9-BBC5-44BF-865D-2EDC36E05C71-L0-001] Please don't do this on the NANOG list. -- Niels.

Re: massive facebook outage presently

2021-10-04 Thread Mel Beckman
Suspiciously, this comes the morning after Facebook whistleblower Frances Haugen disclosed on 60 Minutes that Facebook's own research shows that it chose to profit from misinformation and political unrest through deliberate escalation of conflicts. Occam’s razor says “When multiple causes are p

RE: massive facebook outage presently

2021-10-04 Thread aaron1
Yes, embedded ISP CDN’s show a huge drop -Aaron From: NANOG On Behalf Of Eric Kuhnke Sent: Monday, October 4, 2021 11:22 AM To: George Herbert ; nanog@nanog.org list Subject: Re: massive facebook outage presently Considering the massive impact of this it would be interesting to see s

Re: massive facebook outage presently

2021-10-04 Thread Sabri Berisha
- On Oct 4, 2021, at 10:07 AM, Anne P. Mitchell, Esq. amitch...@isipp.com wrote: Hi Anne, > On a related note, what do you think the scene is like in FB HQ right now? > (shaking head) Very quiet, as their offices are still closed for all but essentials :) But, from experience I can tell yo

Re: massive facebook outage presently

2021-10-04 Thread Sabri Berisha
Hi, Oops, this was not supposed to go to the list, apologies for the clutter. Thanks, Sabri - On Oct 4, 2021, at 10:46 AM, Sabri Berisha sa...@cluecentral.net wrote: > - On Oct 4, 2021, at 10:07 AM, Anne P. Mitchell, Esq. amitch...@isipp.com

Re: massive facebook outage presently

2021-10-04 Thread Jonathan Kalbfeld via NANOG
With Facebook down, how are people doing their vaccine research? It’s got to be more than just DNS. Jonathan Kalbfeld office: +1 310 317 7933 fax:+1 310 317 7901 home: +1 310 317 7909 mobile: +1 310 227 1662 ThoughtWave Technologies, Inc. Studio City, CA 91604 https://thoughtwave.co

The Outage

2021-10-04 Thread Andy Ringsmuth
I suppose it could just be me. Or it could be more than just me. Anyone else noticing that at the same time as the giant FB outage, both the outages and outages-discussion lists are suddenly slow as molasses in January too? Checking some headers on messages that do make it through to outages,

Re: massive facebook outage presently

2021-10-04 Thread Jay Hennigan
On 10/4/21 10:35, Mel Beckman wrote: Suspiciously, this comes the morning after Facebook whistleblower Frances Haugen disclosed on 60 Minutes that Facebook's own research shows that it chose to profit from misinformation and political unrest through deliberate escalation of conflicts. Occam’s r

Re: The Outage

2021-10-04 Thread Mike Hammett
Jared, would you be able to post any system load graphs showing what the mail server is like during these kinda of massive outages? With that many users on the mailing list, I can only imagine that it takes forever to process sending those messages out to tens of thousands of people at a time

Re: massive facebook outage presently

2021-10-04 Thread Martin List-Petersen
Wishful thinking .. but hey, one is allowed to dream. /M On 04/10/2021 18:57, Jay Hennigan wrote: On 10/4/21 10:35, Mel Beckman wrote: Suspiciously, this comes the morning after Facebook whistleblower Frances Haugen disclosed on 60 Minutes that Facebook's own research shows that it chose to

Re: massive facebook outage presently

2021-10-04 Thread Billy Croan
I know what this is. They forgot to update the credit card on their godaddy account and the domain lapsed. I guess it will be facebook.info when they get it back online. The post mortem should be an interesting read. On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl wrote: > Looks like they run

Re: The Outage

2021-10-04 Thread Matthew Petach
On Mon, Oct 4, 2021 at 10:58 AM Andy Ringsmuth wrote: > I suppose it could just be me. > > Or it could be more than just me. > > Anyone else noticing that at the same time as the giant FB outage, both > the outages and outages-discussion lists are suddenly slow as molasses in > January too? > > C

Re: The Outage

2021-10-04 Thread Andy Ringsmuth
> On Oct 4, 2021, at 1:31 PM, Matthew Petach wrote: > > > > On Mon, Oct 4, 2021 at 10:58 AM Andy Ringsmuth wrote: > I suppose it could just be me. > > Or it could be more than just me. > > Anyone else noticing that at the same time as the giant FB outage, both the > outages and outages-di

Re: massive facebook outage presently

2021-10-04 Thread Baldur Norddahl
I got a mail that Facebook was leaving NLIX. Maybe someone botched the script so they took down all BGP sessions instead of just NLIX and now they can't access the equipment to put it back... :-) man. 4. okt. 2021 20.31 skrev Billy Croan : > I know what this is. They forgot to update the cr

Re: massive facebook outage presently

2021-10-04 Thread Blake Dunlap
You laugh but that kind of sounds like what happened so far as oops we isolated prod and are scrambling on DR. There was someone supposedly live tweeting from their incident response for a bit before their account panic deleted. On Mon, Oct 4, 2021, 13:42 Baldur Norddahl wrote: > I got a mail th

Re: massive facebook outage presently

2021-10-04 Thread Sabri Berisha
- On Oct 4, 2021, at 11:41 AM, Baldur Norddahl baldur.nordd...@gmail.com wrote: Hi, > I got a mail that Facebook was leaving NLIX. Maybe someone botched the script > so > they took down all BGP sessions instead of just NLIX and now they can't access > the equipment to put it back... :-) Th

RE: massive facebook outage presently

2021-10-04 Thread Luke Guillory
From what I believe was a FB employee on Reddit, account now deleted it seems. As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration

Re: massive facebook outage presently

2021-10-04 Thread Jason Kuehl
I mean, you're an idiot if you post that public on the internet about your own place of work. What do you think would happen? Nothing? He should never of said anything, but now the Facebook hitman got him. Facebook will have to send out a Reason For Outage with all the services it's effecting, lik

RE: massive facebook outage presently

2021-10-04 Thread Tony Wicks
Didn't write that part of the automation script and that coder left... > I got a mail that Facebook was leaving NLIX. Maybe someone botched the > script so they took down all BGP sessions instead of just NLIX and now > they can't access the equipment to put it back... :-)

Re: massive facebook outage presently

2021-10-04 Thread Jason Kuehl
https://twitter.com/disclosetv/status/1445100931947892736?s=20 On Mon, Oct 4, 2021 at 3:01 PM Tony Wicks wrote: > Didn't write that part of the automation script and that coder left... > > > I got a mail that Facebook was leaving NLIX. Maybe someone botched the > > script so they took down all B

Re: massive facebook outage presently

2021-10-04 Thread tomocha
Hi Some of the DNS addresses are no longer prefix from AS32934. On 2021/10/05 1:23, Hugo Slabbert wrote: Looks like their auth DNS dropped out of the DFZ: https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20 https://twitter.com/g_bonfiglio/status/1445058771261313046?s=20

Re: massive facebook outage presently

2021-10-04 Thread bzs
Although I believe it's generally true that if a company appears prominently in the news it's liable to be attacked I assume because the miscreants sit around thinking "hmm, who shall we attack today oh look at that shiny headline!" I'd hate to ascribe any altruistic motivation w/o some evidence

RE: massive facebook outage presently

2021-10-04 Thread Jean St-Laurent via NANOG
Maybe the key to solve this issue is in an email sent to some_very_important_t...@facebook.com -Original Message- From: NANOG On Behalf Of tomocha Sent: October 4, 2021 2:32 PM To: nanog@nanog.org Subject: Re: massive facebook outage presently Hi Some of the DNS addresses are no longer

Re: massive facebook outage presently

2021-10-04 Thread Jay Hennigan
On 10/4/21 12:11, b...@theworld.com wrote: Although I believe it's generally true that if a company appears prominently in the news it's liable to be attacked I assume because the miscreants sit around thinking "hmm, who shall we attack today oh look at that shiny headline!" I'd hate to ascribe

Re: massive facebook outage presently

2021-10-04 Thread Töma Gavrichenkov
Peace, On Mon, Oct 4, 2021, 10:17 PM Jean St-Laurent via NANOG wrote: > Maybe the key to solve this issue is in an email sent to > some_very_important_t...@facebook.com Yeah except MX records on facebook dot com aren't working either -- Töma

Re: IRR for IX peers

2021-10-04 Thread Nick Hilliard
Randy Bush wrote on 04/10/2021 17:44: what are others in this space doing? not using import/export lines in their RS or router configs, for starters. Probably you could count the number of IXPs that inspect import/export lines on the fingers of one hand, and possibly of one finger. General

Re: massive facebook outage presently

2021-10-04 Thread Michael Thomas
On 10/4/21 11:48 AM, Luke Guillory wrote: I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical

Re: IRR for IX peers

2021-10-04 Thread Rubens Kuhl
Some IX'es set communities telling which member announced that prefix; if SIX is one of those, that can be used to automate origin verification. Rubens On Mon, Oct 4, 2021 at 2:08 PM Randy Bush wrote: > > so i have an AS (3130) which peers at the SIX (RSs and some direct). > > in the hope that

Re: massive facebook outage presently

2021-10-04 Thread Baldur Norddahl
On Mon, 4 Oct 2021 at 21:58, Michael Thomas wrote: > > On 10/4/21 11:48 AM, Luke Guillory wrote: > > > I believe the original change was 'automatic' (as in configuration done > via a web interface). However, now that connection to the outside world is > down, remote access to those tools don't ex

Re: IRR for IX peers

2021-10-04 Thread Randy Bush
>> what are others in this space doing? > > not using import/export lines in their RS or router configs, for > starters. Probably you could count the number of IXPs that inspect > import/export lines on the fingers of one hand, and possibly of one > finger. > > Generally speaking, IXPs try to ai

Re: massive facebook outage presently

2021-10-04 Thread Baldur Norddahl
Not in such a primitive fashion no. But they could definitely have a secondary network that will continue to work even if something goes wrong with the primary. On Mon, 4 Oct 2021 at 22:16, PJ Capelli wrote: > Seems unlikely that FB internal controls would allow such a backdoor ... > > "Never to

Re: massive facebook outage presently

2021-10-04 Thread Matthew Petach
On Mon, Oct 4, 2021 at 11:59 AM Jason Kuehl wrote: > I mean, you're an idiot if you post that public on the internet about > your own place of work. What do you think would happen? Nothing? He should > never of said anything, but now the Facebook hitman got him. > > Some of us have done that, and

Re: massive facebook outage presently

2021-10-04 Thread Łukasz Bromirski
Dual homing won’t help you if your automation template will do „no router bgp X” and at this point session will terminate as suddenly advertisement will be withdrawn… It won’t you either if the change triggers some obscure bug in your BGP stack. I bet FB tested the change on smaller scale and

Re: massive facebook outage presently

2021-10-04 Thread Blake Dunlap
If there isn't an undernetwork capable of being backdoored with the proper keys (I'd be shocked if there isn't - the big players have very good infra and DR people), I suspect there will be one soonish. It doesnt do much good to have DR plans and keys otherwise if you can't even get to the locks w

Re: massive facebook outage presently

2021-10-04 Thread Eric Kuhnke
I am starting to see reports that in ISPs with very large numbers of residential users, customers are starting to press the factory-reset buttons on their home routers/modems/whatever, in an attempt to make Facebook work. This is resulting in much heavier than normal first tier support volumes. The

Re: massive facebook outage presently

2021-10-04 Thread richey goldberg
No evidence that it’s intentional but….. What’s going to be the big headline tonight?A Facebook whistleblower or a global outage that kept everyone from arguing all day long? -richey From: NANOG on behalf of Jay Hennigan Date: Monday, October 4, 2021 at 3:30 PM To: nanog@nanog.org S

Re: massive facebook outage presently

2021-10-04 Thread Hank Nussbacher
On 04/10/2021 22:05, Jason Kuehl wrote: BGP related: https://twitter.com/SGgrc/status/1445116435731296256 as also related by FB CTO: https://twitter.com/atoonk/status/1445121351707070468 -Hank https://twitter.com/disclosetv/status/1445100931947892736?s=20

Re: massive facebook outage presently

2021-10-04 Thread Miles Fidelman
Jay Hennigan wrote: On 10/4/21 09:27, Tom Beecher wrote: Curious if there is a malicious angle after the 60 Minutes story last night. Hanlon's razor. "Never attribute to malice that which is adequately explained by stupidity" Corollary - "But don't rule out malice." "Once is happenstance

Re: massive facebook outage presently

2021-10-04 Thread Aaron C. de Bruyn via NANOG
It looks like it might take a while according to a news reporter's tweet: "Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors." https://twitter.co

Re: massive facebook outage presently

2021-10-04 Thread Mel Beckman
I’m not the only one who finds this timing suspicious, Starting with the publishers of 60 Minutes themselves :-) CBS: The outage comes the morning after "60 Minutes" aired an interview with a whistleblower who said Facebook is aware of how it amplifies hate, misinformation and unrest but claim

RE: massive facebook outage presently

2021-10-04 Thread Jean St-Laurent via NANOG
The glue records for the NS are set at 48 hours. dig @c.gtld-servers.net. facebook.com. NS ;; facebook.com. 172800 IN NS a.ns.facebook.com. facebook.com. 172800 IN NS b.ns.facebook.com. facebook.com. 172800 IN NS c.ns.facebook.c

Re: massive facebook outage presently

2021-10-04 Thread Mark Tinka
On 10/4/21 18:22, Eric Kuhnke wrote: Considering the massive impact of this it would be interesting to see some traffic graphs from ISPs that have PNIs with Facebook, or high volume peering sessions across an IX, showing traffic to FB falling off a cliff. Our PNI's and FNA's in Africa hav

Re: massive facebook outage presently

2021-10-04 Thread Matt Hoppes
Yes, We've seen that. On 10/4/21 4:33 PM, Eric Kuhnke wrote: I am starting to see reports that in ISPs with very large numbers of residential users, customers are starting to press the factory-reset buttons on their home routers/modems/whatever, in an attempt to make Facebook work. This is res

Re: massive facebook outage presently

2021-10-04 Thread None None
Not all employees are having this issue But it is curious why the records can be moved to the name servers that are still being advertised over the Internet still . On Mon, Oct 4, 2021 at 4:53 PM Aaron C. de Bruyn via NANOG wrote: > It looks like it might take a while according to a news report

Re: facebook outage

2021-10-04 Thread Bill Woodcock
They’re starting to pick themselves back up off the floor in the last two or three minutes. A few answers getting out. I imagine it’ll take a while before things stabilize, though. -Bill signature.asc Description: Message signed with OpenPGP

Re: massive facebook outage presently

2021-10-04 Thread Jay Hennigan
Starting to see some prefixes return on our peering session with them in Los Angeles but DNS still not resolving. -- Jay Hennigan - j...@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

Re: massive facebook outage presently

2021-10-04 Thread Bryan Fields
On 10/4/21 4:47 PM, Jean St-Laurent via NANOG wrote: > > dig @c.gtld-servers.net. facebook.com. NS > ;; > facebook.com. 172800 IN NS a.ns.facebook.com. > facebook.com. 172800 IN NS b.ns.facebook.com. > facebook.com. 172800 IN NS c.ns.

Re: facebook outage

2021-10-04 Thread Bill Woodcock
> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > > They’re starting to pick themselves back up off the floor in the last two or > three minutes. A few answers getting out. I imagine it’ll take a while > before things stabilize, though. nd we’re back: WoodyNet-2:.ssh woody$ dig ww

Re: facebook outage

2021-10-04 Thread Jeff Shultz
Now they just need to get the site itself back up. On Mon, Oct 4, 2021 at 2:25 PM Bill Woodcock wrote: > > > > On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > > > > They’re starting to pick themselves back up off the floor in the last > two or three minutes. A few answers getting out. I i

Re: facebook outage

2021-10-04 Thread Bill Woodcock
> On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: >> >> They’re starting to pick themselves back up off the floor in the last two or >> three minutes. A few answers getting out. I imagine it’ll take a while >> before things st

Re: IRR for IX peers

2021-10-04 Thread Ben Maddison via NANOG
Hi Randy, On 10/04, Randy Bush wrote: > so i have an AS (3130) which peers at the SIX (RSs and some direct). > > in the hope that leak detectors such as artemis would stop false > positives when they see my prefixes announced customer cones of SIX > peers, i want to add the SIX peers to my aut-nu

Re: facebook outage

2021-10-04 Thread Mike Lyon
Can't wait to see the RFO! -Mike On Mon, Oct 4, 2021 at 2:35 PM Bill Woodcock wrote: > > > > > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: > > > > > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > >> > >> They’re starting to pick themselves back up off the floor in the last tw

Re: massive facebook outage presently

2021-10-04 Thread bzs
17:35EDT: I'm suddenly getting: Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can. Go Back Facebook © 2020 · Help Center and: % host facebook.com facebook.com has address 157.

Re: facebook outage

2021-10-04 Thread Baldur Norddahl
man. 4. okt. 2021 23.33 skrev Bill Woodcock : > > > > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: > > > > > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > >> > >> They’re starting to pick themselves back up off the floor in the last > two or three minutes. A few answers gettin

Re: IRR for IX peers

2021-10-04 Thread Randy Bush
hi ben, a SIX peer's customer could be the feed to RIS randy

RE: massive facebook outage presently

2021-10-04 Thread Tony Wicks
Back and working by the looks.

Re: facebook outage

2021-10-04 Thread Ryan Brooks
> On Oct 4, 2021, at 4:30 PM, Bill Woodcock wrote: > > > >> On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: >> >> >> >>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: >>> >>> They’re starting to pick themselves back up off the floor in the last two >>> or three minutes. A few an

Re: massive facebook outage presently

2021-10-04 Thread bzs
One might think in over six hours they could point facebook.com's DNS somewhere else and put up a page with some info about the outage there, that this would be a practiced firedrill. Yeah yeah cache blah blah but it'd get around and at least would be coming from them. I'd imagine some mutual pa

Re: facebook outage

2021-10-04 Thread Jeff Shultz
On Mon, Oct 4, 2021 at 2:49 PM Baldur Norddahl wrote: > > > man. 4. okt. 2021 23.33 skrev Bill Woodcock : > >> >> >> > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: >> > >> > >> > >> >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: >> >> >> >> They’re starting to pick themselves back up

Re: facebook outage

2021-10-04 Thread Patrick W. Gilmore
On Oct 4, 2021, at 5:30 PM, Bill Woodcock wrote: > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: >>> >>> They’re starting to pick themselves back up off the floor in the last two >>> or three minutes. A few answers getting out. I imagi

Re: facebook outage

2021-10-04 Thread Michael Thomas
On 10/4/21 2:41 PM, Baldur Norddahl wrote: man. 4. okt. 2021 23.33 skrev Bill Woodcock >: > On Oct 4, 2021, at 11:21 PM, Bill Woodcock mailto:wo...@pch.net>> wrote: > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock mailto:wo...@pch.net>> wrote:

Re: facebook outage

2021-10-04 Thread Bill Woodcock
> On Oct 4, 2021, at 11:50 PM, Ryan Brooks wrote: > DNS was a victim in this outage, not the cause. You are absolutely correct. However, people who don’t have this problem avoid having this problem by not putting all their DNS eggs in one basket. And then forgetting where they put the basket

Re: facebook outage

2021-10-04 Thread chris
Hopefully this will show people they can enjoy life and survive without it and that it will be just like myspace some day :) On Mon, Oct 4, 2021 at 5:31 PM Jeff Shultz wrote: > Now they just need to get the site itself back up. > > On Mon, Oct 4, 2021 at 2:25 PM Bill Woodcock wrote: > >> >> >>

Re: facebook outage (resolving)

2021-10-04 Thread mike tancsa
Getting the odd message through, but DNS looks good via their Toronto, Canada pop % traceroute -q1 -I a.dns.facebook.com traceroute to star.c10r.facebook.com (31.13.80.8), 64 hops max, 48 byte packets  1  torix-core1-10G (67.43.129.248)  0.140 ms  2  facebook-a.ip4.torontointernetxchange.net (206

Re: facebook outage

2021-10-04 Thread Bill Woodcock
> On Oct 4, 2021, at 11:41 PM, Baldur Norddahl > wrote: > > > > man. 4. okt. 2021 23.33 skrev Bill Woodcock : > > > > On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: > > > > > > > >> On Oct 4, 2021, at 11:10 PM, Bill Woodcock wrote: > >> > >> They’re starting to pick themselves back up

Re: IRR for IX peers

2021-10-04 Thread Ben Maddison via NANOG
Hi Randy, On 10/04, Randy Bush wrote: > hi ben, > > a SIX peer's customer could be the feed to RIS > Sure, but how do you describe the policy between your peer and their customer in your aut-num?! That's not a thing. Cheers, Ben signature.asc Description: PGP signature

Re: IRR for IX peers

2021-10-04 Thread Randy Bush
>> a SIX peer's customer could be the feed to RIS > Sure, but how do you describe the policy between your peer and their > customer in your aut-num?! That's not a thing. yup these rat holes are a pita randy

Re: facebook outage

2021-10-04 Thread Bill Woodcock
> On Oct 5, 2021, at 12:16 AM, Bill Woodcock wrote: > > > >> On Oct 4, 2021, at 11:41 PM, Baldur Norddahl >> wrote: >> >> >> >> man. 4. okt. 2021 23.33 skrev Bill Woodcock : >> >> >>> On Oct 4, 2021, at 11:21 PM, Bill Woodcock wrote: >>> >>> >>> On Oct 4, 2021, at 11:10 PM, Bi

Re: massive facebook outage presently

2021-10-04 Thread Niels Bakker
* b...@theworld.com (b...@theworld.com) [Tue 05 Oct 2021, 00:01 CEST]: I only mean a single, simple information page like the "sorry we're working on it" I saw just before they came back. Which would subsequently receive the world's FB session cookies. -- Niels.

  1   2   >