As is Generalissimo Francisco Franco. Indeed, folks; please move these meta conversations to the -discuss list; they are off topic for the main notification list.
Cheers, -- jr '<admin/>' a ----- Original Message ----- > From: "Mike Lyon via Outages" <outages@outages.org> > To: "T.Suzuki" <tss-out...@e-ontap.com> > Cc: "Michael Loftis via Outages" <outages@outages.org> > Sent: Sunday, March 26, 2023 8:17:25 PM > Subject: Re: [outages] FAA.gov nameserver outage > Can’t believe it’s still dead… > > -Mike > >> On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages@outages.org> wrote: >> >> On Sun, 26 Mar 2023 08:35:29 -0700 >> Hugo Slabbert <h...@slabnet.com> wrote: >> >>> What would be the symptoms here of a "water torture attack" rather than >>> what John had indicated as a firewall failure in their infrastructure: >>> >>>> Initial looks from the firewall team point to an automatic failover event >>> and the secondary failed. >>> >>> And the symptoms of which lined up with network level info from Paul >>> earlier: >>> >>>> They only seem to have two auth nameservers for faa, both within the >>> faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the >>> servers are in all die just within each block run by the FAA. >>>> >>>> Seems like an internal routing meltdown making the only 2 nameservers >>> unreachable reliably. >>> >>> Are you saying that your open resolvers have a per client rate limit >>> applied, that rate limit got tripped, and shortly thereafter the resolvers >>> became unavailable, suggesting query floods for the domain(s) that knocked >>> the resolvers offline (or from the other discussion, possibly was the thing >>> that overwhelmed that firewall layer, causing the initial failover and >>> possibly also causing the firewall secondary to fail to come online)? >> >> Yes. (limitting per client, and per second for all) >> Perhaps, large numbers open resolvers including no ratelimit are used. >> Then massive random subdomain queries caused the firewall symptoms. >> (It's only my guess.) >> >>>> On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org> >>>> wrote: >>>> >>>> Hi, I'm a researcher of DNS vulnerabilities. >>>> >>>> It loos like random subdomain attacks (water tourtue attack). >>>> >>>> This is the data of my rate-limitted openresolver as a honeypot. >>>> http://www.e-ontap.com/dns/todaydowngov.txt >>>> http://www.e-ontap.com/dns/todaydown.txt >>>> (You can not view these page if you are using 8.8.8.8, sorry.) >>>> >>>> Raw logs of my Unbound (Time is JST) >>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head >>>> -5 >>>> Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL < >>>> unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. >>>> Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 >>>> unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 >>>> Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: >>>> exceeded ratelimit for zone faa.gov. >>>> Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN >>>> SERVFAIL 15.112813 0 30 >>>> Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: >>>> exceeded ratelimit for zone faa.gov. >>>> local/etc/unbound% >>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | >>>> head -5 >>>> Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: >>>> all servers for this domain failed, at zone faa.gov. from >>>> 2620:74:27::2:30 no server to query nameserver addresses not usable >>>> Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. >>>> A IN>: all servers for this domain failed, at zone faa.gov. no server to >>>> query nameserver addresses not usable >>>> Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all >>>> servers for this domain failed, at zone faa.gov. no server to query >>>> nameserver addresses not usable >>>> Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: >>>> all servers for this domain failed, at zone faa.gov. upstream server >>>> timeout >>>> Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. >>>> A IN>: all servers for this domain failed, at zone faa.gov. upstream >>>> server timeout >>>> local/etc/unbound% >>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | >>>> tail -5 >>>> Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all >>>> servers for this domain failed, at zone faa.gov. no server to query >>>> nameserver addresses not usable >>>> Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. >>>> A IN>: all servers for this domain failed, at zone faa.gov. no server to >>>> query nameserver addresses not usable >>>> Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL < >>>> eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at >>>> zone faa.gov. no server to query nameserver addresses not usable >>>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < >>>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, >>>> at zone faa.gov. no server to query nameserver addresses not usable >>>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A >>>> IN>: all servers for this domain failed, at zone faa.gov. no server to >>>> query nameserver addresses not usable >>>> local/etc/unbound% >>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail >>>> -5 >>>> Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 >>>> eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 >>>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < >>>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, >>>> at zone faa.gov. no server to query nameserver addresses not usable >>>> Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 >>>> faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 >>>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A >>>> IN>: all servers for this domain failed, at zone faa.gov. no server to >>>> query nameserver addresses not usable >>>> Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A >>>> IN SERVFAIL 0.000000 0 34 >>>> local/etc/unbound% >>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc >>>> -l >>>> 1408 >>>> >>>> -- >>>> T.Suzuki >>>> -- >>>> T.Suzuki / E.F.シューマッハーとI.イリイチを読もう >>>> _______________________________________________ >>>> Outages mailing list >>>> Outages@outages.org >>>> https://puck.nether.net/mailman/listinfo/outages >>>> >> >> >> -- >> T.Suzuki / E.F.シューマッハーとI.イリイチを読もう >> _______________________________________________ >> Outages mailing list >> Outages@outages.org >> https://puck.nether.net/mailman/listinfo/outages > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
