Re: Digicert revoking certain certs failing CNAME validation
On 31/07/2024 7:14, Peter Fisher wrote: These short and immediate revocations as well as other issues will keep happening since the CA/B has no end user representation. See my blog post from 4 years ago: https://www.iucc.ac.il/en/blog/internet-certificates/ Regards, Hank Actually it looks like they have updated their incident page (https://status.digicert.com/incidents/3sccz3v31lc9) with a new revocation date depending on if you get an exception. Also more details can be found here(https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c5). On Tue, Jul 30, 2024 at 12:45 PM Peter Fisher wrote: I have not noticed any revocation yet for my affected certificates. Has anyone had their affected certificates revoked yet? On Tue, Jul 30, 2024 at 12:35 PM Innocent Obi wrote: Luckily it seems my org has since mitigated this, but it would be interesting to know the broader impacts/who is broadly impacted. On Tue, Jul 30, 2024 at 12:20 PM Tom Beecher wrote: If you're only getting this now, you're probably in trouble, because they're revoking affected certs in about 15 mins. On Tue, Jul 30, 2024 at 2:53 PM Innocent Obi wrote: Just in-case this hasn't made its way around: https://www.digicert.com/support/certificate-revocation-incident.
Dissecting the FCC’s Proposal to Improve BGP Security
https://www.kentik.com/blog/dissecting-the-fccs-proposal-to-improve-bgp-security/ -Hank
Re: Mailing list SPF Failure
On 17/05/2024 5:45, Karl Auer wrote: On Thu, 2024-05-16 at 19:27 -0700, Michael Thomas wrote: On 5/16/24 7:22 PM, Scott Q. wrote: Mike, you do realize Google/Gmail rejects e-mails with invalid/missing SPF right ? I was receiving the mail while NANOG had no SPF record, so no? Any receiver would be really stupid take a single signal as disqualifying. For small-scale senders, it's either or both. For large-scale senders (5000+ per day) it's both. At least according to this: https://support.google.com/a/answer/81126 I think some may have missed these announcements: https://labs.ripe.net/author/fergalc/enhancing-email-delivery-at-the-ripe-ncc/ https://blog.google/products/gmail/gmail-security-authentication-spam-protection/ Regards, Hank
Re: puck not responding
On 29/02/2024 17:21, Jared Mauch wrote: On behalf of cisco-nsp and outages - we salute you. -Hank On Feb 28, 2024, at 1:30 AM, Daniel Marks via NANOG wrote: We’re getting rocked by storms here in Michigan, could be related. [ brief version of what happened from what I can tell reconstructing things] I was alerted ~4am US/E yesterday about the issue. This machine has been generously hosted by my previous employer for quite some time, funnily enough it was 7 years ago almost to the day since I started my current employment. The IPMI was not responsive and the machine was located in 350 Cermak, on a floor that was not impacted with the heat/cold event. I have been meaning to move things off and on, but never quite had the motivation to tackle the task. Yesterday forced my hand. Once I confirmed that we could get the machine out of the colocation facility (thank you again NTT) I drove from Michigan to Chicago, got lunch and picked up the machine and headed back to the colocation that I have in Michigan at the 123Net/DetroitIX site. Once I had a console on it, I determined that this old machine had a few things that had been gradually updated and upgraded over time, not all the filesystem options were set correctly and after some tune2fs options were set and fstab updated to ensure everything is migrated fully from ext2 -> ext4 the system was able to be booted without issues. Afterwards I’ve determined that there is still a hardware related problem, so I am now going to move it to new hardware later today schedule permitting as I want to go onsite and make sure that the I/O is performant. Feb 28 22:09:05 kernel: Memory: 32816872K/33544380K available (20480K kernel code, 3276K rwdata, 14748K rodata, 4588K init, 4892K bss, 727248K reserved, 0K cma-reserved) Feb 29 00:20:07 kernel: Memory: 16326408K/16767164K available (20480K kernel code, 3276K rwdata, 14748K rodata, 4588K init, 4892K bss, 440496K reserved, 0K cma-reserved) Not quite a great thing when nobody is onsite and the machine requires being power cycled and the amount of memory changes. If you are seeing any other issues, do let me know, I did move the IPv4 space but have renumbered for v6, so if you use my free secondary dns service, and your own vanity name, you will need to update your records. If you are seeing any reachability issues let me know, there should be ROA and other objects in place for things. Sorry everyone got this email, feel bad it’s like when warren asked the list some personal details :-) - Jared (Even more details: changing disk images from qcow -> qcow2 and other things like ext2 -> ext3/4 over all the years as the machine has gone from Linux -> FreeBSD -> Linux again and other things is always a fun way to keep bringing your legacy around with you, it’s good overall)
puck not responding
Any clue as to when bgp.he.net will be back?
Thanks, Hank
Re: Issue with Geolocation in Lasvegas
On 04/01/2024 9:13, Raja Sekhar Gullapalli via NANOG wrote: https://www.iucc.ac.il/en/blog/2021-05-google-geo-location/ :-) Regards, Hank Team, We are having issues in our lasvegas office & it shows geolocation in all browsers as Israel instead of US region when we access news.google.com in our PC. Our public ip is 129.46.96.20. Can you help to whom we can contact to resolve the issue. Regards, Raja
Re: Any comprehensive listing of where Google's IPs originate from?
On 04/12/2023 16:09, Drew Weaver wrote: Although not an answer to your specific question, when I need to reduce latency to a Google cloud region I use: https://gcping.com/ Regards, Hank Hello, We are trying to reduce latency to a region in Google Cloud which we are in the same city of. Latency is currently about 22ms rt for the traffic to go 9 miles. I am having the hardest time finding any comprehensive list of what exchanges, transit, etc their IP addresses are being announced over. Specifically trying to get closer to addresses in these prefixes: 34.162.192.0/18 34.162.64.0/18 Any info is greatly appreciated. Thanks, -Drew
BGP hijack?
We just had every single prefix in AS378 start being announced by AS2027. Every announcement by AS2027 is failing RPKI yet being propagated a bit. Is this yet another misbehaving device or an actual attack? Thanks, Hank
Re: Low to Mid Range DWDM Platforms
On 06/10/2023 16:07, Mike Hammett wrote: I have found that for low end DWDM solutions, https://www.packetlight.com/ has always been the cheapest available. Regards, Hank I've been using various forms of passive WDM for years. I have a couple different projects on my plate that require me to look at the next level of platform. In some projects, I'll be looking for needing to have someone long distances of glass without any electronics. Some spans could be over 60 miles. In some projects, I'll need to transport multiple 100-gig waves. What is the landscape like between basic passive and something like a 30 terabit Ciena? I know of multiple vendors in that space, but I like to learn more about what features I need and what features I don't need from somewhere other than the vendor's mouth. Obviously, the most reliability at the least cost as well. Thanks. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com
Re: Providing geofeed info to Google
Old topic: if one doesn't have access to https://isp.google.com how does one update their geo-location data so Google sees it? Thanks, Hank On Tue, Aug 30, 2022 at 12:34:41PM -0700, Hugo Slabbert wrote: Google folks: I see historical reference to needing to use the Google Peering Portal ( http://peering.google.com) if you need to provide Google with geofeed info for GeoIP info on network blocks, ref https://mailman.nanog.org/pipermail/nanog/2015-May/075229.html. Is that still the case? Are there any avenues to provide Google with geofeed info if you're *not* currently peering with 15169? Or to get access to just the geofeed update portion of the Peering Portal?
Re: v6 route mess frm AS266970
On 29/08/2023 18:41, Randy Bush wrote: is a massive route leak not even menntioned when it is only ipv6? the guess i heard was it looked like a classic config reorigination disaster. randy Has the route leak been resolved? BGPstream still shows it as active: https://bgpstream.crosswork.cisco.com/ RPKI only worked where it is implemented. I saw one path via Lumen (AS3356) and was disappointed to see it based on their blog from 2.5 years ago: https://blog.lumen.com/lumen-enhances-routing-security-with-resource-public-key-infrastructure-rpki/ "Once implemented, Lumen will use RPKI route validation on all BGP sessions for both customers and peers. Lumen’s RPKI validation servers download the ROAs, examine them, then send the tables to routers that can determine the validity of an IP prefix." MANRS confirms that AS3356 does not do much RPKI (see attachment). Regards, Hank
Re: Geolocastion and FF and Whatsapp
On 18/08/2023 16:36, J. Hellenthal wrote: Private (incognito) in FF gives English page. Interesting. Thanks, Hank Move your FF profile out of the way and reopen FF... diff results ? On Aug 18, 2023, at 00:45, Hank Nussbacher wrote: When I go to https://www.whatsapp.com/ I see the page in Latvian - but only on FF. When using Chrome I see the page in English. So who is doing bad geolocation? FF or Whatsapp? Thanks, Hank -- J. Hellenthal The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.
Geolocastion and FF and Whatsapp
When I go to https://www.whatsapp.com/ I see the page in Latvian - but only on FF. When using Chrome I see the page in English. So who is doing bad geolocation? FF or Whatsapp? Thanks, Hank
Re: Do ISP's collect and analyze traffic of users?
On 19/05/2023 15:27, Justin Streiner wrote: It amazes me how people can focus on Netflow metadata and ignore things like Microsoft telemetry data from every Windows box, or ignore the massive amount of html cookies that are traded by companies or how almost every corporate firewall or anti-spam box "reports" back to the mother ship and sends tons of information via secret channels like hashed DNS lookups just to be avoided. Regards, Hank There are already so many different ways that organizations can find out all sorts of information about individual users, as others have noted (social media interactions, mobile location/GPS data, call/text history, interactions with specific sites, etc), that there probably isn't much incentive for many providers to harvest data beyond what is needed for troubleshooting and capacity planning. Plus, gathering more data - potentially down to the level packet payload - is not an easy problem to solve (read: expensive) and doesn't scale well at all. 100G links are very common today, and 400G is becoming so. I doubt that many infrastructure providers would be able to justify the major investments in extra infrastructure to support this, for a revenue stream that likely wouldn't match that investment, which would make such an investment a loss-leader. Content providers - particularly social media platforms - have a somewhat different business model, but those providers already have many different ways to harvest and sell large troves of user data. Thank you jms
Re: Best Linux (or BSD) hosted BGP?
On 02/05/2023 17:56, Warren Kumari wrote: For those that like FRR: https://thehackernews.com/2023/05/researchers-uncover-new-bgp-flaws-in.html Regards, Hank +lots. I've used a number of Linux routing thingies (BIRD, Quagga, VyOS/Ubiquiti, OpenBGPd, ExBGP), and FRR is (for me at least) by far the friendliest. It's trivial to spin this up on a cloud VM and start announcing a prefix. For doing something like Anycast though (where you are mostly just announcing a route on demand), ExaBGP is great. W On Mon, May 01, 2023 at 2:03 PM, Jean Franco wrote: https://frrouting.org/ <https://frrouting.org/>
RPKI in Lacnic?
Just got this: Possible TA malfunction or incomplete VRP file: 100.00% of the ROAs disappeared from lacnic DETAILS: -- Event type: rpki-diff When event started: 2023-04-15 17:17:30 UTC Last event: 2023-04-15 17:17:30 UTC Only us or everyone? Thanks, Hank
Re: Starlink routing
On 23/01/2023 0:42, Michael Thomas wrote: I read in the Economist that the gen of starlink satellites will have the ability to route messages between each satellite. Would conventional routing protocols be up to such a challenge? Or would it have to be custom made for that problem? And since a lot of companies and countries are getting on that action, it seems like fertile ground for (bad) wheel reinvention? Mike For further reading try: https://www.internetsociety.org/wp-content/uploads/2022/11/Perspectives-on-LEO-Satellites.pdf -Hank
Re: Getting Fiber to My Town by Jared Mauch
On 10/09/2020 15:16, Jared Mauch wrote: Go Jared: https://www.dailystar.co.uk/tech/news/man-who-built-broadband-avoid-27717560 -Hank On Sep 10, 2020, at 8:06 AM, Jared Brown wrote: I believe this belongs here: Getting Fiber to My Town by Jared Mauch https://www.youtube.com/watch?v=ASXJgvy3mEg (YouTube video of NLnog presentation) https://nlnog.net/static/live/nlnog_live_sep_2020_jared.pdf (slides for presentation) https://news.ycombinator.com/item?id=24424910#24430901 (discussion on Hacker News with Jared participating) https://washftth.com/ (project homepage) I find this an interesting description of how to apply skills that we normally only use at work to solve connectivity issues at home. Quite timely too, as home connectivity is needed more than ever. As my kids are in the other room on 4x zooms at once, the prior connection could not have survived the load with them and me working as well. I’m working with the PC on giving another version of this talk (I want to provide more financial details) for an upcoming meeting. Stay tuned to the NANOG Agenda :-) - Jared
Re: Test email
On 20/06/2022 11:30, Peter Potvin wrote: I did not send this to the list. I assume the admins are testing out what has been blocking my emails for the past month and somehow this email slipped thru. Just ignore and delete. -Hank Why did moderation let this through the filters? I don't believe that testing email functionality is the intended use case of the NANOG mailing list. Also worth noting that the website for the domain this came from says the owner of the site specializes in "anti-spam", which based on this email alone doesn't look to be the case. Anyone else agree? Regards, Peter On Mon., Jun. 20, 2022, 4:15 a.m. , <mailto:h...@interall.co.il>> wrote: Hello, Checking Email Functionality. Hosting Support Thank you, The information contained in this message may be privileged, confidential and protected from disclosure. This message is intended only for the designated recipient(s). It is subject to access, review and disclosure by the sender's Email System Administrator. If you have received this message in error, please advise by return e-mail so that our address records can be corrected and please delete immediately without reading, copying or forwarding to others. Any unauthorized review, use, disclosure or distribution is prohibited. Copyright © 2022 Accuris Technologies Ltd. All Rights Reserved. L'information contenue dans ce message pourrait être de nature privilégiée, confidentielle et protégée contre toute divulgation. Ce message est destiné à l'usage exclusif du(des) destinataire(s) visé(s). Le gestionnaire de système du courrier électronique de l'expéditeur pourrait avoir accès à ce message, l'examiner et le divulguer. Si ce message vous est transmis par erreur, veuillez nous en aviser par courrier électronique à notre adresse, afin que l'on puisse corriger nos registres, puis veuillez le supprimer immédiatement, sans le lire, le copier ou le transmettre à des tiers. Tout examen, toute utilisation, divulgation ou distribution non autorisé de cette information est interdit. Droit d'auteur © 2022 Accuris Technologies Ltd. Tous droits réservés.
Test email
Hello, Checking Email Functionality. Hosting Support Thank you,
Re: Newbie x Cisco IOS-XR x ROV: BCP to not harassing peer(s)
On 14/05/2022 00:16, Jakob Heitz (jheitz) via NANOG wrote: 'RPKI-dropped-only' causes the dropped routes to be stored. This will prevent the unnecessary route-refreshes described above. It does not prevent all route-refreshes, but uses significantly less memory than 'RPKI-tested-only' Regards, Jakob. In the end, the reason for all this RPKI-thingy is to prevent route spoofing by malicious actors. It sure would be nice if someone from the top 20: https://asrank.caida.org/ would be able to have an auto-updated site that showed all RPKI dropped from their end. This would complement https://bgpstream.crosswork.cisco.com/ for those of us who want to know who is trying to hijack our routes at the core. Regards, Hank
Re: Sabotage: several severed cables at the origin of a major internet outage in France
On 27/04/2022 17:29, Nick Hilliard wrote: https://twitter.com/apb_laudrain/status/1519252859598032898 -Hank + pics: https://twitter.com/acontios_net/status/1519296590015606787 https://twitter.com/acontios_net/status/1519280710762348545 https://twitter.com/acontios_net/status/1519276453350805504 Nick Paul Ferguson wrote on 27/04/2022 15:17: On 4/27/22 7:08 AM, Sean Donelan wrote: Multiple physical cable cuts in multiple diverse locations in France. Several networks that connect the internet infrastructures of major French cities were cut overnight, in a short interval. A state source evokes with "the Obs" a "coordinated malicious act", which confirms SFR and Free affected. An investigation has been opened. https://www.nouvelobs.com/faits-divers/20220427.OBS57722/plusieurs-cables-sectionnes-a-l-origine-d-une-importante-panne-internet-en-france.html English language news article, fwiw: https://www.telegraph.co.uk/world-news/2022/04/27/internet-multiple-cities-across-france-suspected-sabotage/ Cheers, - ferg
Re: Any sign of supply chain returning to normal?
On 23/04/2022 01:28, na...@jima.us wrote: Ordered a pair of ASR9906s in Jan 2022 with delivery Aug 2022. -Hank Anecdotally, I had a pair of Nexus 93180s that I ordered in May 2021 show up in February 2022, so 9 months. The estimated ship date got punted several times (probably due to being preempted by folks employing the approach Laura outlined ;-) ). I haven't ordered anything since then, but I understand that 4-8 months isn't unexpected, still. - Jima From: NANOG On Behalf Of Drew Weaver Sent: Friday, April 22, 2022 07:24 To: 'nanog@nanog.org' Subject: Any sign of supply chain returning to normal? I'm not sure if this is the right place for this discussion but I can't think of anywhere better to ask. Has anyone seen any progress whatsoever on supply chain issues with networking hardware? I've noticed that primary market lead times have been increasing and at the same time secondary market pricing has also been going higher at the same time, still. What have you seen?
SATCOM terminals under attack in Europe
https://www.reversemode.com/2022/03/satcom-terminals-under-attack-in-europe.html -Hank
Re: Russia to disconnect from global Internet
Bill Woodcock wrote: > This applies exclusively to Russian federal government networks, not ISPs or telecom operators. https://twitter.com/krisnova/status/1500590779047170048?s=12 says otherwise. -Hank
Re: identity.cisco.com certificate has expired
On 05/03/2022 21:08, Matthew Huff wrote: Arghh… Just an FYI, id.cisco.com is fubar’ed. Hopefully cisco has already fixed it and the proxies/caches/cdns just need to timeout, but just in case anyone knows a contact at Cisco’s ops group… You mean in addition to all the other Cisco boxes that are fubered already? https://www.itnews.com.au/news/old-quovadis-certificate-issue-starting-to-break-cisco-products-576794 https://tools.cisco.com/security/center/resources/Q-CA-Root-Change -Hank *Matthew Huff*| Director of Technical Operations | OTA Management LLC /Office: 914-460-4039/ /mh...@ox.com <mailto:mh...@ox.com> | //www.ox.com <http://www.ox.com>/ *...*
Conflicts and fiber cuts
As the discussion rages on NANOG, RIPE, CENTR and many other uber-technical forums, I would like to see whether we can focus on what we know best - networking. Perhaps a weekly report of fiber cuts throughout Europe (starting from Feb 15) and the RFO that the carrier provided. Of especial interest would be undersea/underocean cuts or strange outages that the carrier cannot explain. Perhaps we can then map where some nation/state is sabotaging fiber or tapping into such fiber. Anyone willing to run with this? -Hank
Re: enom giving Google a bad name
On 16/01/2022 19:57, Rubens Kuhl wrote: On Sun, Jan 16, 2022 at 2:38 PM Hank Nussbacher wrote: Many of you might be following the enom weekend fiasco: https://twitter.com/enomsupport/status/1482621466151571456 https://twitter.com/enomsupport/status/1482707275529678849 https://enomstatus.com/ Thousands of domains have been knocked out. Because they used URL redirection services, right ? Domains under management are unlikely to be affected unless the registrant needs a domain update. From what I see enom's nameservers are down. -Hank
enom giving Google a bad name
Many of you might be following the enom weekend fiasco: https://twitter.com/enomsupport/status/1482621466151571456 https://twitter.com/enomsupport/status/1482707275529678849 https://enomstatus.com/ Thousands of domains have been knocked out. But I just found out that Google is an enom reseller: https://help.enom.com/hc/en-us/articles/115005222367-My-Domain-was-registered-through-G-Suite-Google-Apps "Google handles all of the billing and renewals of your domain name, while Enom offers technical support to manage your domain." So who takes responsibility when a fiasco happens like this: Google or Enom? -Hank
Re: Contact request AS 6453
On 15/01/2022 10:00, jim deleskie wrote: Did you try: https://www.peeringdb.com/net/437 peering-pol...@as6453.net peering-...@as6453.net ip...@tatacommunications.com Regards, Hank Have you found anyone. Not there any more but can probably still find someone for you. -jim On Thu, Jan 13, 2022, 10:11 AM Drew Weaver <mailto:drew.wea...@thenap.com>> wrote: Does anyone have a contact for AS 6453 or are there any AS 6453 folks on list? __ __ Seeing some routing trouble from their customers to the US. __ __ Thanks, -Drew __ __
Re: Cloudflare Abuse Contact
On 07/01/2022 21:35, Töma Gavrichenkov wrote: I would try n...@cloudflare.com based on: https://www.peeringdb.com/net/4224 Regards, Hank Peace, On Fri, Jan 7, 2022 at 8:42 PM Mike Hale wrote: The abuse email sends an auto-responder that tells you to use the web form. The web form is centered around their web hosting business; I figured I'd try general, but you can't submit it without punching in a URL that is hosted by Cloudflare (and they validate it ... you can't do https://bogus.site). What I'm seeing is a ton of abusive DNS traffic that's causing some issues, and there's no abuse form that works for this scenario. Most probably, that means that the company doesn't have any counter abuse process whatsoever for requests like yours, so no matter where you push that, there won't be any action. Having said that, the aforementioned form accepts "https[: slash slash]cloudflare.com" as a valid URL so chances are requests to that URL are treated in the general sense. In the meantime, are you sure you'll be able to support your case with data? DNS is *mostly* a connection-less protocol, so how do you know these queries are coming from Cloudflare and not from a spoofed source? Lastly, have you tried to block the problematic Cloudflare IP range to see what would happen? E.g. does 1.1.1.1 still resolve your domains then, etc.? -- Töma
Re: Anyone seeing ping corruption?
On 20/12/2021 04:41, J Doe wrote: Hi, Out of curiosity - does anyone know why Google is truncating ICMP responses ? As Google has stated in many forums and I quote: "Google Public DNS is a Domain Name System service, not an ICMP network testing service." -Hank
Re: Log4j mitigation
On 13/12/2021 15:28, bofh139 wrote: Now we have the long journey from Java, Ldap, DNS, Birds and Coffee Cups behind us. Does anyone else have any advice on prevention? Scan your systems: https://github.com/logpresso/CVE-2021-44228-Scanner https://github.com/fullhunt/log4j-scan -Hank
Re: Latency/Packet Loss on ASR1006
On 07/12/2021 17:32, Blake Hudson wrote: Suggestion: move this thread to cisco-nsp where you might find more assistance. Regards, Hank On 11/26/2021 1:09 PM, Colin Legendre wrote: Hi, We have ... ASR1006 that has following cards... 1 x ESP40 1 x SIP40 4 x SPA-1x10GE-L-V2 1 x 6TGE 1 x RP2 We've been having latency and packet loss during peak periods... We notice all is good until we reach 50% utilization on output of... 'show platform hardware qfp active datapath utilization summary' Literally ... 47% good... 48% good... 49% latency to next hop goes from 1ms to 15-20ms... 50% we see 1-2% packet-loss and 30-40ms latency... 53% we see 60-70ms latency and 8-10% packet loss. Is this expected... the ESP40 can only really push 20G and then starts to have performance issues? I haven't experienced that across about a dozen ASR 1ks. Though I just checked and we are not pushing any of our ESP's over 50% currently (the closest we have is an ESP 40 doing 18Gbps). However, I'm pretty sure we've pushed older ESPs (5, 10's, and 20's) to ~75% or so in the past. Given the components you have, I would have expected your router to handle 40Gbps input and 40Gbps output. That could either be 40Gbps into the 6 port card [and 40Gbps out of the four 1 port cards] or it could be 40Gbps input that is spread across the 6 port and 1 port cards [that is then output across both cards as well]. Despite other comments, I think your components are well matched. The only non-obvious thing here is that the 6 port card only has a ~40Gbps connection to the backplane so you cannot use all 6 ports at full bandwidth. I think this router is well suited to handle 20-30Gbps of customer demand doing standard destination based routing (if you're doing traffic shaping, NAT, tunnelling, or something else more involved than extended ACLs you may need something beefier at those traffic levels).
Re: .bv ccTLD
On 04/12/2021 00:45, Jay R. Ashworth wrote: My favorite youtuber has just pointed out that Bougainville will separate formally from Papua New Guinea in 2027, which, surprisingly, is only 5 or 6 years from now. So I looked up .bv, and of course... it's assigned to Bouvet Island, an uninhabited island whose political superior says anything that might go in that TLD will go in .no instead. [Wikipedia] So, what's the actual status of .bv? Assigned, or reserved? And if it is reserved at the 3166 secretariat level, can they reassign it? NORID might try to make a case that BV is the common corporate abbreviation in their political subdivision... but they're not selling those domains now, so that doesn't seem compelling. Anyone here got a buddy on the secretariat? :-) Cheers, -- jra All handling of ccTLDs are handled via the ccNSO of ICANN: https://ccnso.icann.org/en For example ccTLD retirement is a multi-year and perhaps multi-decade process: https://ccnso.icann.org/sites/default/files/field-attached/ccpdp3-retirement-vote-report-05aug21-en.pdf https://community.icann.org/pages/viewpage.action?pageId=64081623&preview=/64081623/166266006/Final%20Report%20ccPDP3%20Retirement%20-%20June%202021.pdf Regards, Hank
Re: Google location question
On 23/11/2021 22:31, Chuck Church wrote: Old issue. Everyone encounters this at some point: https://www.iucc.ac.il/en/blog/2021-05-google-geo-location/ You can try reporting it to Google: https://support.google.com/websearch/workflow/9308722?hl=en and wait a month or so to see if the issue gets resolved. Regards, Hank NANOG, Sorry if this is the wrong forum, but I figured we could all use a distraction from IPV4 expansion for a bit. We’re facing a problem with corporate use of browsers and google location services. Our users across North and South America seem to be getting location info for Rio de Janeiro, BR as of the last few days. All of our users when browsing hit general internet via a NAT pool for IPs out of Atlanta, GA. That hasn’t changed in over a year. The IP space itself and the upstream routers PTR haven’t changed for longer than that. For a normal laptop without a GPS chip and using default settings, it would seem that something else must be informing google of location. We’ve had a few theories. I discovered our Cisco wireless in RIO office is broadcasting SSID. The same SSID name we use in all offices. But I believe that google uses BSSID which is MAC based. And would be different between sites, as each have their own wireless controller, and certainly different APs. At least that’s what I believe to be true, that google didn’t associate our standard SSID name with a physical location. I’m curious if anyone has ever dealt with this before. I don’t want to blame our workstation folks for something that might be a geolocation issue, either based on wi-fi being detected, or a messed up geolocation for our IP space. Is there any place in Google-land where you can see exactly why it thinks you’re in a given location? I did see some articles on API usage to view values, but nothing that shows you the process when it takes all variables into effect. Thanks, Chuck
BGP hujack by AS25478?
Based on my own observation as well as via bgpstream there was a massive BGP hijack attempt last night by IHOME-AS iHome LLC, RU (AS 25478). Lasted about 10 minutes. Noction? Finger faddle? Malicious? Thanks, Hank
Re: Internet history
On 21/10/2021 21:52, Patrick W. Gilmore wrote: On Oct 21, 2021, at 2:37 PM, Michael Thomas wrote: [changed to a more appropriate subject] On 10/20/21 3:52 PM, Grant Taylor via NANOG wrote: On 10/20/21 3:26 PM, Michael Thomas wrote: Just as an interesting aside if you're interested in the history of networking, When Wizards Stayed Up Late is quite elucidating. +10 to Where Wizards Stay Up Late. I recently re-acquired (multiple copies of) it. (Multiple because I wanted the same edition that I couldn't locate after multiple moves.) One of the things about the book was that it finally confirmed for me what I had heard but thought might be apocryphal which was that one of my co-workers at Cisco (Charlie Klein) was the first one to receive a packet on ARPAnet. I guess it sent an "l" and then immediately crashed. They fixed the problem and the next time they got "login:". It also casts shade on another early well known person which gives me some amount of schadenfreude. It was “LO”, and Mr. Kline sent the packets, but you got it essentially right. Source: https://uclaconnectionlab.org/internet-museum/ The last picture confirms Mr. Kline sent the LO and crashed the WHOLE INTERENT (FSVO “Internet”) just a couple seconds after it started. Reminds me of the time the entire Swift network crashed when the capital of Ecuador (Quito) was added to the network. :-) -Hank
Geolocation accuracy
Can anyone recommend a geo-location service with high city accuracy? Maxmind, for most countries (broadband, which does move) is below 50% accuracy (they claim 68% accuracy for USA cities): https://www.maxmind.com/en/geoip2-city-accuracy-comparison?country=&resolution=city&cellular=excluding Thanks, Hank
Re: DNS pulling BGP routes?
On 06/10/2021 22:38, Jon Lewis wrote: But I just don't understand why this is a good idea at all. Network topology is not DNS's bailiwick so using it as a trigger to withdraw routes seems Everything I've seen posted about this (whether from Facebook directly, or others) is so vague as to what happened, that I think everyone's just making assumptions based on their own experiences or best guesses as to what really happened. Better question is why do we not see any FB netadmins on NANOG? I'm not talking about October 2021 but rather over the past 3-5 years how many FB techies have posted here like we see people from Google, Cloudflare, Akamai, etc.? -Hank
Re: Facebook post-mortems...
On 05/10/2021 21:11, Randy Monroe via NANOG wrote: Updated: https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ Lets try to breakdown this "engineering" blog posting: - "During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network" Can anyone guess as to what command FB issued that would cause them to withdraw all those prefixes? - "it was not possible to access our data centers through our normal means because their networks were down, and second, the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this. Our primary and out-of-band network access was down..." Does this mean that FB acknowledges that the loss of DNS broke their OOB access? -Hank
Re: Facebook post-mortems...
On 05/10/2021 13:17, Hauke Lampe wrote: On 05.10.21 07:22, Hank Nussbacher wrote: Thanks for the posting. How come they couldn't access their routers via their OOB access? My speculative guess would be that OOB access to a few outbound-facing routers per DC does not help much if a configuration error withdraws the infrastructure prefixes down to the rack level while dedicated OOB to each RSW would be prohibitive. https://research.fb.com/wp-content/uploads/2021/03/Running-BGP-in-Data-Centers-at-Scale_final.pdf Thanks for sharing that article. But OOB access involves exactly that - Out Of Band - meaning one doesn't depend on any infrastructure prefixes or DFZ announced prefixes. OOB access is usually via a local ADSL or wireless modem connected to the BFR. The article does not discuss OOB at all. Regards, Hank
Re: Facebook post-mortems...
On 05/10/2021 05:53, Patrick W. Gilmore wrote: Update about the October 4th outage https://engineering.fb.com/2021/10/04/networking-traffic/outage/ Thanks for the posting. How come they couldn't access their routers via their OOB access? -Hank
Re: massive facebook outage presently
On 04/10/2021 22:05, Jason Kuehl wrote: BGP related: https://twitter.com/SGgrc/status/1445116435731296256 as also related by FB CTO: https://twitter.com/atoonk/status/1445121351707070468 -Hank https://twitter.com/disclosetv/status/1445100931947892736?s=20 <https://twitter.com/disclosetv/status/1445100931947892736?s=20> On Mon, Oct 4, 2021 at 3:01 PM Tony Wicks <mailto:t...@wicks.co.nz>> wrote: Didn't write that part of the automation script and that coder left... > I got a mail that Facebook was leaving NLIX. Maybe someone botched the > script so they took down all BGP sessions instead of just NLIX and now > they can't access the equipment to put it back... :-) -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.ku...@gmail.com <mailto:jason.w.ku...@gmail.com>
FYI: NANOG and ICANN
https://www.icann.org/en/announcements/details/icann-signs-a-memorandum-of-understanding-with-nanog-27-9-2021-en Regards, Hank
Re: An update on the AfriNIC situation
On 27/08/2021 18:36, Bill Woodcock wrote: As many of you are aware, AfriNIC is under legal attack by Heng Lu / “Cloud Innovation.” John Curran just posted an excellent summary of the current state of affairs here: https://teamarin.net/2021/08/27/afrinic-and-the-stability-of-the-internet-number-registry-system/ If, like me, you feel like chipping in a little bit of money to help AfriNIC make payroll despite Heng having gotten their bank accounts frozen, some of the African ISP associations have put together a fund, which you can donate to here: https://www.tespok.co.ke/?page_id=14001 It’s an unfortunate situation, but the African Internet community has really pulled together to defend themselves, and they’ve got a lot less resources than most of us do. -Bill Why not just use the RIR Joint Stability Fund which was created for exactly these situations before asking for donations? Details here: https://www.nro.net/accountability/rir-accountability/joint-rir-stability-fund/ Regards, Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer
Re: netflow in the core used for surveillance
On 26/08/2021 00:13, Randy Bush wrote: https://www.vice.com/en/article/jg84yy/data-brokers-netflow-data-team-cymru used to get dissidents, activists, and journos killed at&t, comcast, ... zayo, please tell us you do not do this. randy I'm confused. Quoting from the article: "In a recent research report on an Israeli spyware vendor called Candiru, Citizen Lab thanked Team Cymru. Thanks to Team Cymru for providing access to their Pure Signal Recon product. Their tool’s ability to show Internet traffic telemetry from the past three months provided the breakthrough we needed to identify the initial victim from Candiru’s infrastructure," the report reads. Citizen Lab did not respond to multiple requests for comment." So Team Cymru helped expose themselves as to getting dissidents, activists and journalists killed? -Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer
Re: Setting sensible max-prefix limits
On 18/08/2021 13:03, Chriztoffer Hansen wrote: On Wed, 18 Aug 2021 at 11:33, Lars Prehn wrote: I guess for long standing peers one could just eyeball it, e.g., current prefix count + some safety margin. How does that work for new peers? If you have automation in place. Another approach is to count the received prefix. Store the counted value in a database. Based on the avg prefix count over X (time period). Add e.g. 10 - 25 % headroom over the avg prefix count and use the calculated value as the max-prefix limit? That works but all too often people forget to update it. Set a quarterly reminder in your calendar to check max-prefix setting. -Hank
Re: "Tactical" /24 announcements
On 12/08/2021 17:59, William Herrin wrote: If you prune the routes from the Routing Information Base instead, for any widely accepted size (i.e. /24 or shorter netmask) you break the Internet. How does this break the Internet? I would think it would just result in sub-optimal routing (provided there is a covering larger prefix) but everything should continue to work. Clue me in, please. -Hank
Re: "Tactical" /24 announcements
On 09/08/2021 18:47, Billy Croan wrote: How does the community feel about using /24 originations in BGP as a tactical advantage against potential bgp hijackers? All of our allocations are larger and those prefixes we announce for clients as well usually are. But we had a request recently to originate everything as distinct /24 prefixes, to reduce the effect of a potential bgp hijack. It seemed a little bit like a tragedy of the commons situation. Is this seen as route table pollution, or a necessary evil in today's world? How many routers out there today would be affected if everyone did this? Are there any big networks that drop or penalize announcements like this? In addition to what everyone else said, announcing /24s will not help you one bit since ASNs announce /25s, /26s, /27s, etc. Attached is a 7800+ line text file sorted by ASN with prefixes being announced that are more specific than /24 (only /25+/26+/27 listed). This is based on http://www.ris.ripe.net/dumps/riswhoisdump.IPv4.gz from about a month ago. That dump lists all the IPv4 prefixes seen in the collective of latest RIS table dumps, together with origin AS and number of peers that passed the routes to RIS. So good luck with announcing /24s. Regards, Hank more-specifics-sorted-by-asn.7z Description: Binary data
Re: Where to get IPv4 block these day
Been a while since I had to deal with NetOps stuff. Was wondering, where do you go these days to get IPv4 blocks? It seems like getting assignments is hard due to exhaustion. I have found some "Auction" sites but it all feels very scammy. Any info would be appreciated. -James https://www.ripe.net/manage-ips-and-asns/resource-transfers-and-mergers/brokers https://www.ripe.net/manage-ips-and-asns/resource-transfers-and-mergers/listing https://www.ripe.net/manage-ips-and-asns/resource-transfers-and-mergers/transfer-statistics/within-ripe-ncc-service-region/ipv4-transfer-statistics -Hank
Re: Global Akamai Outage
On 25/07/2021 09:18, Saku Ytti wrote: Hey, Not a critique against Akamai specifically, it applies just the same to me. Everything seems so complex and fragile. Complex systems are apt to break and only a very limited set of tier-3 engineers will understand what needs to be done to fix it. KISS -Hank
Re: Global Akamai Outage
On 23/07/2021 09:24, Hank Nussbacher wrote: From Akamai. How companies and vendors should report outages: [07:35 UTC on July 24, 2021] Update: Root Cause: This configuration directive was sent as part of preparation for independent load balancing control of a forthcoming product. Updates to the configuration directive for this load balancing component have routinely been made on approximately a weekly basis. (Further changes to this configuration channel have been blocked until additional safety measures have been implemented, as noted in Corrective and Preventive Actions.) The load balancing configuration directive included a formatting error. As a safety measure, the load balancing component disregarded the improper configuration and fell back to a minimal configuration. In this minimal state, based on a VIP-only configuration, it did not support load balancing for Enhanced TLS slots greater than 6145. The missing load balancing data meant that the Akamai authoritative DNS system for the akamaiedge.net zone would not receive any directive for how to respond to DNS queries for many Enhanced TLS slots. The authoritative DNS system will respond with a SERVFAIL when there is no directive, as during localized failures resolvers will retry an alternate authority. The zoning process used for deploying configuration changes to the network includes an alert check for potential issues caused by the configuration changes. The zoning process did result in alerts during the deployment. However, due to how the particular safety check was configured, the alerts for this load balancing component did not prevent the configuration from continuing to propagate, and did not result in escalation to engineering SMEs. The input safety check on the load balancing component also did not automatically roll back the change upon detecting the error. Contributing Factors: The internal alerting which was specific to the load balancing component did not result in blocking the configuration from propagating to the network, and did not result in an escalation to the SMEs for the component. The alert and associated procedure indicating widespread SERVFAILs potentially due to issues with mapping systems did not lead to an appropriately urgent and timely response. The internal alerting which fired and was escalated to SMEs was for a separate component which uses the load balancing data. This internal alerting initially fired for the Edge DNS system rather than the mapping system, which delayed troubleshooting potential issues with the mapping system and the load balancing component which had the configuration change. Subsequent internal alerts more clearly indicated an issue with the mapping system. The impact to the Enhanced TLS service affected Akamai staff access to internal tools and websites, which delayed escalation of alerts, troubleshooting, and especially initiation of the incident process. Short Term Completed: Akamai completed rolling back the configuration change at 16:44 UTC on July 22, 2021. Blocked any further changes to the involved configuration channel. Other related channels are being reviewed and may be subject to a similar block as reviews take place. Channels will be unblocked after additional safety measures are assessed and implemented where needed. In Progress: Validate and strengthen the safety checks for the configuration deployment zoning process Increase the sensitivity and priority of alerting for high rates of SERVFAILs. Long Term In Progress: Reviewing and improving input safety checks for mapping components. Auditing critical systems to identify gaps in monitoring and alerting, then closing unacceptable gaps. On 22/07/2021 19:34, Mark Tinka wrote: https://edgedns.status.akamai.com/ Mark. [18:30 UTC on July 22, 2021] Update: Akamai experienced a disruption with our DNS service on July 22, 2021. The disruption began at 15:45 UTC and lasted for approximately one hour. Affected customer sites were significantly impacted for connections that were not established before the incident began. Our teams identified that a change made in a mapping component was causing the issue, and in order to mitigate it we rolled the change back at approximately 16:44 UTC. We can confirm this was not a cyberattack against Akamai's platform. Immediately following the rollback, the platform stabilized and DNS services resumed normal operations. At this time the incident is resolved, and we are monitoring to ensure that traffic remains stable.
Re: Global Akamai Outage
On 22/07/2021 19:34, Mark Tinka wrote: https://edgedns.status.akamai.com/ Mark. [18:30 UTC on July 22, 2021] Update: Akamai experienced a disruption with our DNS service on July 22, 2021. The disruption began at 15:45 UTC and lasted for approximately one hour. Affected customer sites were significantly impacted for connections that were not established before the incident began. Our teams identified that a change made in a mapping component was causing the issue, and in order to mitigate it we rolled the change back at approximately 16:44 UTC. We can confirm this was not a cyberattack against Akamai's platform. Immediately following the rollback, the platform stabilized and DNS services resumed normal operations. At this time the incident is resolved, and we are monitoring to ensure that traffic remains stable.
Re: Google Geo Location Issues
On 30/06/2021 17:43, Jason Kuehl wrote: Once you have access to isp.google.com your problems are far from over. You would assume that they would use whois info to know which prefix belongs to your ASN. That would be wrong. If you have, for example, a multi-homed customer and you provided them with a smaller prefix (/27) from within your larger prefix (/19), and since they are multihomed some other ASN announces that smaller prefix (/27), Google will see that and invalidate any geo-location info you provide for your /19. Google follows the BGP routing table as authoritative for ownership of prefixes. Regards, Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer This is want I'm working on setting up, here is hoping they approve my account. On Wed, Jun 30, 2021 at 10:30 AM Benjamin Hatton <mailto:bhat...@htva.net>> wrote: If they are your subnets, and you have your own AS (and possibly enough traffic to Google, I'm not sure what exactly their criteria is but we were able to make an account with ~2Gbps peak and no peering relationships) you can create an account with the Google ISP Portal (isp.google.com <http://isp.google.com>) which has a self service tool to upload a csv that will update your prefixes in the Google IP Geolocation database. Took ~2 weeks after upload for it to be reflected in live data. *Ben Hatton* *Network Engineer | Haefele Connect* d:(607)589-8000 | b...@haefeleconnect.com <mailto:b...@haefeleconnect.com> *<https://www.htva.net>* On Wed, Jun 30, 2021 at 10:21 AM Nate Burke mailto:n...@blastcomm.com>> wrote: Same here, YoutubeTV Geolocation problem, has one of my subnets in Tulsa instead of Chicago. Ticket open for 8 months. Every reply was that it was 'with engineering' and no ETA. I just got a notice from them last week that they're closing the ticket and sent a survey to fill out to rate the support received. I told them that the issue was still not resolved and never heard back from them. On 6/30/2021 8:55 AM, Josh Luthman wrote: Been months since I was told they'd get it fixed. To be fair they did say they weren't sure on how long it would take. I feel like I've been forgotten about. Josh Luthman 24/7 Help Desk: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Jun 30, 2021 at 9:18 AM Mike Hammett mailto:na...@ics-il.net>> wrote: I've discovered that if you *CAN* get a Google ISP account, you can manage it all there. If you can't, well, you're up shit creek without a paddle. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com <http://www.ics-il.com> Midwest-IX http://www.midwest-ix.com <http://www.midwest-ix.com> *From: *"Jason Kuehl" mailto:jason.w.ku...@gmail.com>> *To: *"NANOG" mailto:nanog@nanog.org>> *Sent: *Tuesday, June 29, 2021 6:25:06 PM *Subject: *Google Geo Location Issues I'm looking for a contact, email, number, smoke signals for someone at Google I can talk to on geolocation issue. For some reason Google has labeled our IP ranges as Belarus when we're located in the states. If anyone can point me at any contact I would be really happy.. . -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.ku...@gmail.com <mailto:jason.w.ku...@gmail.com> -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.ku...@gmail.com <mailto:jason.w.ku...@gmail.com>
Re: Google Geo Location Issues
On 30/06/2021 02:25, Jason Kuehl wrote: Good luck. I had a case where Israeli users were located in Iceland. Took me well over a month to get it resolved. You can read about it in my blog entry: https://www.iucc.ac.il/en/blog/2021-05-google-geo-location/ Just heard of two more incidents this week where other Israeli users from different ISPs are seen as being in Belgium and the other is seen as being in Greece. I have told those 2 new incidents to try: https://support.google.com/websearch/workflow/9308722?hl=en and wait a month to see if the issue gets resolved (as stated on that Google page). Clearly some process since about March 2021 is broken inside Google geo-location land. Regards, Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer I'm looking for a contact, email, number, smoke signals for someone at Google I can talk to on geolocation issue. For some reason Google has labeled our IP ranges as Belarus when we're located in the states. If anyone can point me at any contact I would be really happy.. . -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.ku...@gmail.com <mailto:jason.w.ku...@gmail.com>
Re: shadowserver.org
What is the difference between shodan.io and shadowserver.org ? Jean Just those 2? Greynoise maps them all. See an old preso from 2018: https://www.slideshare.net/andrewwantsyou/identifying-and-correlating-internetwide-scan-traffic-to-newsworthy-security-events See slide 7 for a 4 year old list which has only grown :-) -Hank
Re: shadowserver.org
On 28/06/2021 06:19, Scott Aldrich wrote: Anyone have an idea how to get HE/ShadowServer,org servers to stop attempting to penetrate the comcast drop at my house? Their website claims altruism.. but my logs dont support that claim. Scott Scott, Did you look at: https://www.shadowserver.org/what-we-do/network-reporting/dns-open-resolvers-report/ https://scan.shadowserver.org/dns/ If you still think they are penetrating you see their section of blocklisting: To be removed from this set of scanning you will need to send an email to dnsscan [at] shadowserver [dot] org with the specific CIDR's that you would like to have removed. You will have to be the verifiable owner of these CIDR's and be able to prove that fact. Any address space that is blocklisted will be publicly available here: https://scan.shadowserver.org/dns/exclude.html Regards, Hank
Re: ROA coverage info
On 24/08/2020 17:49, Rayhaan Jaufeerally (NANOG) wrote: There's also this site run by NIST: https://rpki-monitor.antd.nist.gov/ <https://rpki-monitor.antd.nist.gov/> which contains further breakdowns Anyone know why https://rpki-monitor.antd.nist.gov/ is down? Thanks, Hank
Re: Google uploading your plain text passwords
On 12/06/2021 08:31, Damian Menscher via NANOG wrote: The Chrome password manager is convenient, and the sync can be incredibly handy (I can sign into stuff on different computers or even my phone without needing to copy over the passwords), but you might consider leaving your highest-value passwords out of that system, or really any system. Personally, my financial passwords are not known by Chrome, myself, or even my password manager. (Yes, you heard that right -- no single entity knows the passwords. How? By using a simple secret-splitting scheme -- I memorize part of the password, and my password manager stores the rest.) Or: https://doubleoctopus.com/ -Hank Damian
Re: Google IP Geolocation
On 22/04/2021 11:36, Hank Nussbacher wrote: Jared wrote earlier: I've had a similar issue in the past trying to get ready to peer with them. I wanted portal access to look at things. I may yet post a geofeed file just because. (I was also rejected a portal account, didn't escalate to friends at google because I know my traffic is under a gig for now). - Jared Don't bother. We have Google ISP Portal access. We updated our geo-location feed file as they requested: http://noc.ilan.net.il/GGC/iucc-geo-feed-for-google.csv 15 of 16 prefixes were processed on April 30. Till today all our users in Israel are still located in Iceland and have now started to learn Icelandic in order to complete Google pages. Regards, Hank The issues that others had earlier this month just hit us this morning. Users in Israel (132.74.0.0/15) trying to access Google.com or Youtube.com appear as coming from Iceland (see screenshot). Change happened overnight. Someone internally in Google's geo-location group typo'ed Israel as Iceland. We have GGC access and Google ISP Portal access but why should we have to change the geolocation which worked well since forever just because someone internally messed up? Regards, Hank
Re: Google IP Geolocation
On 22/04/2021 11:36, Hank Nussbacher wrote: The issues that others had earlier this month just hit us this morning. Users in Israel (132.74.0.0/15) trying to access Google.com or Youtube.com appear as coming from Iceland (see screenshot). Change happened overnight. Someone internally in Google's geo-location group typo'ed Israel as Iceland. We have GGC access and Google ISP Portal access but why should we have to change the geolocation which worked well since forever just because someone internally messed up? Regards, Hank Now I am hearing that other universities in Israel have seen this issue since as early as April 4. -Hank
Re: Google IP Geolocation
The issues that others had earlier this month just hit us this morning. Users in Israel (132.74.0.0/15) trying to access Google.com or Youtube.com appear as coming from Iceland (see screenshot). Change happened overnight. Someone internally in Google's geo-location group typo'ed Israel as Iceland. We have GGC access and Google ISP Portal access but why should we have to change the geolocation which worked well since forever just because someone internally messed up? Regards, Hank
Re: BGP and The zero window edge
On 22/04/2021 02:24, Job Snijders via NANOG wrote: On Wed, Apr 21, 2021 at 09:22:57PM +, Jakob Heitz (jheitz) wrote: I'd like to get some data on what actually happened in the real cases and analyze it. [snip] TCP zero window is possible, but many other things could cause it too. Indeed. There could be a number of reasons that caused it. Switchings away from TCP win=0 towards "Zombie Routes": *RIGHT NOW* (at the moment of writing), there are a number of zombie route visible in the IPv6 Default-Free Zone: One example is http://lg.ring.nlnog.net/prefix_detail/lg01/ipv6?q=2a0b:6b86:d15::/48 2a0b:6b86:d15::/48 via: BGP.as_path: 204092 57199 35280 6939 42615 42615 212232 BGP.as_path: 208627 207910 57199 35280 6939 42615 42615 212232 BGP.as_path: 208627 207910 57199 35280 6939 42615 42615 212232 (first announced April 15th, last withdrawn April 15th, 2021) Another one is http://lg.ring.nlnog.net/prefix_detail/lg01/ipv6?q=2a0b:6b86:d24::/48 2a0b:6b86:d24::/48 via: BGP.as_path: 201701 9002 6939 42615 212232 BGP.as_path: 34927 9002 6939 42615 212232 BGP.as_path: 207960 34927 9002 6939 42615 212232 BGP.as_path: 44103 50673 9002 6939 42615 212232 BGP.as_path: 208627 207910 34927 9002 6939 42615 212232 BGP.as_path: 3280 34927 9002 6939 42615 212232 BGP.as_path: 206628 34927 9002 6939 42615 212232 BGP.as_path: 208627 207910 34927 9002 6939 42615 212232 (first announced March 24th, last withdrawn March 24th, 2021) Just now, I literally rebooted the BGP speaker behind lg.ring.nlnog.net to make ensure that those routes are not stuck in the BGP looking glass itself. 2a0b:6b86:d24::/48 was first announced on March 24th, 2021, and withdrawn at the end of March 24th, 2021 by the originator, and now almost a month later, this prefix still is visible in the default-free zone despite WITHDRAW messages having been sent and the AS 212232 operator confirming they are not announcing that IP prefix anywhere. I checked the AS 6939 Looking glass, but the d24::/48 route is not visible in the http://lg.he.net/ web interface. This leads me to believe the the route got stuck somewhere along way in either of 201701, 204092, 206628, 207910, 207960, 208627, 3280, 34927, 35280, 44103, 50673, 57199, and/or 9002. This implies indeed might be multiple reasons a BGP route gets stuck ('stuck' as in - a WITHDRAW was not generated, or ignored). Perhaps on any one of these edges there is a very high Out Queue for one reason or another: 34927 9002 206628 34927 44103 50673 207960 34927 3280 34927 9002 6939 201701 9002 208627 207910 I'm not sure all the these sightings of stuck routes can be pinpointed to one specific BGP vendor (or one bug). I would guess that all the stuck route sightings manifest from one undiscovered TCP library bug that some BGP vendors are all commonly using. -Hank Kind regards, Job
Re: Perhaps it's time to think about enhancements to the NANOG list...?
On 20/03/2021 21:34, Stan Barber wrote: +1 -Hank +1 from the peanut gallery On Sat, Mar 20, 2021 at 2:30 PM Allen Kitchen mailto:allenmckinleykitc...@gmail.com>> wrote: On Sat, Mar 20, 2021 at 2:07 PM Randy Bush mailto:ra...@psg.com>> wrote: i do not find the volume or diversity on the nanog list problematic. in fact, i suspect its diversity and openness are major factors in it being the de facto global anything-ops list. perhaps we do not need to fix that. randy +1 (or as much more as I can be credited for) ..Allen
RPKI invalid logs?
Is there a place where one can examine RPKI invalid logs for a specific date & time or even better logs showing those that dropped RPKI invalid announcements? Thanks, Hank
Re: bgp.he.net?
On 18/02/2021 15:08, Hank Nussbacher wrote: Is it down? -Hank Back up. -Hank
bgp.he.net?
Is it down? -Hank
Re: Problems with newish IP block assignment issues from ARIN
On 08/02/2021 22:14, Justin Wilson (Lists) wrote: It acts like the IP block was blacklisted at some point and got on some bad lists but I don’t want ti limit myself to that theory. I have opened up a ticket with ARIN asking for any guidance. Has anyone ran into this with new space assigned? Any tools, sites, etc. I can use to do further troubleshooting. The IP block does not appear to have any blacklisted IPs according to MX toolbox, and some others. Try: http://multirbl.valli.org/lookup/ -Hank The block in question is 134.195.44.0/22. It has been RPKI certified and has IRR entries. Thanks in advance Justin Wilson j...@mtin.net — https://j2sw.com - All things jsw (AS209109) https://blog.j2sw.com - Podcast and Blog
Re: RTBH and Flowspec Measurements - Stop guessing when the attack will over
On 02/02/2021 19:08, Douglas Fischer wrote: Well... That is a point of view! And I must respect that. Against this position, there are several companies, including some tier 1, that sells this as an $extra$. About the "Please break me at my earliest inconvenience." part: I believe that the same type of prefix filtering that applies to Downstream-BGP-Routes applies to RTBH and Flowspec. So, exactly as in common BGP Route-Filtering: - If the network operator does it correctly, it should work correctly. - If the network operator deals with that without the needed skills, expertise, attention+devotion, wrong things will come up. You forgot to mention software bugs: https://kb.juniper.net/InfoCenter/index?page=content&id=JSA11101&cat=SIRT_1&actp=LIST Note what Juniper states: Workaround: There are no viable workarounds for this issue -Hank But, this still does not helps to find a solution do an organization A that sends some flowspec our RTBH to organization B(presuming organization B will accept that), and organization B do some reports of what is match with that flowspec or RTBH. That, in my opinion, is the only way to stop guessing how long will an attack will last, and start to define the end of a flowspec/RTBH action based on real information related to that. I want to close the feedback loop. Em ter., 2 de fev. de 2021 às 13:07, Tom Beecher escreveu: Personally, I would absolutely, positively, never ever under any circumstances provide access to a 3rd party company to push a FlowSpec rule or trigger RTBH on my networks. No way. You would be handing over a nuclear trigger and saying "Please break me at my earliest inconvenience." On Tue, Feb 2, 2021 at 5:56 AM Douglas Fischer <fischerdoug...@gmail.com> wrote: OK, but do you know any company the sells de Flowspec as a service, in the way that the Attack Identifications are not made by their equipment, just receiving de BGP-FlowSpec and applying that rules on that equipments... And even then give back to the customer some way to access those statistics? I just know one or two that do that, and(sadly) they do it on fancy web reports or PDFs. Without any chance of using that as structured data do feedback the anomaly detection tools to determine if already it is the time to remove that Flowsperc rule. What I'm looking for is something like: A) XML/JSON/CSV files streamed to my equipment from the Flowspec Upstream Equipments saying "Heepend that, that, and that." Almost in real time. B) NetFlow/IPFIX/SFlow streamed to my equipment from the Upstream Equipment, restricted to the DST-Address that matches to the IP blocks that were involved to the Flowspec or RTBH that I Annouced to then. C) Any other idea that does the job of gives me the visibility of what is happening with FlowSpec-rules, or RTBH on theyr network. Em seg., 1 de fev. de 2021 às 22:07, Dobbins, Roland <roland.dobb...@netscout.com> escreveu: On Feb 2, 2021, at 00:34, Douglas Fischer <fischerdoug...@gmail.com> wrote: Or even know if already there is a solution
Re: Parler
Toma, I would assume Google and Azure would act the same to Parler. So what will end up happening is that US based fringe content will end up being hosted in China or Russia, and Chinese and Russian fringe content will end up being hosted in the USA. -Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer -- In my experience, moving off Amazon services isn't that much of a trouble, especially if compared to moving off Azure. Cloud Active Directory + Sentinel + PowerBI ? and boom, your company is with Azure for life. -- Töma
Re: Centurylink having a bad morning?
On 30/08/2020 20:08, Baldur Norddahl wrote: https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/ Sounds like Flowspec possibly blocking tcp/179 might be the cause. But that is Cloudflare speculation. Regards, Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer An outage is what it is. I am not worried about outages. We have multiple transits to deal with that. It is the keep announcing prefixes after withdrawal from peers and customers that is the huge problem here. That is killing all the effort and money I put into having redundancy. It is sabotage of my network after I cut the ties. I do not want to be a customer at an outlet who has a system that will do that. Luckily we do not currently have a contract and now they will have to convince me it is safe for me to make a contract with them. If that is impossible I guess I won't be getting a contract with them. But I disagree in that it would be impossible. They need to make a good report telling exactly what went wrong and how they changed the design, so something like this can not happen again. The basic design of BGP is such that this should not happen easily if at all. They did something unwise. Did they make a route reflector based on a database or something? Regards, Baldur On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho <mikeboli...@gmail.com> wrote: Exactly. And asking that they somehow prove this won't happen again is impossible. - Mike Bolitho On Sun, Aug 30, 2020, 8:10 AM Drew Weaver <drew.wea...@thenap.com> wrote: I’m not defending them but I am sure it isn’t intentional. From: NANOG thenap@nanog.org> On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org Subject: Re: Centurylink having a bad morning? How is that acceptable behaviour? I shall remember never to make a contract with these guys until they can prove that they won't advertise my prefixes after I pull them. Under any circumstances. søn. 30. aug. 2020 15.14 skrev Joseph Jenkins <j...@breathe-underwater.com>: Finally got through on their support line and spoke to level1. The only thing the tech could say was it was an issue with BGP route reflectors and it started about 3am(pacific). They were still trying to isolate the issue. I've tried failing over my circuits and no go, the traffic just dies as L3 won't stop advertising my routes. On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG <nanog@nanog.org> wrote: Hello, Woke up this morning to a bunch of reports of issues with connectivity had to shut down some Level3/CTL connections to get it to return to normal. As of right now their support portal won’t load:
Re: Centurylink having a bad morning? [EXTERNAL]
On 30/08/2020 18:22, Joseph Jenkins wrote: Well at least it looks like the issue is starting to resolve and stuff is coming back up. On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes <mattli...@rivervalleyinternet.net> wrote: Is this what happens when your entire network is database driven? See: https://status.ctl.io/ and specifically: https://status.ctl.io/history/f19a0555-abbd-4038-91cb-b55a7645c1f5 Regards, Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer
Bottlenecks and link upgrades
At what point do commercial ISPs upgrade links in their backbone as well as peering and transit links that are congested? At 80% capacity? 90%? 95%? Thanks, Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer
ISPs are hit hardest by COVID-19 disruption
https://betanews.com/2020/08/04/isps-covid-19-disruption/ Really? -Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer
Re: BGP route hijack by AS10990
On 01/08/2020 00:50, Mark Tinka wrote: On 31/Jul/20 23:38, Sabri Berisha wrote: Kudos to Telia for admitting their mistakes, and fixing their processes. Considering Telia's scope and "experience", that is one thing. But for the general good of the Internet, the number of intended or unintentional route hijacks in recent years, and all the noise that rises on this and other lists each time we have such incidents (this won't be the last), Telia should not have waited to be called out in order to get this fixed. Do we know if they are fixing this on just this customer of theirs, or all their customers? I know this has been their filtering policy with us (SEACOM) since 2014, as I pointed out earlier today. There has not been a shortage of similar incidents between now and then, where the community has consistently called for more deliberate and effective route filtering across inter-AS arrangements. AS level filtering is easy. IP prefix level filtering is hard. Especially when you are in the top 200: https://asrank.caida.org/ That being said, and due to these BGP "polluters" constantly doing the same thing, wouldn't an easy fix be to use the max-prefix/prefix-limit option: https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/25160-bgp-maximum-prefix.html https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/prefix-limit-edit-protocols-bgp.html For every BGP peer, the ISP determines what the current max-prefix currently is. Then add in 2% and set the max-prefix. An errant BGP polluter would then only have limited damage to the Internet routing table. Not the greatest solution, but easy to implement via a one line change on every BGP peer. Smaller ISPs can easily do it on their 10 BGP peers so as to limit damage as to what they will hear from their neighbors. -Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer
Re: BGP route hijack by AS10990
On 30/07/2020 20:32, Sadiq Saif wrote: On Thu, 30 Jul 2020, at 13:09, Patrick Schultz wrote: so, bgp optimizers... again? -- Patrick More like shame on Telia for not filtering properly. But wait - MANRS indicates that Telia does everything right: https://www.manrs.org/isps/participants/?gv_search=telia&mode=any How can that be? -Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer
Re: BGP route hijack by AS10990
On 30/07/2020 05:46, Clinton Work wrote: See: https://bgpstream.com/event/245264 https://bgpstream.com/event/245265 -Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer We saw a bunch of our IP blocks hijacked by AS10990 from 19:15 MDT until 20:23 MDT. Anybody else have problems with that. ASpath: 1299 7219 10990 50.92.0.0/17 AS10990 198.166.0.0/17 AS10990 198.166.128.0/17 AS10990 162.157.128.0/17 AS10990 162.157.0.0/17 AS10990 50.92.128.0/17 AS10990 -- Clinton Work Airdrie, AB
Re: Survey on the use of IP blacklists for threat mitigation
On 16/06/2020 22:08, J. Hellenthal via NANOG wrote: This issue was raised in Reddit and Github: https://www.reddit.com/r/sysadmin/comments/h149em/calls_to_replace_blacklist_whitelist_black_hat/ https://www.techspot.com/news/85631-github-replace-terms-whitelist-blacklist-masterslave-racially-insensitive.html and is not limited to the term "blacklist". -Hank Note: the views expressed above are my own and do not necessarily reflect the views of my employer Guess we all better start rewriting all of the documentation out there because some PC marketing snowflake wants to get extra brownie points and attention for classifying a color in RGB into a racial divide for which it never originated. blacklists are not always deny/block/disallow and conformed of things that allow you to take actions whatever your choosing upon their contents and your policies. What’s next ? redlisting ? Don’t offend the Russians ... blue ? Don’t want to offend the police ... Leave this crap off the list, it’s not helping anyone. SMH -- J. Hellenthal The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume. On Jun 16, 2020, at 13:27, Ryan Landry wrote: In kind, I'd like to encourage the use of terms like permit/accept list or deny/block list. Respectfully, -Ryan On Tue, Jun 16, 2020 at 11:33 AM Rachee Singh <rachee.si...@gmail.com> wrote: Hi NANOG community, We are a group of researchers studying the use of IP blacklists as a mechanism to mitigate security threats -- particularly over the IPv6 Internet. We would like to understand if and how you use IP blacklists to secure your networks. Please consider taking our short survey: https://forms.gle/ZEsxyiBivJAfLF7e6 The survey will be anonymous unless you choose to identify yourself. Thanks, Rachee UMass Amherst
IBM Cloud global outage caused by "incorrect" BGP routing
https://www.bleepingcomputer.com/news/technology/ibm-cloud-global-outage-caused-by-incorrect-bgp-routing/ -Hank Note: the views expressed above are my own and do not necessarily reflect the views of my employer
Spike in traffic to Google&Akamai caches?
Did anyone notice a huge jump in traffic today between 11:30-11:40 (GMT) directed at Google and Akamai caches coming from Amazon and Google? Gaming updates? Thanks, Hank Caveat: The views expressed above are solely my own and do not express the views or opinions of my employer
Re: Backhoe season?
On 26/03/2020 20:02, Aaron Gould wrote: Numerous gov'ts and municipalities, which had planned constructions jobs but postponed them to the summer due to heavy traffic volume, have started to implement all those construction jobs, which includes backhoes. -Hank I heard, and am seeing that construction type jobs don't seem to be affected much with the virus shutdown. I mean I see guys building homes and working on roads all around me... furthermore, we've heard of a couple fiber cuts that have brought portions of our network down a couple times in the last week or so. -Aaron -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William Herrin Sent: Thursday, March 26, 2020 12:57 PM To: nanog@nanog.org Subject: Backhoe season? Howdy, With so much work shut down, I'm curious how backhoe season is shaping up this year? How do the circuit and fiber outage numbers look? Regards, Bill Herrin
Re: Gmail email blocking is off the rails (again)
On 04/12/2019 05:04, Matthew Pounsett wrote: Cute way to promote Google Groups over Mailman. Gotta give 'em credit for being creative :-) -Hank For some reason Gmail has started blocking mailman administrative emails to someone who's an admin on a list I host. Their SMTP 552 error message points to <https://support.google.com/mail/?p=BlockedMessage>, which implies the "problem" is the URLs in the email, but is otherwise completely unhelpful. If anyone here has any pull with Gmail postmasters, could you please suggest to them that they whitelist messages that are as consistent and well-known as mailman's admin and moderator messages?
Equinix Hong Kong
Hello, Anybody from Equinix willing to help me out with an issue in Hong Kong? Please contact me off-list. Thanks. -Hank Disuko
Re: "Using Cloud Resources to Dramatically Improve Internet Routing"
On 07/10/2019 17:42, Stephane Bortzmeyer wrote: On Fri, Oct 04, 2019 at 03:52:26PM -0400, Phil Pishioneri wrote a message of 9 lines which said: Using Cloud Resources to Dramatically Improve Internet Routing UMass Amherst researchers to use cloud-based ‘logically centralized control’ Executive summary: it's SDN for BGP. Centralizing Internet routing, what could go wrong? (As the authors say, "One reason is there is no single entity that has a big picture of what is going on, no manager". I wonder who will be Internet's manager.) Centralized Internet routing - sounds like DoH for BGP. What could possibly go wrong?! -Hank
Re: Art and Tech is madness
On 05/09/2019 08:09, Kasper Adel wrote: No. This is art & tech from 12 years ago: https://www.youtube.com/watch?v=_y36fG2Oba0 -Hank In SPRING a time when segment and routing had no mismatch, a time when isis and ospf ate a forbidden encap, all they had to do was forward bgp like its hot, but crazy flapping doesnt leave any real LDP without some real FSM check, My dynamic unnumbered neighbor. Suddenly, Out of order, an AS is overridden, we see frames dropping, we sniff a bit and it turns out, sfps are burning, we are in a place right now where ping and pong are jittery, their latency is tested, they cant strengthen their icmp bond with a warm bfd message, how can they keep everyone in ACK, safe from teardown and dampening, with this kind of ixp relationship??! but oh admin, we know forwarding works in its own mysterious ways. We are left with two non rfc compliant scavengers, bastard 802.1ah fools in a leaky yet shaped, buffer display of some runts and nimbles, and a giant too. They start their life of a packet, leaving one interface to a neighbor, from an adjacency to a peer, an endless loop, its a prefix hijack, but as they move from one stack to another, finding their way through a tunnel of memory failures and RMAs, one hell of an LSP ride, through firewall horrors and MTU mismatches, leaving behind, a sea of syslog messages and snmp alarms. Anyway, Their ttl expired and one funny access list abruptly denies them life, sending them to Null0, where they can be peacefully discarded. Thats what tech does to yeh
Looking for Cloudfront clue
Can someone with routing/BGP/peering clue in AWS's Cloudfront, please contact me offlist? Thanks, Hank
Re: Mx204 alternative
On 02/09/2019 11:16, Mark Tinka wrote: On 8/Aug/19 05:33, Brandon Martin wrote: MX204 is a very nice pizza box router for service providers. I'm not aware of anything quite like it in terms of having a mature control plane. I like the JunOS config language better than Cisco-style that most other folks use. The MX204 is pretty hard to beat. It fits well as a peering/transit router, as well as a Metro-E router where you need a 100Gbps ring to carry 10Gbps customers, as well as downstream cheaper routers that will do sub-10Gbps quite nicely. That said, at least for the Metro, I still believe a lighter version of the MX204, with dense 1Gbps capability, is still needed. Been asking since 2007. Mark. What about handling LAG on 1Gb/sec links? That is a major showstopper if indeed it is missing: https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/speed-gigether-options.html • On MX10003 and MX204 routers, rate selectability at PIC level and port level does not support 1-Gbps speed. • On MX10003 and MX204 routers, the interface name prefix must be xe. • On MX10003 and MX204 routers, even after configuring 1-Gbps speed, the protocol continues to advertise the bandwidth as 10-Gigabit Ethernet. • On MX10003 and MX204 routers, Link Aggregation Group (LAG) is supported on 10-Gbps speed only. It is not supported on 1-Gbps speed. -Hank
Re: Corporate Identity Theft: Azuki, LLC -- AS13389, 216.179.128.0/17
On 15/08/2019 06:16, Ronald F. Guilmette wrote: - If the resource owner is no where to be found, why should we as a community care? I'm so glad you asked. Regardless, in -either- the case where no heir can be found -or- in the case where the rightful heir is either just too dumb or just too lazy to take the minimal steps necessary to reclaim the property (and/or before this has ocurred) the community should care because the kind of people who either steal or squat on IPv4 blocks are, almost without exception, not the kind of people who anybody sane wants to be accepting packets from, let alone peering with. There is, in my opinion and experience, a high degree of correlation between skulduggery with respect to -obtaining- (illicitly) IPv4 address blocks and using those addresses in a manner which is not at all conducive to the general welfare of the Internet or its users. So if the rightful is apathetic, then won't these new "malicious blocks" just end up in numerous blacklists and all the illegal activity being performed from those usurped blocks will just be blocked in the end? Since the RIRs won't do much(as much as we have tried) why not just leave it be (as much as it may hurt to do that) and let the bad blocks just become part of the blacklist sludgepool? Report it on some webpage and call it "Internet Resources stolen", document every incident as you do via email, send a copy to the appropriate RIR and upstream ISP allowing the hijack in question to show that you did the appropriate effort and we can then move on. I can and will stop posting here, and go off an blog about this stuff instead, if the consensus is that I'm utterly off-topic or utterly uninteresting and useless. But a few folks have told me they find this stuff interesting, and it has operational significance, I think. So for now, at least, I'd like to continue to share here. Suggestion: post here a link to your new blog for every incident you find. State here something like "/22 stolen from registered in country aaa by yyy located in country bbb". Those that are interested will click on the link and I suggest you allow comments on every blog post so that people can respond and comment. Regards, Hank
Re: RPKI adoption
On 14/08/2019 06:24, John Curran wrote: When you did that Whois look up at the ARIN website, you did agree to terms of use for the Whois service which contains indemnification provisions and are legally enforceable. <https://www.arin.net/resources/registry/whois/tou/> If you instead used a command line interface (e.g. "whois -h whois.arin.net <http://whois.arin.net> …”), then you received output from ARIN’s Whois server along with notice of the applicable terms of service… I would observe that continued use at that point has been held to indicate agreement on your part [ref: Register.com <http://Register.com>, Inc. v. Verio, Inc., 356 F.3d 393 (2d Cir. 2004)] Thanks, /John John Curran President and CEO American Registry for Internet Numbers Just like to add kudos to John for being open and responsive on this list and other lists to numerous issues and questions in regards to ARIN. Not many CEOs are willing or able to respond as you do. Thanks for your time and effort, -Hank
Re: Corporate Identity Theft: Azuki, LLC -- AS13389, 216.179.128.0/17
On 13/08/2019 22:17, Ronald F. Guilmette wrote: Just as an observer to your long resource theft postings: - Do you attempt to contact directly the organization or person who have had their resource taken over? - Do they care or are they apathetic? - If the resource owner is no where to be found, why should we as a community care? Report it on some webpage and call it "Internet Resources stolen", document every incident as you do via email, send a copy to the appropriate RIR and upstream ISP allowing the hijack in question to show that you did the appropriate effort and we can then move on. Regards, Hank
Re: Bgpmon alternatives?
On 18/07/2019 08:44, Töma Gavrichenkov wrote: On Thu, Jul 18, 2019 at 3:16 AM TJ Trout wrote: Anyone know of a hosted alternative to bgpmon? I'm testing Qrator but I can't determine if it will notify in real-time of a prefix hijack? Qrator guy there. Real-time notifications are there but are only available on a commercial basis, because basically real time is expensive to compute. The rest is free. -- Töma What about once a day notification of BGP hijack? Is that also expensive to compute? I have an account and cannot find any documentation of realtime notifications nor its cost. All I found was this - https://qrator.net/en/pricing . Can you send links to the BGP hijack notification service and its cost? Thanks, -Hank
Re: Performance metrics used in commercial BGP route optimizers
On 16/07/2019 20:41, Job Snijders wrote: On Tue, Jul 16, 2019 at 3:33 PM Mike Hammett <mailto:na...@ics-il.net>> wrote: More like do whatever you want in your own house as long as you don't infringe upon others. That's where the rub is; when using "BGP optimisers" to influence public Internet routing, you cannot guarantee you won't infringe upon others. The argument against route optimizers (assuming appropriate ingress\egress filters) is a religious one and should be treated as such. There is a difference between BGP optimizers and route optimizers. When was the last time you heard a complain about Akamai screwing up the global routing table over the past 12 years: https://www.akamai.com/us/en/about/news/press/2007-press/akamai-introduces-advanced-communications-protocol-for-accelerating-dynamic-applications.jsp https://developer.akamai.com/legacy/learn/Optimization/SureRoute.html -Hank
Re: CloudFlare issues?
On 25/06/2019 08:17, Christopher Morrow wrote: On Tue, Jun 25, 2019 at 12:49 AM Hank Nussbacher wrote: On 25/06/2019 03:03, Tom Beecher wrote: Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work on 701. My comments are my own opinions only. Respectfully, I believe Cloudflare’s public comments today have been a real disservice. This blog post, and your CEO on Twitter today, took every opportunity to say “DAMN THOSE MORONS AT 701!”. They’re not. Perhaps suggest to VZ management to use their blog: https://www.verizondigitalmedia.com/blog/ #coughwrongvz I think anyway - you probably mean: https://enterprise.verizon.com/ This post is unrelated to Verizon Enterprise? https://www.verizondigitalmedia.com/blog/2019/06/exponential-global-growth-at-75-tbps/ -Hank GoodLuck! I think it's 3 clicks to: "www22.verizon.com" which gets even moar fun! The NOC used to answer if you called: +1-800-900-0241 which is in their whois records... to contrandict what CF blogged about? -Hank
Re: CloudFlare issues?
On 25/06/2019 03:03, Tom Beecher wrote: Disclaimer : I am a Verizon employee via the Yahoo acquisition. I do not work on 701. My comments are my own opinions only. Respectfully, I believe Cloudflare’s public comments today have been a real disservice. This blog post, and your CEO on Twitter today, took every opportunity to say “DAMN THOSE MORONS AT 701!”. They’re not. Perhaps suggest to VZ management to use their blog: https://www.verizondigitalmedia.com/blog/ to contradict what CF blogged about? -Hank
Re: Russian Anal Probing + Malware
On 24/06/2019 00:23, Randy Bush wrote: e.g. i am aware of researchers scanning to see patching spread and trying to make a conext paper dreadline this week or infocom next month. hard to tell the sheep from the goats and the wolf from the sheep. i get the appended. sheep or wholf? i sure do not claim to be smart enough to know. but i sure am glad others are . Greynoise can be your friend: https://greynoise.io/about https://viz.greynoise.io/table -Hank randy ---
Re: Bgpmon alternatives?
On 16/06/2019 12:28, Töma Gavrichenkov wrote: On Sun, Jun 16, 2019, 4:57 AM TJ Trout <mailto:t...@pcguys.us>> wrote: Any simple and easy bgpmon alternatives you guys could recommend? https://radar.qrator.net/ (this is not an advertisement!) -- Töma I have been a subscribed member to your service for a number of years and do not see where I can receive an email pushed to my my inbox of a suspected BGP hijack. Can that be added? Regards, Hank
Re: Cisco Crosswork Network Insights - or how to destroy a useful service
https://bgpmon.net/wp-content/uploads/2019/01/BGPMon.net-EOL-EOS-faq.pdfOn May 15, 2019 14:52, "Mann, Jason" wrote: Is BGPmon going away? From: NANOG on behalf of Hank Nussbacher Sent: Wednesday, May 15, 2019 3:50 AM To: nanog@nanog.org Subject: Cisco Crosswork Network Insights - or how to destroy a useful service I have started to use Cisco Crosswork Network Insights which is the replacement for BGPmon and I am shocked at how Cisco has managed to destroy a useful tool. I have had a paid 50 prefix account since the day BGPmon became available and helped two clients implement a 500 prefix license over the past 4 years. None will be buying Cisco Crosswork Network Insights, based on my recommendation. I really don’t know where to begin since there is so much to dislike in this new GUI. I will try to give you just a small taste but I suggest you request a 90 day trial license and try it out for yourself. This was not designed by someone who deals with BGP hijacks or who manages a network. It was probably given to some GUI developer with a minimal understanding of what the users needed. How do I know this? Take for example the main configuration menu: https://crosswork.cisco.com/#/configuration with the first tab of “prefixes”. On that page there is no mention of which ASN the prefix is associated with. That of course was fundamental in the BGPmon menu: https://portal.bgpmon.net/myprefixes.php Or take for example its “express configuration”, where you insert an ASN and it automatically finds all prefixes and creates a policy. But does it know the name of the ASN? Nope. Something again that was basic in BGPmon via: https://portal.bgpmon.net/myasn.php is non-existent in CNI. Or how about the alarms one gets to an email? Want to see how that looks? From: Crosswork Admin [mailto:ad...@crosswork.cisco.com] Sent: 15 May 2019 11:39 To: Hank Nussbacher Subject: CCNI Notification Active alarm count 1 starting at 2019-05-15 08:34:42.960762315 + UTC. Please click on the link for each alarm below: https://crosswork.cisco.com/#/alarm/ba7c5084-f05d-4c12-a17f-be9e815d6647 Compare that with what we used to get: Possible Prefix Hijack (Code: 10) Your prefix: 99.201.0.0/16: Prefix Description: Kuku net Update time: 2018-08-12 17:50 (UTC) Detected by #peers: 140 Detected prefix: 99.201.131.0/24 Announced by: AS46 (BGP hijacking Ltd) Upstream AS: AS11 (Clueless ISP allowing customer hijacking Ltd) ASpath: 55 44 33 11 46 Alert details: https://portal.bgpmon.net/alerts.php?details&alert_id=830521190 Mark as false alert: https://portal.bgpmon.net/fp.php?aid=830521190 That is just a small sampling. Maybe two years down the road, Cisco will speak to customers first before destroying a useful service. Anyone else trying this out and feels the same or feels differently? Disappointed, Hank
Cisco Crosswork Network Insights - or how to destroy a useful service
I have started to use Cisco Crosswork Network Insights which is the replacement for BGPmon and I am shocked at how Cisco has managed to destroy a useful tool.I have had a paid 50 prefix account since the day BGPmon became available and helped two clients implement a 500 prefix license over the past 4 years.None will be buying Cisco Crosswork Network Insights, based on my recommendation. I really don’t know where to begin since there is so much to dislike in this new GUI.I will try to give you just a small taste but I suggest you request a 90 day trial license and try it out for yourself. This was not designed by someone who deals with BGP hijacks or who manages a network.It was probably given to some GUI developer with a minimal understanding of what the users needed.How do I know this?Take for example the main configuration menu: https://crosswork.cisco.com/#/configuration with the first tab of “prefixes”.On that page there is *no* mention of which ASN the prefix is associated with.That of course was fundamental in the BGPmon menu: https://portal.bgpmon.net/myprefixes.php Or take for example its “express configuration”, where you insert an ASN and it automatically finds all prefixes and creates a policy.But does it know the name of the ASN?Nope.Something again that was basic in BGPmon via: https://portal.bgpmon.net/myasn.php is non-existent in CNI. Or how about the alarms one gets to an email?Want to see how that looks? From: Crosswork Admin [mailto:ad...@crosswork.cisco.com] Sent: 15 May 2019 11:39 To: Hank Nussbacher Subject: CCNI Notification Active alarm count 1 starting at 2019-05-15 08:34:42.960762315 + UTC. Please click on the link for each alarm below: https://crosswork.cisco.com/#/alarm/ba7c5084-f05d-4c12-a17f-be9e815d6647 Compare that with what we used to get: Possible Prefix Hijack (Code: 10) Your prefix:99.201.0.0/16: Prefix Description:Kuku net Update time:2018-08-12 17:50 (UTC) Detected by #peers:140 Detected prefix:99.201.131.0/24 Announced by:AS46 (BGP hijacking Ltd) Upstream AS:AS11 (Clueless ISP allowing customer hijacking Ltd) ASpath:55 44 33 11 46 Alert details:https://portal.bgpmon.net/alerts.php?details&alert_id=830521190 Mark as false alert:https://portal.bgpmon.net/fp.php?aid=830521190 That is just a small sampling.Maybe two years down the road, Cisco will speak to customers first before destroying a useful service. Anyone else trying this out and feels the same or feels differently? Disappointed, Hank