Re: Frustration with increasing information demands from Network Vendors
On Fri, Oct 11, 2024 at 08:59:00PM +1100, Karl Auer wrote: > But there is a simple solution. Lie! Lie strongly and continuously. I strongly endorse this, doubly so given that the companies insisting on pointlessly collecting all this private information are the same companies that are staffed by miserably incompetent CIOs/CTOs/CSOs and have no possible chance of keeping it secure. I've been doing this for decades, and it's been quite instructive to note which companies have leaked which information, often without publicly acknowledging a security breach/dataloss incident. (Of course it's possible they have no idea that such an incident has occurred or that someone inside is doing a little freelancing using customer data.) And regardless of which company is involved, *under no circumstances* should anyone ever provide a truthful answer to security questions (e.g. "What was your first car?"). Every answer to every question at every site should be different and every one of them should be wrong. ---rsk
Re: it's mailman time again
On Fri, Sep 01, 2023 at 10:16:05AM -0700, Randy Bush wrote: > and i just have to wonder about sending passords over the net in > cleartext in 2023. really? This is a non-issue. Given that pretty much every SMTP connection is encrypted and that the worst thing that an attacker in possession of one of your Mailman passwords can do is unsubscribe you (in which case you and the list manager will be notified, and you can solve the problem quite rapidly), no, this isn't a problem that anyone needs to worry about. I've run (and am running) a lot of mailing lists with Mailman including some large-ish ones for what's now approaching 20 years. The scenario above has never happened. Nobody's even tried, which isn't surprising given that such an attack is increasingly difficult and yields little, if any, benefit to the attacker. Moreover, any hypothetical attacker possessing the resources and expertise required to pull this off could certainly find far more effective things to do. ---rsk
Re: About emails impersonating Path Network
On Mon, Feb 06, 2023 at 12:41:43PM -0800, Michael Thomas wrote: > This seems like a perfect object lesson on why you should use DKIM and SPF > and make sure the sending domain can set up a p=reject policy for DMARC. But it's not. DKIM and SPF are mostly useless against competently executed email forgery -- and can even be exploited to make it worse. Thanks to a combination of increasingly bad user interfaces, user ignorance, TLD proliferation, and the ill-advised conflation of identification with authenticity, it's not currently possible to stop email forgery in any meaningful sense of the word "stop". Let me illustrate part -- *part*, only part because a full explanation would take many pages -- of this problem by example. There are many variations on this theme, so after reading it you're tempted to say "But": don't. Because for every one of those "buts" there's a counter, and again, expounding those would take many pages. Focus on the concept here, not the precise details of the particular example I've chosen. Let's suppose that example.net is the target of an impersonation campaign. And let's suppose that I'm the attacker. Thanks to ICANN and unchecked registrar greed, I can now register example.lol or example.guide or example.life or any of a thousand variations. Thanks to unscrupulous hosting operations who don't care who their customers are or what they're doing and can't be bothered to perform even perfunctory due diligence, I'll have no problem getting a web/mail host for example.lol. I can then clone example.net's web site, modifying the URLs as appropriate. I can also set up MX records, DKIM, SPF, etc. all of which are syntactically and semantically correct for example.lol. It shouldn't be too difficult to figure out which email addresses are valid at example.net: I could scrape their web site, or go through the NANOG or IETF or other mailing lists, or scrape social media, or just write to them from a freemail account somewhere and see what turns up. I can replicate those at example.lol. So what I have now is a web site and mail system that copies example.net in every detail EXCEPT for the TLD. And that matters little, (a) because all-singing all-dancing email interfaces are increasingly getting dumber and hiding the actual email addresses of correspondents and (b) even if they expose it, e.g.: From: Some Person nearly every user out there will not pick up on the TLD. (If you think "lol" is a bit too obvious, then go through the list. There are plenty of others that aren't. Not that it matters much, because I could always just namesquat on example-pro in the .net TLD or some other variation on that theme, just like was done to Path Networks in this instance. And nearly every user out there won't catch that either.) And as the chef's kiss on this, an increasing number of email user interfaces will check the DKIM record and dutifully mark messages like this as "authentic" -- which, from the viewpoint of DKIM, they absolutely are. Users will see the green address bar or checkmark or whatever signifies this, and their brains will turn off, and they'll proceed on the assumption that such messages are authentic. TL;DR: email anti-forgery technologies are useless wallpaper slapped over an underlying set of serious problems that nobody has any interest in fixing because they're highly profitable problems. They've completely failed to justify their cost/complexity and are readily exploited as part of attacks. ---rsk
Re: Perhaps it's time to think about enhancements to the NANOG list...?
On Sat, Mar 20, 2021 at 12:46:57PM -0600, David Siegel wrote: > The board has been thinking about enhancements to the NANOG list for a > couple of years now, with the goal of creating a modern interface that the > younger generation of engineers will be more comfortable using. This isn't a valid goal. It's fine that some people can't handle mailing lists -- but then they shouldn't be network engineers or system admins, because (a) using mailing lists is a fundamental skill required in those fields and (b) anyone can't master such a rudimentary task in relatively short order really isn't equipped to be an engineer/admin. (Just like someone who can't do binary arithmetic or grasp multi-step processes shouldn't be an engineer/admin. This doesn't make them bad people, it just makes them people who are unlikely to be successful in the field.) > Those of you that have attended recent NANOG members meetings may recall > that we are currently beta testing a new community interface called > discourse as part of our NANOG modernization strategic initiative. Discourse is a MAJOR downgrade from the functionality of mailing lists. Oh, it's shiny and pretty and all that, but it's not a good tool for serious professional or even amateur communication. (And, of particular interest to *this* list, it performs extremely poorly -- if at all -- when confronted with (a) network outages and congestion and (b) attacks and abuse. Two of the *many* significant advantages of properly-run mailing lists are that they continue to function plausibly well under highly adverse conditions and that there are numerous, well-understood tactical and strategic mechanisms for defending them.) ---rsk
Re: Perhaps it's time to think about enhancements to the NANOG list...?
On Thu, Mar 18, 2021 at 03:28:31PM -0700, Matthew Petach wrote: > If only we had some way to segregate out different topics > of interest or disinterest, so that people who weren't interested > in questions about bandwidth availability could unsubscribe > from those topics, and only subscribe to the topics that *did* > interest them... 1. Anyone who's using a professional-quality mail client like mutt already has this capability. There's no need to modify the mailing list to accomodate people who choose to use any of the lesser mail clients that don't facilitate basic threading, filtering, and sorting. 2. This is a low-traffic list, so even without appropriate mail client support it's really not a big deal. ---rsk
Re: OVH datacenter SBG2 in Strasbourg on fire ????
On Fri, Mar 12, 2021 at 02:46:51PM +, David Hubbard wrote: > After sending them abuse reports for years with only an increase in > malicious traffic, I have no expectation of anything they do getting > better or being for the benefit of the internet as a whole. This is a shared experience. It's abundantly clear that OVH is either acting in concert with the abusers, or they *are* the abusers. It doesn't really matter which, the operational outcome is the same in either case. The best course of action is to remove them from your (generic you) view of the Internet via whatever means are most expedient. ---rsk
Re: OVH datacenter SBG2 in Strasbourg on fire ????
If you give people the means to hurt you, and they do it, and you take no action except to continue giving them the means to hurt you, and they take no action except to keep hurting you, then one of the ways you can describe the situation is "it isn't scaling well". --- Paul Vixie (on NANOG) 1. I have every OVH network block that I'm aware of permanently firewalled out. I recommend the same to everyone else, unless you're doing research. 2. When something burns that's not supposed to burn, that was designed not to burn, that was built not to burn, that was operated not to burn, then one of the distinct possibilities is that it wasn't an accident. ---rsk
Re: Texas internet connectivity declining due to blackouts
On Mon, Feb 22, 2021 at 08:44:32PM +0200, Saku Ytti wrote: > On Mon, 22 Feb 2021 at 20:28, Rich Kulawiec wrote: > > > right: artificial sweeteners are safe, WMDs were in Iraq, and Anna Nicole > > Hope you meant to write 'unsafe', as the conspiracy theory is that > aspartame is unsafe, the science says it is safe. Those last three points are a quote from a movie -- which is why I included the shout-out to Levon Helm (warning, spoilers, the quote's at about 2:00): Shooter: Levon Helm as Mr Rate - YouTube https://www.youtube.com/watch?v=xVw8FPIvOZc I included them as a joke because anyone who disputes AGW at this point *is* a joke and should be laughed out of the room. Less snarktastically, a very good starting point for people who want to understand the science of global warming is this document: Climate Change 2013: The Physical Science Basis https://www.ipcc.ch/site/assets/uploads/2018/02/WG1AR5_all_final.pdf It's exhaustive in its coverage (which is why it's 1500+ pages) and reading it will require basic literacy in math/stat and physical science. But it's part of the required homework for anyone trying to understand this topic. The 2021 version is now in preparation and if things go well, it should be out mid-to-late summer. Another highly useful document is: Fourth National Climate Assessment https://nca2018.globalchange.gov/downloads/NCA4_2018_FullReport.pdf which also clocks in at 1500+ pages. This document has both a broader focus (for example, it discusses impacts and mitigation) and a narrower focus (it's US-centric). It's also written for a more general audience and requires less math/science background for comprehension, so I recommend that anybody who struggles to get through the one above try this instead. Also: this one is arguably more useful for NANOG/operational/planning purposes. I think it'd be a good read for anyone who's trying to figure out what's going to happen to their physical assets/locations, or for anyone who's trying to plan where to put things and how to build them. Additional resources: Climate Change and Infrastructure, Urban Systems, and Vulnerability Technical Report for the US DoE Thomas Wilbanks and Steven J. Fernandez (This one is also useful for NANOG denizens. Chapter 5 on risk mitigation strategies is particularly interesting.) The Encyclopedia of Global Warming and Climate Change, 2ed S. George Philander, editor (A general reference. Having this handy while reading the IPCC report I mentioned above can be helpful.) Atmospheric Thermodynamics - Elementary Physics and Chemistry Gerald R. North and Tatiana L. Erukhimova (You'll need integral and differential calculus for this one, and a previous course in introductory thermodynamics will help. This is not about climate -- or weather -- per se, but it provides some of the fundamental science necessary to understand both.) Attribution of Extreme Weather Events in the Context of Climate Change Committee of Extreme Weather Events and Climate Change Attribution (Attribution science is relatively new but is making rapid progress. The ability to look back and demonstrate causal relationships is going to be invaluable as we look forward.) rsk
Re: Texas internet connectivity declining due to blackouts
On Mon, Feb 22, 2021 at 05:48:06PM +, Mel Beckman wrote: > Sorry Global Warmists, Right. Sure. Also, the earth is 6,000 years old (and flat), the moon landings were faked, creationism is real, dinosaurs and humans co-existed, vaccines cause autism, Elvis is alive, and...how does that line go? Oh, right: artificial sweeteners are safe, WMDs were in Iraq, and Anna Nicole married for love. [shout-out to Levon Helm] This trash doesn't deserve rebuttal: it deserves ridicule. ---rsk
Re: Texas internet connectivity declining due to blackouts
On Wed, Feb 17, 2021 at 10:34:35AM -0800, Sabri Berisha wrote: > With apologies to those on the list who still use mutt/pine etc. 1. "still"? Competent professionals with security awareness use text-only email clients as a matter of basic self-defense. I trust it's obvious why those of us who are responsible for systems/networks/data need to protect ourselves in order to protect the resources we run. 2. It's pretty easy to handle attachments gracefully while using mutt. By default it checks ~/.mailcap, and in that file one can specify external programs to handle various attachment types, e.g.: text/html; w3m -dump -T text/html %s | less application/pdf; /usr/bin/evince %s Of course this needs to be done carefully, since it subjects the user to attacks against those applications carried via attachments, but that's where judicious choices on the part of the user come in -- choices as in "which attachment types, which applications, and whether or not to exercise this capability on a per-message basis". ---rsk
Re: Texas internet connectivity declining due to blackouts
On Tue, Feb 16, 2021 at 12:23:22PM +, Bret Clark wrote: > Texas doesn't generally experience this type of extreme cold. That was then; this is now. As scientist Jeff Masters put it most of a decade ago: The atmosphere I grew up with no longer exists. My new motto with regards to the weather is, "expect the unprecedented." In the years since he's said that we've seen a number of unprecedented events: Sandy, Harvey, California wildfires, last year's midwest derecho, and so on. This event in Texas is just another one; there will be more; they'll get worse. We should probably get ready for that. ---rsk
Re: Famous operational issues
On Tue, Feb 16, 2021 at 01:37:35PM -0600, John Kristoff wrote:
> Which examples would make up your top three?
Morris worm, November 1988. Much confusion and eventually the realization
the John Brunner had called it from 13 years out ("The Shockwave Rider", 1975).
But sloppy coding meant it could be defeated with one line of /bin/sh.
---rsk
Re: Texas internet connectivity declining due to blackouts
On Tue, Feb 16, 2021 at 04:17:15AM +, Robert Jacobs wrote: > How about letting us Texans have more natural gas power plants or even > let the gas be delivered to the plants we have so they can provide more > power in an emergency. Did not help that 20% of our power is now wind > which of course in an ice storm like we are having is shut off... Lots > of issues and plenty of politics involved here.. First, wind-generation-is-responsible-for-this is a canard that's already been debunked elsewhere in this thread. Second, wind generation works just fine all winter in places like Quebec and the Alps because they design, build, and operate it to. Various de-icing solutions have been available for years, and market competition is continuously making them better and cheaper. Third, trying to slap a fossil fuel band-aid on a problem whose root cause is...wait for it...fossil fuels...while certainly a tempting option, is not a viable long-term solution. ---rsk
Re: Past policies versus present and future uses
On Mon, Jan 25, 2021 at 11:26:51AM -0500, Rob McEwen wrote: > Is DDoS-Guard without blame? Probably not, but them hosting some occasional > criminals is NOT UNLIKE EVERY OTHER GLOBAL NETWORK! You might wish to scroll back up to the message I sent here on January 21 with the Subject "DDOS-Guard" and note the list of domains that I provided. That's not a network with "occasional" issues, that's a network with pervasive issues. > By these SAME standards, many other large and famous > networks should lose most or much of their IPs too! Yes, that's exactly what should happen. "Large and famous" operations, by their very nature, have plenty of money to spend on large, trained, competent, empowered, 24x7 abuse staff as well as on customer screening -- and should do that. Those that don't should not have their problematic allocations confiscated: they should have *all* their allocations confiscated. Why? Well, first because there are no acceptable excuses for running an operation like that. NONE. And second, because when those operations refuse to pay the costs of keeping abusers out, you know who *does* pay for that? We do. ---rsk
Re: opportunistic email encryption by the MTA (not MUA)
While I agree pretty much entirely with everything you've expressed, there is another force in the world quietly chugging away to make sure that email privacy remains largely hypothetical...and that is: cloud computing. A lot of people have outsourced their mail service to cloud operations, so even if the mail servers on both ends do everything "right", which (to your point) might include a refusal to transmit messages unless an over-the-wire encryption method is agreed on, messages thus sent are available in plaintext on both sides of the transmission and thus can be inspected/collected/etc. at will by the operators of the cloud [1]. Or by anyone who's penetrated the cloud. Note that while use of PGP/similar to encrypt the message itself helps with this, that doesn't stop traffic analysis on either side of the transmission. ---rsk [1] Which includes the people working there and working for them, as well as the people working there and not working for them.
Re: Looking for hosted SMTP service for small ISP
On Thu, Jan 14, 2021 at 03:45:06PM -0700, Grant Taylor via NANOG wrote: > Be mindful that there is and has been a concerted effort to block parts of > SendGrid. There's even a relatively new RBL -- which started because of > SendGrid -- specifically for blocking some of their worse customers. Many > people in the effort have been advocating for simply blocking SendGrid > completely. I recommend the latter course. I have spam samples from SendGrid going back to 2010 and continuing to the present, including spam to never-existed addresses. They've had 10 years to figure out how not to spam -- something that should take less than 10 minutes -- and they still haven't managed it. ---rsk
Re: Re Parler
On Thu, Jan 14, 2021 at 11:01:19AM -0700, Keith Medcalf wrote: > This result will only come to pass if Parler wins their lawsuit (which is > likely) The first hearing in this case was held today. Per reporting by Katherine Long of the Seattle Times, during that hearing Parler's attorney: - forgot the name of Parler's CEO - stated that he's unfamiliar with some of the terminology because he's not on social media - admitted that he filed a day late because he needed to update his PACER account This is the same attorney who filed Parler's complaint -- the one that names Twitter as a defendant in section 5 while omitting them from the title page of the complaint. Here, look for yourself: PARLER LLC v. AMAZON WEB SERVICES https://www.courtlistener.com/recap/gov.uscourts.wawd.294664/gov.uscourts.wawd.294664.1.0.pdf Read the first sentence of section 5. It's on page 3. Oh heck, let me save you the trouble: "Thus, AWS is in violation of Section 1 of the Sherman Antitrust Act in combination with Defendant Twitter." I am not an attorney but my general understanding is that if you wish to file a civil complaint against multiple defendants that you should actually go through the trouble of naming them all as defendants on the complaint (and serving them). ---rsk
Re: DoNotPay Spam?
On Wed, Jan 13, 2021 at 05:06:15PM -0500, Robert Webb wrote: > Anyone else getting spam from DoNotPay everytime they send an email to the > list? This is solvable by permanently blocking all traffic from Mailgun in your MTA. This should be a good start and may suffice: mailgun.info mailgun.net mailgun.org mailgun.us ---rsk
Re: Parler
Given that people on Parler are currently discussing/planning attacks against Amazon/Google/Apple/etc.'s facilities and personnel, this seems wise. ---rsk
Re: WhatsApp's New Policy Has...
On Fri, Jan 08, 2021 at 01:31:56PM -0600, Dave Phelps wrote:
> Keybase was purchased by Zoom (
> https://www.cnbc.com/2020/05/07/zoom-buys-keybase-in-first-deal-as-part-of-plan-to-fix-security.html).
> >From what I've gathered, Zoom is too tight with, owned by, or run by China,
> so I believe there was a similar mass exodus from Keybase for lack of trust.
I've been maintaining a page of relevant links concerning Zoom since
late winter 2020. It's here:
Zoom
http://www.firemountain.net/zoom.html
I need to add a link there concerning the complaint filed in the EDNY,
USA v. Xinjiang Jin (JIN). As pointed out by File411, there are repeated
references in that complaint to "under 1 minute", as in:
Employee-1 explained that "The current requirement" -- apparently
referring to Company-1's internal restrictions -- "is that domestic
engineers cannot access the data of us clusters" -- indicating
that PRC-based software engineers were not permitted to access user
data stored on U.S.-based servers. JIN responded "Net Security's
requirement is that [the employer] must have the authority to
directly handle it, and it must be handled within one minute.
For example, including U.S. users, if the issue of June 4th is
being discussed in a meeting, it must be handled within one minute
of [the meeting being reported], otherwise will be [rate] as
security non-compliant."
("June 4th" refers to Tiananmen Square - June 4, 1989.)
It's unclear yet exactly what this means/implies, but my working assumption
for the moment is that everything passing through Zoom is being made
available in real or close-to-real time to the PRC.
Also in the complaint:
JIN wrote in an electronic messages to other individuals who are
Company-1 employees stating that, even if other U.S. social media
and search companies had no business in the PRC, they still terminated
accounts and posted at the request of the "CN zf". Based on open
source information and my training and experience, the "CN" in "CN zf"
refers to "China" (the PRC) and "zf" is shorthand for zhengfu,
a Chinese word for government.
---rsk
Re: NDAA passed: Internet and Online Streaming Services Emergency Alert Study
On Thu, Jan 07, 2021 at 01:27:07AM +1300, Mark Foster wrote: > I respect this in principle, but hyperbole serves no-one - a smartphone > only creates a "morass of privacy/security issues" if you let it. You can't be serious. Have you paid *any* attention to what's been going on in this ecosystem for the past N years? It's not as bad as the raging dumpsterfire in the IoT, but it's still bad. Why would I want to give myself security/privacy issues (that I currently don't have and thus don't need to solve/manage on an ongoing basis) in exchange for functionality I don't need or want? > A basic smartphone can be had for less than $100 USD, which would give > you calling, text messaging and emergency alerts. It would also give me a much less sturdy device and one that chews up its battery doing things that I have no use for. I [sometimes] use my phone for critical communications in hostile environments, so anything that doesn't increase the probability that it will work is just baggage. And as a bonus it would cost me more every time I lose or destroy one. ---rsk
Re: NDAA passed: Internet and Online Streaming Services Emergency Alert Study
On Mon, Jan 04, 2021 at 09:08:06PM -0600, Billy Crook wrote: > Then again how many people would benefit from adding this to online > streaming, but don't already have cellphones that have emergency alert > popups that get their attention. The kind of people who don't have > smartphones are going to be the ones still watching bunny ears television > anyway. I've worked in science and technology for a long time, and I've chosen not to have a smartphone. Not that I have to justify this decision to you, but: (1) I spend a fair amount of my time in environments with poor/no connectivity (2) I participate in activities that are likely to result in the destruction or loss of the phone and (3) I use my phone...for phone calls and the occasional text. So spending $40 rather than $800, avoiding the morass of privacy/security issues involved in a smartphone, and maximizing available battery life seems like the best move. I know others who've made the same decision for similar reasons. ---rsk
Re: NDAA passed: Internet and Online Streaming Services Emergency Alert Study
On Sun, Jan 03, 2021 at 03:26:07AM -0500, Valdis Kl??tnieks wrote: > Meanwhile, this causes yet another problem - if Hulu has to be able to > know what alerts should be piped down to my device, this now means that > every single police and public safety agency has to be able to send the > alerts to Hulu (and every other streaming company) - and do this securely. And then there's another problem (that I'm going to bet you've already thought of, given what you've written here): Hulu and every other streaming company need to be able to authenticate the alerts from all those different agencies. Those agencies also need to secure their sending infrastructure...and good luck with that. And then there's another problem, which is that once all those different agencies have this facility, they're going to (ab)use it as they see fit. I've noticed that over the last decade or so that weather alerts I've received are covering increasingly-less-severe events, e.g., we've slowly gone from "there's a tornado on the ground" to "there's going to be a thunderstorm". And at this particular point in history, I can think of one person who would be using this every five minutes simply because it's there. ---rsk
Re: NDAA passed: Internet and Online Streaming Services Emergency Alert Study
On Fri, Jan 01, 2021 at 05:07:22PM -0500, Sean Donelan wrote: > Not later than 180 days after the date of > enactment of this Act, and after providing public notice and > opportunity for comment, the Commission shall complete an > inquiry to examine the feasibility of updating the Emergency > Alert System to enable or improve alerts to consumers provided > through the internet, including through streaming services. I do hope that they consider the certainty that this will be subjected to abuse the moment it goes live. ---rsk
Re: Are the days of the showpiece NOC office display gone forever?
On Tue, Dec 22, 2020 at 10:41:43PM -0700, Wayne Bouchard wrote: > And if the last 15 years has shown us anything, it is that when you > can't get past the auto-attendant and talk to a real human, and if > that person can't talk to you like a person instead of reading scripts > at you, your stress levels go way up as does your desire to break > things. Automation in customer service (or excessive emphasis on > procedures) is a really nice way of taking a five minute problem and > turning it into an hour long ordeal. There are some easy methods for service/support organizations to decrease the pain that this inflicts on people reporting problems. For example, one thing that I've taught people to do is to make liberal use of procmail in order to sort incoming traffic to role accounts. It requires diligence, but that diligence is repaid many times over by how it expedites dealing with problems. A simple example of this is that when a problem report is received at the RFC 2142 security@ role address, and it's clueful, well-written, and important, a procmail rule gets created for the sending address so that all future messages from that address are prioritized...because it obviously came from someone who knows what the heck they're doing and did us a favor by telling us that we have a problem. Chances are that any future messages from them will be similarly helpful and that if we respond to those quickly we may be able to forestall a lot more messages that aren't going to be as clueful. The opposite thing is done with clueless/misdirected/etc. reports: they're not discarded, but they go into the low-priority queue. Everything else goes somewhere in the middle. Repeated hundreds or thousands of times over many years, this builds a ruleset that pre-sorts messages rather well. It's not perfect, it's not foolproof, but it helps us *and* it helps lower the frustration level of people sending clueful messages, because it better positions us to read, act on, and respond to those. Those people are catching our mistakes, the least we can do is try to pay attention. (Hint: a useful way to begin building such a ruleset is to grab all the addresses from NANOG, dnsops, outages, etc. and pre-load the ruleset with them...because traffic received at role accounts from participants in these mailing lists is probably useful.) ---rsk
Re: "Hacking" these days - purpose?
On Mon, Dec 14, 2020 at 09:58:01AM -0500, Tom Beecher wrote: > Questionable cloud / VPS / hosting companies are great for spammers and > botnet C&C, but not so great for DDoS "ion cannons". You still need a large > volume of geographically diverse endpoints for those to be effective. To piggyback on this: when launching a DDoS, diversity along multiple axes is helpful: geography, topology, connectivity, operating system, etc. Each additional form of diversity slightly raises the bar for defenders. Also, every compromised device may be a source of useful/saleable data, or the gateway to more of the same or to more valuable targets or to the compromise of people. The IoT is particularly fertile ground for this because to a very good first approximation, "IoT security" is an oxymoron. --rsk
Re: Weather Service faces Internet bandwidth shortage, proposes limiting key data
On Thu, Dec 10, 2020 at 09:48:25AM -0500, Jared Mauch wrote: > I miss weather underground before it became slow as molasses with > openstreetmap and other things. As do I, and the demise of uswx.com took away one of the alternatives. I spent some time earlier this year unsuccessfully trying to contact the people behind uswx.com in the hopes of restarting it as a very lightweight site that could be read in a text-only web browser *and* scripted -- since I think it would be handy to be have a weather site that could easily be scraped for custom uses such as "cron job that sends me a brief 3-day forecast for locations X Y Z every day at 6 AM". ---rsk
Re: The Real AI Threat?
On Thu, Dec 10, 2020 at 12:34:33AM +, Mel Beckman wrote: > So don???t be fooled by Siri and Google voice response. There is no > intellect there, only pattern matching. Which we???ve been doing with > machines since the Jacquard Loom. On this particular point: many years ago, some of us at Purdue discussed this at great length and eventually coined the term "ad-hockery" -- which found its way into the New Hacker's Dictionary. The gist of the idea is that it's possible to craft a sufficient number of ad hoc rules, plug them into a pattern matcher, and present a modestly plausible appearance of "intelligence"... when in fact nothing resembling actual intelligence is involved. Such systems are brittle and demonstrate it when presented with input not covered by those ad hoc rules, which is why they're often made progressively less so by repeated tweaking. (Also known as "release 3.0" and accompanied by prose touting it as an innovative upgrade.) But to borrow Mel's phrasing, even a very large collection of ad hoc rules that performs its task tolerably well is no more intelligent than the loom. ---rsk
Fwd: Weather Service faces Internet bandwidth shortage, proposes limiting key data
- Forwarded message from Dave Farber - > From: Dave Farber > Date: Thu, 10 Dec 2020 15:47:44 +0900 > Subject: [IP] Weather Service faces Internet bandwidth shortage, proposes > limiting key data > > Weather Service faces Internet bandwidth shortage, proposes limiting key data > The National Weather Service is proposing to place limits on accessing its > life-saving weather data in a bid to fix Internet outages. > By Jason Samenow and Andrew Freedman > > https://www.washingtonpost.com/weather/2020/12/09/nws-data-limits-internet-bandwidth/ [snip] This seems like a problem that this group could solve rather rapidly with minimal incremental expense. It also seems like one that's very much worth solving. ---rsk
Re: Technology risk without safeguards
/Friday afternoon On Thu, Nov 05, 2020 at 09:05:34AM -0800, William Herrin wrote: > Following staff home and picking them off with a rifle is so much > cheaper and carries a better probability of success. So does following them home and leaving them brand new unopened large bottles of Woodford Reserve. I highly recommend this approach for anyone who has selected me as a target and promise that I will duly report on its progressive deleterious effects. For accuracy, repeated trials over an extended period of time may be necessary but this is an ordeal I'm selflessly prepared to undertake for the sake of science. ---rsk p.s.1: I've worked in high-energy EM environments twice, in two different contexts. The safety measures were thorough and rigorous: it would have been very hard to screw up and even if any of us had, the inverse-square law would probably have saved us from serious harm. p.s.2: The large quantities of power conduits, cables, shelving, racks, HVAC ductwork, etc. that are typical of datacenters constitute a haphazard but modestly effective EM shield, as measured on an ad hoc basis by anyone who tries to receive external signals inside them (even when everything is powered down) will quickly discover. Thus an attempt to pull off a movie villain-grade underground attack designed to fry a staff member would likely require that the victim stand still on a selected spot (on the lowest-level floor) with a minimal amount of metal under it. I recommend that prospective attackers use the Wile E. Coyote (Sooper Genius) methodology, draw a large X on that spot, and install a sign that says "Free Birdseed". I'm certain this will work.
Re: Virginia voter registration down due to cable cut
On Sat, Oct 17, 2020 at 07:44:01PM -0400, Sean Donelan wrote: > In the USA, absent clear and convincing evidence otherwise, I expect any > outages will be due to the normal things that cause outages on election day. One of those things is the chronic underfunding of the systems/personnel involved. (This isn't the case everywhere of course but it's the case in a lot of places.) This leads to operations that are working -- barely -- and thus not resistant to stress, outages, and mistakes. We could argue that given the criticality of this particular function that resources should be more generously allocated, but unfortunately that's often an unsuccessful argument. ---rsk
Re: Cogent emails
On Tue, Sep 22, 2020 at 07:58:42AM -0500, J. Hellenthal via NANOG wrote: > geeks@nanog works just fine Yes, it works just fine for *that* purpose. However, *this* has a different purpose: Shining a light on ambulance chasers - Noction https://mailman.nanog.org/pipermail/nanog/2020-April/107054.html ---rsk
Re: Cogent emails
On Mon, Sep 21, 2020 at 06:30:24PM -0600, Grant Taylor via NANOG wrote: > Is this simply being aggregated by a NANOG member / subscriber and thus > something unofficial? That's exactly right. Whether NANOG itself ever wants to do anything with the results is entirely up to them. ---rsk
Re: Cogent emails
On Mon, Sep 14, 2020 at 12:45:32PM -0400, Dovid Bender wrote: > Is anyone starting to get the "cogent emails" again? Reminder: forwarding these sorts of things (with full headers please) to: nanog-spamm...@firemountain.net will cause them to be compiled into a list. ---rsk
Re: BGP route hijack by AS10990
On Mon, Aug 03, 2020 at 08:57:53AM -0400, Tom Beecher wrote: > Telia made a mistake. They owned it and will endeavor to do better. What > more can be asked? Figure out how that mistake happened -- what factors led to it? Then make changes so that it can't happen again, at least not in that particular way. (And if those changes are applicable to more than this isolated case: excellent. In that case, share them with all of us so that maybe they'll keep us from repeating the error.) "Stopping myself from making the same mistake twice" has probably been the most effective thing I've ever done. ---rsk
Re: (updated) COVID-19 fast/small resources page
Update on: On Mon, Mar 23, 2020 at 10:42:32PM -0400, Rich Kulawiec wrote: > It's here: http://www.firemountain.net/covid19.html I've been updating this every 24-48 hours. It now includes every applicable case/test tracker I'm aware of and links to quite a few articles and papers. (I may break the latter out into a second page; there's obviously a lot of activity and it's starting to become difficult to figure out what's most germane. I've long since pushed journalism links to a second page, because there are too many.) I've gotten helpful feedback from the NANOG and REN-ISAC folks (thanks!) and have added some things in response to that. If there are other relevant and reliable resources that would assist the community, I'd be happy to add them. Please ping me off-list. ---rsk
Re: Abuse Desks
On Tue, Apr 28, 2020 at 12:40:12PM -0400, Matt Corallo via NANOG wrote: > Please don't use this kind of crap to send automated "we received 3 login > attempts on our SSH box..wa" emails. > This is why folks don't have abuse contacts that are responsive to real > issues anymore. [ "you" = rhetorical "you", throughout ] No, the reason that folks don't have responsive abuse contacts is that they're some combination of: - lazy - cheap [1] - incompetent - unprofessional - actively supporting the abusers A "we received 3 login attempts on our SSH box" complaint should be read, investigated, and acted on. It means that something is going on that shouldn't, and so for your own sake, as well as for the well-being of your Internet neighbors, you should find out what that is. That "for your own sake" clause is often overlooked. An incoming abuse complaint is sometimes the canary in the coal mine. Ignoring it because it appears to be trivial at first glance is extremely foolish. The lesson of the 75-cent accounting error is now 34 years old. This would be a really good time to learn from it. You should also consider that -- thanks to the negligence and incompetence of many abuse desks -- a lot of people simply don't bother reporting incidents any more. Thus what presents to you, on the surface, as "we received 3 login attempts on our SSH box" may in fact be one isolated report of a much larger incident. It thus requires your immediate attention, if you want to even pretend to be a responsible, competent professional. Incidentally, an excellent way to reduce the number of "we received 3 login attempts on our SSH box" complaints is to deal with all of them, thus decreasing incident occurence, which will of course result in a corresponding decrease in complaints. An even better way is to run your operation in such a way that you detect and deal with as many such things as possible before anybody needs to file a complaint. After all, if they can see the traffic arriving on their side, you can see it leaving on yours. ---rsk [1] I note, for example, that Charter -- which is mentioned in the original message in this thread -- currently has a market capitalization of 116.63 billion dollars. Surely they could spare a tiny fraction of that to appropriately staff a 24x7 multi-lingual abuse desk with senior engineers and empower/equip them to do what needs to done. That's what a professional operation would do.
Re: mail admins?
On Thu, Apr 23, 2020 at 10:47:18AM -0700, William Herrin wrote: > One of the annoyances with both those guys and the swedish folks is > that they're not sending messages to the return path, they're > responding to the header from address. Mailman at NANOG never sees it. > It doesn't pass through their servers. Yeah, I know: I've seen similar things on the Mailman-powered mailing lists that I've run. > Even if mailman saw it, mailman doesn't use VERP. It has to scan the > message to match the subscriber and that doesn't consistently work. > The subscriber address forwards to another address, the second address > bounces and the bounce message doesn't necessarily contain the > original subscriber address. > > To identify these jokers the ops will probably have to send a unique > message to each subscriber with crafted headers. That can be folded in > to a message that would go to the list anyway but the capability isn't > baked in to mailman. I get all this, but there are other methods that should help narrow it down. For example (and I really do mean "example", this might or might not be useful in this case): one of the things I've done is to (1) grab the subscriber list (2) reduce it to domains (3) look up the MXs for every domain -- via a script (4) sort/uniq (5) see if any match the domain that appears to be the culprit. (Sometimes works.) Another approach is to look up the MX's for the culprit and see if those match any on the just-generated list. (Usually doesn't work.) Still another is to map the MX list to IP addresses, sort by address, then grab the IP addresses relevant to the culprit (from A, NS, MX) and see if those are local to any of the ones on the list. (Sometimes works.) All of these are hit-or-miss but I've found that pursuing them usually results in some insight, if not an answer. If nothing else it often allows the "exoneration" of some number of subscriber addresses, which reduces the scope of the problem and may render it tractable via other methods. But to your point, VERP or an equivalent tactic may be the only way to really diagnose/repair some of these. ---rsk
Re: mail admins?
On Thu, Apr 23, 2020 at 07:56:30PM -0700, Michael Thomas wrote: > $SHINYNEWSITE has only to entice you to enter your reused password which > comes out in the clear on the other side of that TLS connection.?? basically > password phishing. you can whine all you like about how stupid they are, but > you know what... that is what they average person does. that is reality. js > exploits do not hold a candle to that problem. Two equally large problems -- neither of which have anything to do with encryption in transport -- are backend security and password strength. In the former case, we've seen an ongoing parade of security breaches and subsequent dataloss incidents. That parade is still going on. In the latter case, despite years of screaming from the rooftops, despite myriad enforcement attempts in code, despite another parade of incidents, despite everything, we still have people using "password" as a password. As a side note, I've found it nearly impossible to get users to understand that there is a qualitative and quantitative difference between "password used for brokerage account" and "password used for little league baseball mailing list". The minor problem of passwords-over-the-wire pales into insignificance compared to these (and others -- but that's a long list). ---rsk
Re: mail admins?
[ Bunch of replies to messages in thread bundled here. ] On Tue, Apr 21, 2020 at 06:28:48PM -0400, Bryan Fields wrote: > It's a mailman list, so nanog-ow...@nanog.org should work. If not reach out > to the communications committee. All mailing lists should support that, regardless of what's running them. Mailman, thankfully, makes it easy for configuring it by default. Other topics: - I've received erroneous bounces from @email.uscc.net as well. It should be possible to track down the culprit via Mailman's logs and the MTA's logs. - There is zero point in obfuscating email addresses in archives or anywhere else on the 'net. None. There hasn't been any point for most of twenty years. It's cargo cult "privacy" and that capability should be excised from Mailman's source code, because its presence unfortunately encourages people to indulge in a worst practice. - I also volunteered (in 2018? not sure, need coffee) to help out with -owner technical issues, but never heard anything back. - One of the queries that I've sent but not had an answer to is whether the entire archives are available in "mbox" format. (Including the older ones.) If they are, then it should be pretty easy to fold the old pre-Mailman archives into the current with-Mailman archives. ---rsk
Re: 24x7 vs 24x7x365 Re: Constant Abuse Reports / Borderline Spamming from RiskIQ
(since it's Friday and we're all stressed) I can't believe that out of everything I wrote that we're going to discuss the semantics of this, but then again: yes I can. I should have known. I should have known. I. Should. Have. Known. *bangs head on desk* *reaches for scotch* Alrighty then: 24x7 means every hour of the week, as in "24 by 7". 24x365 means every hour of the year. (modulo those with 366 days but please let's not go there because this is bad enough) (oh wait, too late, someone upthread already went there) (and then leap seconds reared their ugly head, oh good grief) 24x7x365 thus means every hour of 7 years. YES, I know, I know. 60x24x7...no. NO. I will not go there. Nor will you. Just stop. I swear I will turn this car around *right now*. Yeah, I know it's in common use. Like any number of other things in common use (e.g., "going forward" -- really? like there's another direction to go?) it's...annoying. I suspect that someone who just wasn't thinking started this in an attempt to out-promote people who merely said 24x7 or 24x365, and it propagated outwards. If that hypothesis is correct and there is thus a patient 0 for this epidemic, I very much want to find them and pummel them with a bag of Oxford commas. rsk
Re: Constant Abuse Reports / Borderline Spamming from RiskIQ
On Wed, Apr 15, 2020 at 11:33:58PM -0400, Ross Tajvar wrote: > Can you give some examples of the things you mention above? I'm not doing > much in terms of customer filtering and would be interested to hear what > others consider best practice. Sure. These are just examples and are by no means exhaustive. Also, some will work better than others depending on who you are, what services you offer, where you are, etc. There's no substitute for human judgment seasoned with experience. 1. Let's start with a timely one. Whenever there's a national or global crisis, scammers begin registering domains to exploit it. For instance: Thousands of COVID-19 scam and malware sites are being created on a daily basis https://www.zdnet.com/article/thousands-of-covid-19-scam-and-malware-sites-are-being-created-on-a-daily-basis/ [I'll omit the long rant about why ICANN is responsible for this and should be ashamed of what they've not only allowed, but encouraged.] That story contains a link to a repository where somebody is tracking these. I pulled that list a month ago and there were 7500 entries. Now there are over 25,000. (Caveat for anybody doing the same: note carefully the methodology. There are legitimate domains/subdomains/hosts in there, although they're rapidly being swamped by the bogus ones. So don't just blindly use the data: filter out the 1-2% of legitimate entries.) So, if it's April 2020, and a customer comes to you and wants to set up web service for a domain or fifty that have "covid", "corona", "virus", etc., in their names: they're probably up to something. 2. There are longstanding versions of (1) as well. Domains with strings in them like "bulk", "seo", "credit", etc., or domains with variations on the names of financial institutions, or domains which are typos of well-known domains, etc., are all suspect. *That doesn't mean they're all bogus.* It just means that a human being should give them closer scrutiny before the process goes forward. 3. Look at the diversity of their domains. This sort of is a rehash of what I said in (2), but: if all their domains are about one or two topics, yeah, it's probably someone with a business and a hobby or something like that. But if they have domains that suggest they're running 17 different businesses, then look closer. 4. Look at whether they've been, that is, where they were hosted previously, by checking their DNS history. If they've hopped through four different hosts in the last seven months, something is going on. (Note: a few months ago a bunch of cheap VPS services all simultaneously ceased operations. If they were on one of those, then they may have just been caught up in the mess, so don't count that against them.) 5. Check Spamhaus. 6. Find out how many domains they have. People doing legitimate things may have 5 or 17 or something like that. People who have 5,000 are up to something. (Note: I've been doing research in this area for many years. I know of zero instances where registrants with thousands of domains were doing something legitimate. There may still be a counterexample out there, but I haven't seen it yet.) 7. MLM (multi-level marketing) is a red flag. So is Bitcoin et.al. 8. A business putatively located in Iowa but with contact email addresses @163.com or @yandex.com is dubious. Same for other incongruous information: it might really be okay, or it might be a hint that they're up to something. Most of these are just indicators: they're not definitive. And there are counterexamples all over the place. Plus, this list isn't exhaustive: like I said they're just examples. That's why I said at the beginning that there's no substitute for human judgment seasoned with experience. That takes time and probably more than a few bad experiences. But it's worth it, because it's easier to solve problems before you have them. ---rsk
Re: Constant Abuse Reports / Borderline Spamming from RiskIQ
On Mon, Apr 13, 2020 at 12:11:44PM -0700, Matt Corallo via NANOG wrote: > I don???t really get the point of bothering, then. AWS takes about > ~forever to respond to SES phishing reports, let alone hosting abuse, > and other, cheaper, hosts/mailers (OVH etc come up all the time) don???t > bother at all. Unless you want to automate ???1 report = drop customer???, > you???re saying that we should all stop hosting anything? No, you don't have to stop hosting anything/everything. But there are all kinds of things that can be done to detect problematic customers before you sign them up and once they're in place. None of those things are panaceas but all of them done in combination (a) reduce the chances that you'll have a mess to clean up later and (b) enhance one's reputation as a place NOT to go for dubious activities, which in turn discourages future miscreants from trying to get in the door. ---rsk
Re: Constant Abuse Reports / Borderline Spamming from RiskIQ
[ Copied to Jonathan @ RiskIQ because I don't believed he's subscribed. ] On Mon, Apr 13, 2020 at 11:14:11PM +0530, Kushal R. wrote: > All abuse reports that we receive are dealt within 48 business > hours. As far as that tweet is concerned, it???s pending for 16 days > because they have been blocked from sending us any emails due to the sheer > amount of emails they started sending and then our live support chats. > There's a lot to unpack here, both for you and for RiskIQ. Let's start with you. Your home page says that you host over 100,000 web sites. Your home page says that you have over 10,000 customers. Your home page says that you have 24x7x365 support. (Which is wrong, by the way. It's either 24x7 or 24x365 or maybe 24x7x52 depending on what you're trying to express. There is no such thing as 24x7x365. But let's press on:) Given all that, why don't you have have a 24-hour abuse desk that is empowered to act immediately on reports? Do you not understand that -- as Suresh has pointed out -- the lifetime of many abusive activities is measured in hours and that a 48-hour turnaround is far too slow to be effective? Your "about" page says that you're a leading web hosting company. Alright then: *lead*. Show us that you're one of the best at this. Be one of the operations that we can point to and say "this is how it's supposed to be done". Because right now you're the opposite of that. Also: don't use abuse.support@. Use abuse@, per RFC 2142. There is zero reason not to go along with the standard. If you want to alias it internally fine, but at least get this rudimentary thing right. *This is why we have standards*. By the way: did you know that there are multiple COVID-19 scammers who have set up shop on your service in past few weeks? I'm very curious as to why a "leading web hosting company" would allow such a thing to happen, given that much of it's trivial to prevent. And now: RiskIQ, it's your turn. If an operation has exhibited the competence to read and implement RFC 2142, and thus has a working abuse@ address that goes to some combination of people and automation that deals with abuse reports, then that's the one you should be using. If it has a security@ address then that's appropriate for those kinds of events. And while there are obviously cases where it's appropriate to send to both, it's never appropriate to send this stuff to role accounts like sales@ or info@ or anything like that. So: knock it off. What about operations that haven't done that? Okay, that's where you look up their registered contacts. There is of course no reason for addresses like abuse.support@ when abuse@ will do perfectly fine for everyone on this planet but if that's what has to be done, then (a) use it and (b) try to convince them to use abuse@ like competent people who have read RFC 2142 do. We'll all be happy if you succeed. Sending reports repeatedly may make you feel better by venting your frustration, but it won't solve the problem. (Now, if new information arrives about a report you've already filed, then a supplemental message is appropriate.) Bombarding people either means you're (a) annoying people who were already doing something or (b) annoying people who were never going to do anything anyway. So knock that off too. Bugging people in live support chats is probably equally pointless. So if you're doing that: stop. (Actually: given my experience over the past few decades "live support chats" are pretty much pointless, but that's a whole 'nother problem and if I have to deliver *that* rant, I'll need scotch before noon. So again, pressing on:) As to naming-and-shaming on the web or Twitter or wherever: sure, if you want. But if you're going to do that then it's probably worth doing a bit more formally, a la Spamhaus, with a web page that has a unique URL and supporting evidence and an explanation and so on. (Do keep in mind that operations like Twitter are transient and thus not a good choice if you're trying to create a permanent record.) ---rsk
Re: Constant Abuse Reports / Borderline Spamming from RiskIQ
On Mon, Apr 13, 2020 at 07:55:37PM +0530, Kushal R. wrote: > We understand these reports and deal with them as per our policies and > timelines but this constant spamming by them from various channels is not > appreciated. Quoting from: https://twitter.com/RiskIQ_IRT/status/1249696689985740800 which is dated 9:15 AM 4/13/2020: 5 #phishing URLs on admin12.find-textbook[.]com were reported to @Host4Geeks (Walnut, CA) from as far back as 16 days ago, and they are all STILL active 16 days is unacceptable. If you can't do better than that -- MUCH better -- then shut down your entire operation today as it's unworthy of being any part of the Internet community. ---rsk
Re: [EXT] Shining a light on ambulance chasers - Noction
On Wed, Mar 25, 2020 at 08:13:58PM -0400, Chuck Anderson wrote: > Let's start a public blacklist, sort of like a RBL reputation block > list or 800notes.com, but for companies to "never to do business with" > for spamming. So it shall be done. Nominations accepted at: nanog-spamm...@firemountain.net Of course, someone/something is going to harvest that address and self-nominate, but that's a feature, not a bug. Send either references that I can check by going through my archives of this list, i.e. grep'able strings or send messages themselves, preferably quoted with full headers. I'll figure out how to publish it. Note that this isn't a high priority at the moment, so it may be a while before I plow through what's received. ---rsk
Re: free collaborative tools for low BW and losy connections
On Mon, Mar 30, 2020 at 06:30:16AM -0500, Joe Greco wrote: > Actual text traffic has been slowly dying off for years as webforums > have matured and become a better choice of technology for nontechnical > end users on high speed Internet connections. My view is that the move to web forums is a huge downgrade. Mailing lists are vastly superior. Here's a (large) snip from http://www.firemountain.net/why-mailing-lists.html (which I wrote) with a comment from me shoved in the middle. [...] 1. Mailing lists require no special software: anyone with a sensible mail client can participate. Thus they allow you to use *your* software with the user interface of *your* choosing rather than being compelled to learn 687 different web forums with 687 different user interfaces, all of which range from "merely bad" to "hideously bad". 2. Mailing lists are simple: learn a few basic rules of netiquette and a couple of Internet-wide conventions, and one's good to go. Web forums are complicated because all of them are different. In other words, participating in 20 different mailing lists is just about as easy as participating in one; but participating in 20 different web forums is onerous. 3. They impose minimal security risk. 4. They impose minimal privacy risk. Points 3 and 4 stand in stark contrast to the security and privacy risks imposed on users of web forums and "social" media, especially the latter. 5. Mailing lists are bandwidth-friendly -- an increasing concern for people on mobile devices and thus on expensive data plans. Web forums are bandwidth-hungry. 6. Mailing lists interoperate. I can easily forward a message from one list to another one. Or to a person. I can send a message to multiple lists. I can forward a message from a person to this list. And so on. Try doing this with web forum software A on host B with destinations web forum software X and Y on hosts X1 and Y1. Good luck with that. 7. They're asynchronous: you don't have to interact in real time. You can download messages when connected to the Internet, then read them and compose responses when offline. 8. As a result of 7, They work reasonably well even in the presence of multiple outages and severe congestion. Messages may be delayed, but once everything's up again, they'll go through. Web-based forums simply don't work. 9. They're push, not pull, so new content just shows up. Web forums require that you go fishing for it. 10. They scale beautifully. 11. Mailing lists -- when properly run -- are highly resistant to abuse. Web forums, because of their complexity, are highly vulnerable to software security issues as well as spam/phishing and other attacks. [ I'm going to interject a comment here that's not on the web page I'm quoting myself from. There are, of course, counter examples to this. There is a very busy very well-known mailing list that is an absolute cesspool of trivially-blockable spam. Hence the phrase "when properly run", becase when that's done spam incidents should be at the 1 per year level or less. It's not that hard.] 12. They handle threading well. And provided users take a few seconds to edit properly, they handle quoting well. 13. They're portable: lists can be rehosted (different domain, different host) rather easily. 14. They can be freely interconverted -- that is, you can move a list hosted by A using software B on operating system C to host X using software Y on operating system Z. 15. They can be written to media and read from it. This is a very non-trivial task with web forums. 16. The computing resources require to support them are minimal -- CPU, memory, disk, bandwidth, etc. 17. Mailing lists can be uni- or bidirectionally gatewayed to Usenet. (The main Python language mailing list is an example of this.) This can be highly useful. 18. They're easily archivable in a format that is simple and likely to be readable long into the future. Mail archives from 10, 20, even 30 or more years ago are still completely usable. And they take up very little space. (Numerous tools exist for handling Unix "mbox" format: for example, "grepmail" is a highly useful basic search tool. Most search engines include parsers for email, and the task of ingesting mail archives into search engines is very well understood.) 19. You can archive them locally... 20. ...which means you can search them locally with the software of *your* choice. Including when you're offline. And provided you make backups, you'll always have an archive -- even if the original goes away. Web forums don't facilitate this. (Those of who've been around for a while have seen a lot of web-based discussions vanish forever because a host crashed or a domain expired or a company went under or a company was acquired or someone made a mistake or there was a security breach or a government confiscated it.) [...] ---rsk
Re: free collaborative tools for low BW and losy connections
On Wed, Mar 25, 2020 at 05:27:41PM +, Nick Hilliard wrote: > nntp is a non-scalable protocol which broke under its own weight. Threaded > news-readers are a great way of catching up with large mailing lists if > you're prepared to put in the effort to create a bidirectional gateway. But > that's really a statement that mail readers are usually terrible at handling > large threads rather than a statement about nntp as a useful media delivery > protocol. Some mail readers are terrible at that: mutt isn't. And one of the nice things about trn (and I believe slrn, although that's an educated guess, I haven't checked) is that it can save Usenet news articles in Unix mbox format, which means that you can read them with mutt as well. I have trn set up to run via a cron job that executes a script that grabs the appropriate set of newsgroups, spam-filters them, saves what's left to a per-newsgroup mbox file that I can read just like I read this list. Similarly, rss2email saves RSS feeds in Unix mbox format. And one of the *very* nice things about coercing everything into mbox format is that myriad tools existing for sorting, searching, indexing, etc. ---rsk
Re: free collaborative tools for low BW and losy connections
On Wed, Mar 25, 2020 at 09:59:53AM -0600, Grant Taylor via NANOG wrote: > Something that might make you groan even more than NNTP is UUCP. UUCP > doesn't even have the system-to-system (real time) requirement that NNTP > has. It's quite possible to copy UUCP "Bag" files to removable media and > use sneaker net t transfer things. I was remiss not to mention this as well. *Absolutely* UUCP still has its use cases, sneakernetting data among them. It's been a long time since "Never underestimate the bandwidth of a station wagon full of tapes" (Dr. Warren Jackson, Director, UTCS) but it still holds true for certain values of (transport container, storage medium). ---rsk
Re: free collaborative tools for low BW and losy connections
One of the tools that we've had for a very long time but which is often overlooked is NNTP. It's an excellent way to move information around under exactly these circumstances: low bandwidth, lossy connections -- and intermittent connectivity, limited resources, etc. Nearly any laptop/desktop has enough computing capacity to run an NNTP server and depending on the quantity of information being moved around, it's not at all out of the question to do exactly that, so that every laptop/desktop (and thus every person) has their own copy right there, thus enabling them to continue using it in the absence of any connectivity. Also note that bi- or unidirectional NNTP/SMTP gateways are useful. It's not fancy, but anybody who demands fancy at a time like this is an idiot. It *works*, it gets the basics done, and thanks to decades of development/experience, it holds up well under duress. ---rsk
(updated) COVID-19 fast/small resources page
It's here: http://www.firemountain.net/covid19.html There's now a link to Job Snijders' "Internet Operations During Pandemics" PDF, better coverage of mapping/tracking, links to every US state's public health agency, links to Canada and Mexico's CDC-equivalents, etc. I also fixed the character encoding. I may move the commentary at the bottom elsewhere: I'm trying to keep this page very lightweight, which is why there are no graphics, scripting, or anything else. Comments/fixes/additions to me, off-list please. ---rsk
Re: COVID-19 vs. our Networks
On Sat, Mar 21, 2020 at 04:42:51AM +0200, Mark Tinka wrote: > All I'm saying is at the moment, there is no empirical information to > suggest that Netflix will break what's left of the Internet. Nor is > there any empirical information suggesting that singling them out will > help keep it going. My remarks weren't about Netflix or any other particular service. (FWIW, I agree with you on both quoted points about the lack of evidence. Maybe it'll arrive. Maybe it won't.) I was trying to speak, perhaps unsuccessfully, in broader terms about trying to best position ourselves for challenges that we may not see coming despite the aggregated millenia of expertise here. I think at this particular point in time we need to be ready for anything, for a very nebulous and possibly quite surprising value of "anything". ---rsk
Re: COVID-19 vs. our Networks
On Fri, Mar 20, 2020 at 10:00:15AM -0500, Mike Hammett wrote: > Because they're trying to be a responsible Internet citizen instead of just > telling everyone else to bugger off. > > > Perhaps if more entities tried to be responsible instead of entitled, the > Internet wouldn't be as bad as it is? +100. In all the decades that I've been here (on the 'nets), the saddest change I've seen is the lack of responsibility on the part of people who have, by virtue of their positions, been given incredible power. This is the time for those people to step up and (try to) do the right thing. None of us know what's going to be needed. How could we? We could guess, and we *are* guessing, but we don't really know because we're sailing off the edge of the map now. In those circumstances, the virtue of frugality -- a sensible thing at any time -- now becomes a necessity. Every single one of us should be doing whatever we can to prepare for the unknown, and conserving resources is one part of that. "Everything we do before a pandemic will seem alarmist. Everything we do after will seem inadequate." --- Michael Leavitt, former HHS Secretary As I write this, doctors and nurses are working without PPE, risking their own wellbeing to try to save patients. We're not being asked to do anything like that. Hopefully we still have enough left to rise to the comparatively minor challenge in front of us. ---rsk
Re: COVID-19 vs. our Networks
On Wed, Mar 18, 2020 at 03:43:37AM -0600, Keith Medcalf wrote: > So you failed because you did not require the person making the decision > to take responsibility for their decision. That is, your organization > has a severely flawed process wherein the "R" for making the decision is > not the same person as has the "R" for the repercussions. The use of "you/your" here and throughout is misplaced and inappropriate. Also: this not an isolated or unique experience. It's this way pretty much everywhere in the US now. And I can disapprove of it, you can disapprove of it, we can all disapprove of it, but like I said, until money is completely removed from the calculation, this is how it will be. Critiques of process and role and organization and everything else are interesting, maybe even correct -- but will change nothing. ---rsk
Re: COVID-19 vs. our Networks
On Tue, Mar 17, 2020 at 11:35:59AM -0700, Owen DeLong wrote: > Anything in the healthcare vertical that is outside of the medical > providers control/ownership is a result of the medical provider > buying into that model on some level. STOP DOING THAT. (How am I > suddenly reminded of the old adage ???Doctor, doctor, it hurts when I > do this!??) > > I understand how the allure of lower costs and the frustration of ???every > vendor does this, we can???t find one who doesn???t??? plays out. However, > the only way ???every vendor does it??? will continue is if every vendor > continues to be able to make sales without changing. > Fought this battle, lost this battle. Why? Because the people with the authority to make purchasing decisions are not the people who will be on the phone to some vendor's tech support at 3 AM on a Sunday morning, frantically pleading with them to fix a problem because they really need that piece of equipment to work right now. Decisions are no longer based on the greater good or on anticipating worst case scenarios or on maximizing preparedness or anything that we might hope they're based on. They're based, coldly and calculatingly, on money. If you want this to change -- and I sure would like it to change -- then money needs to be entirely removed from that calculation. That is a problem whose solution lies outside the scope of NANOG. Meanwhile, I've updated this: Covid19 http://www.firemountain.net/covid19.html to include some more resources, including CORD-19, which compiles tens of thousands of papers on the virus in one place. I've also included a link to the relevant Folding@Home project -- which could probably use as much CPU as you can throw at it. ---rsk
Re: COVID-19 vs. our Networks
On Tue, Mar 17, 2020 at 08:38:28AM -0700, Mike Bolitho wrote: > Anybody who works in the healthcare vertical will tell you just how > bad medical devices are to work with from an IT perspective. Medical devices are appallingly bad to work with from an IT perspective. They're designed and built to work in idealized environments that don't exist, they make unduly optimistic assumptions, they completely fail to account for hostile actors, and whenever possible they are gratuitously incompatible to ensure vendor lock-in. That's the good news. Here's the bad news: in about 2-3 weeks, when our health care systems are stretched to the breaking point, there will be a window of opportunity for adversaries to maximize the damage. ---rsk
Re: COVID-19 vs. our Networks
On Sat, Mar 14, 2020 at 05:17:01PM -0400, b...@theworld.com wrote: > > On March 14, 2020 at 14:49 r...@gsp.org (Rich Kulawiec) wrote: > > > > 2. Find all the phone chargers, laptop chargers, USB sticks, cables, > > everything. If you're not already obsessive about keeping things > > charged, get that way. > > You're really expecting power interruptions due to the virus (in the > US)? No. I'm guessing that people may be called upon to pack up their gear and work in a place they didn't expect to be working. Or that they may need to improvise solutions in a hurry with whatever they've got handy. So having a bag full of chargers and cables and adapters and everything else just seems like good/easy/cheap preparation. (I think a lot of us already have such a bag, it's just that entropy sets in and the way it was once packed it probably not the way it's packed today.) I'm not really "expecting" anything other than suboptimal conditions. ---rsk
Re: COVID-19 vs. our Networks
On Sat, Mar 14, 2020 at 11:01:48AM -0700, Mike Bolitho wrote: > Third, the trouble we had was a third party service having congestion > issues. This is a tiny sample of what's coming. We're all about to be tested in a major way, and lots of latent problems are about to become real, pressing problems. So: 1. Get some rest. Stock up (judiciously, don't hoard) on supplies including medications, fluids, food, etc. 2. Find all the phone chargers, laptop chargers, USB sticks, cables, everything. If you're not already obsessive about keeping things charged, get that way. 3. Make sure your role addresses are up-to-date and working: postmaster@ webmaster@ security@ abuse@ noc@ and whatever else is appropriate. Make sure that eyeballs are watching everything that comes in there and anticipate that some people -- under stress and anxious -- will send things to the wrong place. Same for your phone contacts. And make sure frontline support personnel have the ability and judgment to rapidly escalate, do not allow urgent needs to get lost in some ticketing system. 4. Make sure your WHOIS contacts on networks and domains are up-to-date and working. Same for your phone contacts. 5. Identify any spare resources that you can lend out. Identify any resources that you can guess will be needed. 6. Everyone who can telecommute should be telecommuting right now. If you need hands on-site, and of course lots of people will, keep those people separated from others. Make sure hands-on people know how to sanitize equipment, tools, etc. 7. Find time in the midst of this for self-care. You can't help anybody if you're exhausted. Take a shower, watch dog videos, do whatever you need to in order to stay functional. Here's a resource page that I threw together with a little help from some epidemiologists. It's short, plain HTML so it should load very fast, and of course because it's short it's probably missing things. Send suggestions to me off-list. http://www.firemountain.net/covid19.html ---rsk
Re: idiot reponse
On Thu, Feb 27, 2020 at 12:25:27AM +, Mark Rousell wrote: > This (or what it appears to be) is happening on an increasing number of > mail lists. It's not many but it's there I don't know who is behind it > or why, but it's an increasing annoyance. There is a partial fix for this, at least for anyone using Mailman to run their lists (e.g., nanog): Set Mailman so that all new subscribers are moderated by default. Either new subscriber X will one day send real content to the list or they won't. If it's the latter, then it is very simple to use Mailman's interface to simultaneously (a) approve the message for distribution and (b) clear their moderation flag. If it's the former, then the message will only be seen by the list-owners and won't bother everyone on the list. [1] This doesn't help with copies that are sent directly to list-members, however. The fix for that is for responsible list owners (a) to be available at the -owner address (per RFC 2142 and decades of best practices) so that they can field problem reports and (b) to use Mailman to (a) unsubscribe the errant address and (b) ban it. I'd also recommend that they (c) publicly announce such actions with an "administrivia" Subject line on-list so that list members can take corresponding actions in their own mail systems. If nanog-owner isn't responding then that's a serious lapse and needs to be corrected immediately. Doing so is a fundamental part of basic mailing list administration. I'd also strongly recommend that list-owners have Mailman configured to notify them of all subscribe/unsubscribe events and/or to require manual list-owner approval for subscriptions. Interposing human beings in the process doesn't solve this problem but it provides the opportunity to detect and quash it early on. ---rsk [1] Note that this is also a partial defense against accounts which are hijacked and turned into bots. Given that -- on most mailing lists and especially on large ones -- the overwhelming majority of subscribers will *never* send any traffic, nothing is lost by doing this. But on the day when an account is hijacked and suddenly starts sending large amounts of traffic, none of of it will get through to the mailing list.
Re: Tell me about AS19111
On Thu, Feb 06, 2020 at 09:08:35AM +0100, Pierfrancesco Caci wrote: > You would sound much more credible if you'd step down the high horse and > stop insulting the very same people you're supposed to work with. You're concerned with policing his tone instead of dealing with the massive security failure -- on the part of *many* of us -- that this represents? If I have something horrible going on with a service/server/network/etc. that I'm responsible for and I don't catch it, then I'm grateful to anyone who reports it -- because they've caught my mistake, which is helpful to me and to everyone impacted by it. I'll worry about my bruised ego later, it won't be the first time. Or the last. ---rsk
Re: FYI - Suspension of Cogent access to ARIN Whois
On Tue, Jan 07, 2020 at 04:54:22PM -0600, Mike Hammett wrote: > That said, if there's a stern warning about Cogent abusing the system, > maybe their customers finding out is a good thing for the overall > community. ;-) And that is what I would suggest: reply to all queries with a notice that explains what is happening, why it's happening, and provides contact information for Cogent executives: preferably their *personal* email addresses and phone numbers. ---rsk
Re: Iran cuts 95% of Internet traffic
And this is why, despite all the disdainful remarks labeling such things as "antiquated", mailing lists and Usenet newsgroups are vastly superior to web sites/message boards/et.al. when it comes to facilitating many-to-many communications between people. Why? Well, there are many reasons, but one of the applicable ones in this use case is that their queues can be written to media, physically transported in/out, and then injected either into an internal or external network seamlessly modulo the time delay. And because the computing resources required to handle this are in any laptop or desktop made in the last decade, probably earlier. If you're trying to get information in/out of a society that is raising network barriers to realtime communication, then you need methods that don't rely on a network and aren't realtime. ---rsk
Re: FCC proposes $10 Million fine for spoofed robocalls
[ Re-sent with proper headers. My apologies for the typo'd previous version. ] On Thu, Dec 19, 2019 at 11:34:48AM -0800, William Herrin wrote: > I don't want to start an arms race with the spam callers, I want to > end it. That means: jump directly to something they can't easily > defeat. It is at this point that I am reminded of the wisdom of former FTC Commissioner Orson Swindle, who was testifying before Congress on the subject of spam when he said "We need a couple of good hangings here." It was true in 2003 (which is I believe when he said it) and it's still true now. Fines, whatever they are, will be evaded and bargained down, companies will be dissolved and reconstituted, money will be laundered, and the problem will persist. ---rsk
[no subject]
Bcc: Subject: Re: FCC proposes $10 Million fine for spoofed robocalls Reply-To: In-Reply-To: On Thu, Dec 19, 2019 at 11:34:48AM -0800, William Herrin wrote: > I don't want to start an arms race with the spam callers, I want to > end it. That means: jump directly to something they can't easily > defeat. It is at this point that I am reminded of the wisdom of former FTC Commissioner Orson Swindle, who was testifying before Congress on the subject of spam when he said "We need a couple of good hangings here." It was true in 2003 (which is I believe when he said it) and it's still true now. Fines, whatever they are, will be evaded and bargained down, companies will be dissolved and reconstituted, money will be laundered, and the problem will persist. ---rsk
We've lost another innovator
- Forwarded message from Russ Allbery - > From: Russ Allbery > Date: Tue, 26 Nov 2019 20:56:23 -0800 > Subject: Brian Kantor has died > > Slashdot reported, via a contributor from the 44Net amateur radio mailing > list, that Brian Kantor died suddenly in his home last week. > > https://tech.slashdot.org/story/19/11/24/0051236/brian-kantor-internet-and-amprnet-pioneer-wb6cyt-dies > > Brian was the co-inventor of NNTP and the co-author of RFC 977, with Phil > Lapsley. I never met him in person, but have had several opportunities to > chat with him electronically, including as recently as last month (via > NNTP and netnews newsgroups, of course). He will be missed. I never met Brian either, but have "known" him electronically for decades. His was a voice I always paid attention to, even when I disagreed with what he had to say. I'm sorry he's gone, and I'll miss him. ---rsk
Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019
On Fri, Nov 08, 2019 at 01:43:41PM -0500, Mark Stevens wrote: > Reading Syniverse's cause of trouble (lame excuse) tells me their data > handling processes are poor and seemingly shady since I do not buy reason > for the trouble. Agreed. So how many other messages have been delayed, lost, forwarded incorrectly, or...sold to third parties? (Note that I'm not saying Syniverse did that last one. What I'm saying is that an operation like this inevitably affords plenty of opportunities for employees to engage in a little freelance capitalism of their own or for third parties to simply help themselves.) ---rsk
Re: Russian government???s disconnection test
On Sat, Nov 02, 2019 at 09:18:36AM -0700, Mike Bolitho wrote: > The very fact that there are > AWS/Azure/Google Cloud data centers located around the globe makes anything > hosted there even more resilient, not less (and for the most part, I still > prefer on prem DC so I'm not even pushing "To the cloud!"). No, this fact makes everything far less resilient, because it means "one stop shopping" for attackers. It also makes the available attacker budget much greater, since the ROI increases every time more resources are concentrated in fewer places. ---rsk
Re: Unable to email anyone from my primary domain name; thanks Google Mail and G Suite.
[ Again, just commenting on one point. ] On Thu, Oct 24, 2019 at 01:21:12PM -0700, Mark Milhollan wrote: > My experience says that: their system has learned that your system(s) > continued to send messages that their user (yes you, but they don't know > that) did not want, i.e., you left it marked as SPAM or deleted it without > reading the message, or at least not enough was noted as not SPAM *and* read > (aka displayed, and not for half a second either) so as to influence their > AI, which makes mistakes and will never correct them if not fed correcting > info. [ Note that the proper term is "spam": it's not an acronym and should never be fully capitalized. ] It is a worst practice in mail systems engineering to allow input from putative users into any decision-making process *absent* manual review of each piece of data by very clueful humans, e.g., curated abuse reports. Consider: either that input is coming from the person who owns the account or it's not. If it's coming from a person, then there is an extremely high probability that it's coming from someone who doesn't have the slightest idea what they're doing. (If users, en masse, knew what they were doing and were even modestly diligent, then spam and phishing would not be serious problems. But they are; they flourish because users are careless, lazy, stupid, etc. They've been proving this daily by the hundreds of millions for decades.) Systems which do this are built on the laugable assumption that the aggregate opinions of a hundred million idiots are somehow magically more valid than the opinions of one. If it's not coming from a person, then it's coming from the new owner of the account. That new owner is hostile by definition, so allowing input from them is an even worse idea than allowing input from users, who may only be hostile most of the time. (Keep in mind that email accounts are compromised by the billions -- e.g., Yahoo -- and that systems are likewise so. And of course anyone who has compromised a system can avail themselves of any/all email credentials stored on or traversing that system. Those who doubt the scale on which both are happening are invited to peruse the darknet marketplace of their choice and observe that accounts/systems are for sale in bulk quantities from a wide variety of sellers.) ---rsk
Re: Unable to email anyone from my primary domain name; thanks Google Mail and G Suite.
[ I'm just going to focus on one point. ] On Wed, Oct 23, 2019 at 06:18:46PM -0600, Constantine A. Murenin wrote: > it is revealed that Postmaster Tools cannot tell me anything at all, with > all tabs and screens being 100% blank, allegedly because I'm not actually a > mass email sender (I don't send hundreds of emails a day or whatnot), and > they're too afraid that I'll figure out why my mail doesn't actually go > through, instead of signing up for G Suite. There is a persistent mythos -- a worst practice, actually -- among many operations that obfuscating the reasons why messages are rejected is useful. This is wrong. Consider: either the sender is benign (as in this case) or they are not. If they are benign, then denying the information necessary to understand and solve the problem helps no one. It's also counter to the decades of cooperation and mutual assistance that have built the Internet. If they're not benign, then either they don't care enough to acquire this information or they do. If they don't care, then providing the information doesn't hurt, because it'll be ignored anyway. If they do care, then they WILL get it, whether by conducting research or by breaching security or by the simpler/cheaper path of paying someone on the inside off. (If you're going to tell me that everyone who works AT Google is working FOR Google, then I'm going to tell you that you're naive and clueless. If I were in the large-scale spam-for-hire business, I'd have already planted my own people there a long time ago.) Best practice when rejecting mail traffic is to (a) provide at least a semblance of a reason why and (b) a remediation path that includes escalation to real live human beings. All mail systems (except for the edge case of those which accept everything) make rejection errors and that must be accounted for in design and operation. ---rsk
Is anybody else getting spam from cytranet.com?
I'm guessing -- because spammer Ben Reynolds (breyno...@cytranet.com) wrote to me about voice/data services -- that it's possible they've been scraping addresses from here. ---rsk
Re: Update to BCP-38?
On Tue, Oct 08, 2019 at 10:03:16AM -0700, William Herrin wrote: > Limiting the server banner so it doesn't tell an adversary the exact > OS-specific binary you're using has a near-zero cost and forces an > adversary to expend more effort searching for a vulnerability. Why would they bother performing that search? Why not use their botnets to throw every exploit they have at a service and see if anything works? That's easier and cheaper and faster than being selective. It also -- if they have happen to have a working exploit -- blows right past (announced) versions, whether real, fake, or elided. Brute force is cheap, analysis is expensive. Case in point: every mail server I have eyeballs on was probed by attackers trying to exploit the recent exim vulnerability -- no matter what MTA they're running, no matter that they all announce the MTA and version, no matter anything. I doubt I'm alone in observing this. Even a diligent, capable attacker -- someone who is willing to invest the time and effort to ascertain what's running which service, down to the version -- could save themselves some homework by launching an attack like the one in the first paragraph above, examining the results, and using those to greatly reduce their search space. It's easy, it's cheap, it's fast, it's automated, and it yields no clues as to where the followup (version-specific) attack is going to come from. ---rsk
Re: Update to BCP-38?
On Tue, Oct 08, 2019 at 01:35:16PM +0100, Mike Meredith via NANOG wrote: > You've ignored step 1 - identifying critical information that needs > protecting. It makes sense to protect information that needs protecting and > don't lose sleep over information that doesn't need protecting. Not many of > us are planning an invasion of a Nazi-infected Europe any time soon. We are heading toward a restatement of Kerckhoff's principle/Shannon's maxim, the latter of which can be paraphrased as "design systems assuming that your adversary will know as much about them as you do". Not that I'm advocating publishing all internal design documents, but systems whose security is predicated on the secrecy of those are brittle and likely to be badly compromised. Better to assume that enemies know or can find out everything and design/build accordingly. ---rsk
Re: Automated Abuse Reports
On Mon, Oct 07, 2019 at 05:28:08PM -0700, Matt Corallo wrote: > Is it time to have ARIN add a ???abuse contact available only after > captcha??? option? No. Captchas are a worst practice and should never be used. ---rsk
Re: "Using Cloud Resources to Dramatically Improve Internet Routing"
On Mon, Oct 07, 2019 at 04:42:11PM +0200, Stephane Bortzmeyer wrote: > Otherwise, an impressive amount of WTF. My favorite: "while > communication by servers ___on the ground___ might take hundreds of > milliseconds, in the cloud the same operation may take only one > millisecond from one machine to another" My favorite: "The researchers expect their cloud-based system will be more secure than the Internet is today [...]" Apparently they're blissfully unaware that there is no such thing as "cloud security". ---rsk
Re: Corporate Identity Theft: Azuki, LLC -- AS13389, 216.179.128.0/17
On Mon, Aug 12, 2019 at 04:11:00PM -0400, Ross Tajvar wrote: > Seems like submitting a fraud request to ARIN is more effective than > writing a novel and sending it to NANOG, and doesn't require the latter... But if he didn't fully document his assertion(s), then he would be faced with a plethora of replies decrying the lack of substantiating evidence. Better to lay the case out in detail so that everyone can see the work and so that anyone who cares to can check it for themselves. And -- given Ron's long history of thorough documentation -- there are some of us who are willing to take his word for it and make operational decisions based on what he reports, independent of what ARIN decides to do or not do, or when it decides to do it. ---rsk
Re: User Unknown (WAS: really amazon?)
On Sun, Aug 04, 2019 at 12:12:48AM -0700, Stephen Satchell wrote: > "The rules" have been around for years, and are codified in the RFCs > that are widely published and available to all at zero cost. (That > wasn't always true, as it wasn't until the DDN Protocol Handbook volumes > were published in 1985 that the RFCs were available to everyone. I seem > to recall there was an FTP site that provided the RFC documents before > that, but my memory is hazy on that.) IIRC, the CSnet CIC provided an RFC-by-mail service in the mid to late 1980's. It allowed anyone to request any RFC by number, e.g., sending it "rfc123" would result in a response containing that RFC. I also share your recollection of an earlier FTP site but a few minutes of checking old documents hasn't turned up its name and it's fallen out of long-term memory, at least for the moment. > During my career as a web server admin, mail admin, and network admin, I > followed "the rules" strictly. As the main abuse contact during my > time at a web hosting company, my postmaster@ and abuse@ contact > addresses were according to Hoyle, and published with the company ASN, > netblock, and domain registration records. I've done the same -- imperfectly, to be sure, I've certainly tried. Half my grump with Amazon here is that they have, for all practical purposes, unlimited money and unlimited personnel. They should be the go-to example for How To Do It Right. They should be the model (or one of the models) that we're all trying to emulate, the gold standard that we can all point to. But they're not. The other half of my grump is that they're enormous, and therefore capable of inflicting enormous damage. The larger an operation, the more critical it is that abuse/security/et.al. be fully supported, highly responsive, empowered to act decisively, etc. But they're not. And I have yet to see anyone from Amazon (a) admit this and (b) ask for help fixing it. ---rsk
Re: really amazon?
On Thu, Aug 01, 2019 at 12:54:07AM +0300, Scott Christopher wrote: > Rich Kulawiec wrote: > > > On Wed, Jul 31, 2019 at 11:13:48PM +0300, Scott Christopher wrote: > > > Because it will get spammed if publicly listed in WHOIS. > > > > Yes. It will. Are you telling us that Amazon, with its enormous financial > > and personnel resources, doesn't have ANYBODY on staff who knows how to > > properly manage an abuse@ address -- part of which includes dealing > > with that exact problem? > > They do, but it's just time-consuming and inefficient. You can't spam-filter > the content of abuse@ obviously. Actually, yes, you can -- but probably not in the way you're thinking, because if you do it *that* way you will break [some of the] required functionality. > But in addition to spam, random (read: non-technical) people will send > complaints outside of the usual purview of spam, network abuse, DMCA, > etc. They find some FAQ on the web telling them to determine the PoC on > whois.domaintools.com and then they start firing crap. This is not my first day on the job. I'm aware of what shows up at role addresses. However, handling the problems you enumerate here is a straightforward (albeit occasionally tedious) matter that any operations engineer above entry-level should be able to handle. Doubly so because people like me have done them the favor of writing about it (here and elsewhere), so they can use our experience without needing to repeat our numerous mistakes. > I prefer openness and transparency and the general spirit of WHOIS but, in > practice, > you really do need the limit the PoC information to a trusted group of > insiders. First, there's no such thing as a trusted group of insiders. Second, even if such a group existed, limiting PoC information to them is impossible. Think about it. Third, besides WHOIS PoC, RFC 2142 (and decades of best practices) specify abuse@, postmaster@, etc. My expectation is that anyone equipped with baseline competence will be fully prepared to handle traffic to those addresses (as applicable) effectively at whatever scale their operation requires. ---rsk
Re: really amazon?
On Wed, Jul 31, 2019 at 11:13:48PM +0300, Scott Christopher wrote: > Because it will get spammed if publicly listed in WHOIS. Yes. It will. Are you telling us that Amazon, with its enormous financial and personnel resources, doesn't have ANYBODY on staff who knows how to properly manage an abuse@ address -- part of which includes dealing with that exact problem? ---rsk
Re: really amazon?
Yes, this is egregious, but on the other hand even when the abuse reporting mechanisms are working my experience has been that they emit no response (other than -- maybe -- boilerplate) and take no action, so it's not terribly surprising. ---rsk
Re: netstat -s
On Wed, Jul 17, 2019 at 05:54:49PM -0700, Randy Bush wrote: > do folk use `netstat -s` to help diagnose on routers/switches? I (mostly) use it on firewalls, but yes, it's something I turn to fairly often (along with other incantations of netstat, plus lsof and other tools). ---rsk
Re: Twitter security team?
On Thu, Jul 18, 2019 at 12:45:25PM -0600, Ken Gilmour wrote: > I have evidence and can't contact anyone due to > the lack of an appropriate form and the fact that the security@ email > address doesn't work. Of course I'm not surprised that the ignorant newbies running Twitter can't manage this: who wouldn't be, given their atrocious track record? But for everyone else: [ engage soapbox ] RFC 2142 was published in 1997, and most of the role addresses it specifies were in relatively common use prior to that. Yet -- nearly every day -- this list carries traffic from someone attempting to help/warn/etc. some allegedly professional operation that has its fingers firmly lodged in its ears in a desperate attempt to prevent basic communication and expects people who are already trying to provide them with free consulting services to jump through various annoying hoops in order to do so. RTFRFC, folks, and implement it. It's operations 101. It's something you should have done in the first hour of the first day, before you turned on the rest of your stuff. It's not hard. And when a day like this comes for your operation, which it will, it may save you considerable pain, time, and/or money. [ soapbox off - for now ;) ] ---rsk
Re: SHAKEN/STIR Robocall Summit - July 11 2019 at FCC
On Mon, Jul 08, 2019 at 06:54:51PM -0600, Keith Medcalf wrote: > This is because DKIM was a solution to a problem that did not exist. This is correct. We have always known the IP address of the connecting MTA, therefore we have always known the network it resides in, therefore we have always known who is responsible for what transits that connection. Worse, this (poorly) attempts to wallpaper over the problems of compromised systems/accounts. Do recall that not long ago we learned that EVERY Yahoo account was compromised. Anyone who thinks that Microsoft or Google or Comcast or anyone else are doing any better is naive: it's not a question of whether they've also suffered mass compromises, only a question of how many and when they'll publicly admit it. This isn't surprising. The real underlying problems here are tough and expensive, thus it's far easire to do (nearly) meaningless feel-good work, declare the problems solved, and engage in a round of self-congratulation. It *appears*, and that's a preliminary assessment on my part, that SHAKEN/STIR is following this same track. ---rsk
Re: CloudFlare issues?
On Mon, Jun 24, 2019 at 09:39:13PM -0400, Ross Tajvar wrote: > A technical one - see below from CF's blog post: > "It is unfortunate that while we tried both e-mail and phone calls to reach > out to Verizon, at the time of writing this article (over 8 hours after the > incident), we have not heard back from them, nor are we aware of them > taking action to resolve the issue." Which is why an operation the size of Verizon should be able to manage the trivial task of monitoring its RFC 2142 role addresses 24x7 with a response time measured in minutes. And not just Verizon: every large operation should be doing the same. There is no excuse for failure to implement this rudimentary operational practice. [ And let me add that a very good way to deal with mail sent to those addresses is to use procmail to pre-sort based on who it's from. Every time a message is received from a new source, a new procmail rule should be added to classify it appropriately. Over time, this makes it very easy to identify traffic from clueful people vs. traffic from idiots, and thus to readily discern what needs to be triaged first. ] ---rsk
Re: Russian Anal Probing + Malware
On Fri, Jun 21, 2019 at 05:13:35PM -0700, Ronald F. Guilmette wrote: > Is there anybody on this list who keeps firewall logs and who > DOESN'T have numerous hits recorded therein from one or more > of the following IP addresses? Well, I *did*, but having noticed their activities and grown tired of them, I now just drop their traffic on the floor (and log it). They are one of several operations that I've noticed who have taken it upon themselves to poke at open (and closed) ports without bothering to ask. Assuming for a moment the most charitable interpretation of their collective actions -- that they are earnest researching problems with the intention of helping to solve them -- this is still highly problematic for two reasons: 1. They didn't ask permission. 2. Whether they realize it or not, they're building a target. When, not if, their results database(s) are compromised, they will have furnished the attackers with a comprehensive target list, painstakingly gathered at no cost to them and thoughtfully annotated with whatever metadata has been collected. ---rsk
Re: Google Post Master experiences
This is probably best asked on the mailop list (where some Google personnel hang out): subscribe via mailop-requ...@mailop.org. ---rsk
Re: PSA: change your fedex.com account logins
On Fri, May 31, 2019 at 01:17:19PM +, Richard wrote: > When I have looked into this type of issue for my unique addressing > some did trace back to back-end db hacks (e.g., adobe), but I found > that the most likely culprit was the 3rd-party bulk mailer that > handled the organization's marketing mail. It could be a non-zeroed > disk thrown into the trash or an inside job, but it almost always > traced back to one or two bulk mailing companies. FYI, I've been running numerous experiments in this area for many years using unique non-guessable non-typo'able addresses. Explaining the results in full would take many pages, so let me summarize: 3rd party bulk mailers leak like sieves. "How?" remains an open question: could be that they're selling, could be that they have security issues, could be that insiders are selling on their own, could be any number of things: it's really not possible to say. But they are unquestionably leaking. This is hardly surprising: many of them are spammers-for-hire, many of them use invasive tracking/spyware, and none of them actually care in the slightest about privacy or security -- after all, it's not *their* data, why should they? Which are some of the many reasons that outsourcing your mailing lists is a terrible idea, doubly so when it's quite easy to run your own with Mailman (or equivalent). ---rsk
Re: Spamming of NANOG list members
On Fri, May 24, 2019 at 06:34:25PM +0300, Scott Christopher wrote: > https://marc.info/?l=nanog&r=1&w=2 and https://lists.gt.net/nanog/ > mangle email addresses in the headers but do nothing about email addresses > that are quoted / attributed in the body. There is zero, as in 0.0, point in mangling/obfuscating/etc. email addresses in forlon and misguided and ultimately futile attempts to keep spammers from getting their hands on them. I wrote about this extensively a few years ago so please let me cite myself in these two messages [1]: http://www.firemountain.net/pipermail/novalug/2014-July/041213.html http://www.firemountain.net/pipermail/novalug/2014-August/041230.html On the other hand, there are a lot of reasons NOT to mangle/obfuscate/etc. email addresses, including the use of archives by people who come along later and are trying to track down authors of messages of interest. ---rsk [1] As long as those are, there's still more: as one thought experiment, consider how many of the addresses on this very list can be correctly deduced by using simple constructions based on real names. By example, let's suppose John Smith at example.net is on this list. We could readily guess: j...@example.net sm...@example.net johnsm...@example.net john-sm...@example.net john.sm...@example.net jsm...@example.net j.sm...@example.net smi...@example.net smit...@example.net and similar variations, and if you compare that to the results of egrep "^From: " nanog | sort -u you'll quickly see that a very simple script could come up with roughly half the addresses on this list immediately. One of the implications of this, given the widespread adoption of uniform algorithmic generation of email addresses by much of the corporate and government and nonprofit &etc. worlds, is that an attacker who has very little knowledge of the corpus of valid email addresses at any such entity can make a first-order pass at enumerating them by combining a script such as the one I posited above with lists of the 1000 most common first and last names in the appropriate locale. Of course if the attacker has even a small sample of known-valid addresses, then it's not necessary to use the myriad variations that such a script would generate, only the one that appears to be in use at the target.
Re: Spamming of NANOG list members
On Fri, May 24, 2019 at 08:17:31AM -0700, Brian Kantor wrote: > Anne, the way that such addresses are often harvested is that one of > the spammers (or his agent) becomes a member of the list and simply > records the addresses of persons posting to the list. They then > get spammed. I rather suspect that's exactly what's happening here. I've gotten three, but a colleague who is subscribed but has never posted has gotten zero, despite sharing the same email infrastructure and thus precisely the same configuration. rsk
Re: FCC Hurricane Michael after-action report
On Mon, May 13, 2019 at 11:48:02PM -0500, frnk...@iname.com wrote: > One of my takeaways from that article was that burying fiber underground > could likely have avoided many/most of these fiber cuts, though I???m > not familiar enough with the terrain to know how feasible that is. I suspect that may not be possible in (parts of) Florida. However, even in places where it's possible, fiber installation is sometimes miserably executed. Like my neighborhood. A couple of years ago, Verizon decided to finally bring FIOS in. They put in the appropriate calls to utility services, who dutifully marked all the existing power/cable/gas/etc. lines and then their contractors (or sub-sub-contractors) showed up. The principle outcome of their efforts quickly became clear, as one Comcast cable line after another was severed. Not a handful, not even dozens: well over a hundred. They managed to cut mine in three places, which was truly impressive. (Thanks for the extended outage, Verizon.) After this had gone on for a month, Comcast caught on and took the expedient route of just rolling a truck every morning. They'd park at the end of the road and just wait for the service calls that they knew were coming. Of course Comcast's lines were not the only victims of this incompetence and negligence. Amusingly, sometimes Verizon had to send its own repair crews for their copper lines. There's a lot more but let me skip to the end result. After inflicting months of outages on everyone, after tearing up lots of lawns, after all of this, many of the fiber conduits that are allegedly underground: aren't. ---rsk
Re: EXERCISE: 2019 IAA Planetary Defence Conference - Day 5 Scenario
On Wed, May 08, 2019 at 10:11:10AM -0400, Sean Donelan wrote: > Many exercise designers could use help coming up with useful Internet > disaster sub-plots. Bad enough to inject stress into the exercise, but not > extinction. > > All ISP tech support agents are infected, and become brain eating zombies. We call that "Tuesday". ---rsk p.s. On a more serious note, disaster exercises that include partial failures of emergency response infrastructure are often quite challenging. As I write this, the IT infrastructure of Baltimore is down due to a ransomware attack. As a consequence, while 911 is functional, fire department computers are down. If a significant event requiring BCFD happened tonight, it would be challenging for them to coordinate a large-scale response.
Re: Packetstream - how does this not violate just about every provider's ToS?
On Fri, Apr 26, 2019 at 06:31:08PM -0700, William Herrin wrote: > On Fri, Apr 26, 2019 at 6:06 PM John Levine wrote: > > > I assumed that something this sleazy would be offshore, but their > > terms of service say they're in Los Angeles. > > > > They tricked you. [snip] Also, unless I'm misreading their site, they expect users to download/run an application program of unknown provenance and function, from an operation that has gone to great lengths to conceal its location and principals. What could possibly go wrong? ---rsk
Re: Comcast storing WiFi passwords in cleartext?
On Fri, Apr 26, 2019 at 07:06:40PM +0300, T??ma Gavrichenkov wrote: > Also, I've seen people who use the same password (sometimes with few easily > reversible modifications) for virtually all the purposes, from the WiFi > router up to their e-mail and banking accounts. This is one of the many risks incurred here: password re-use is amazingly common (sometimes, as you note, with a few easily reversible modifications). Accruing a database full of these means building a target, and the bigger it is, the more valuable target it will become. Also, given that this is a public mailing list, lots of people who didn't know the target existed last week could certainly know it now. I hear all the arguments in favor of convenience but it's worth noting that making things convenient for support ops in this fashion has the unpleasant side effect of making them convenient for attackers. ---rsk
Re: Comcast storing WiFi passwords in cleartext?
On Wed, Apr 24, 2019 at 03:13:33PM +, Benjamin Sisco wrote: > The bigger concern should be the cleartext portion of the subject. Yes, and the availability of all this to anyone who hacks Comcast customer support. ---rsk
P2P [was: Special Counsel Office report web site]
On Thu, Apr 18, 2019 at 12:56:03PM +, Kain, Rebecca (.) wrote: > I can???t believe p2p isn???t used more, even inside companies. It does have > legit uses It does, and some of the use cases for it are quite compelling. However, there is often deep mistrust associated with it: years of propaganda from the copyright lobby have fostered the impression that it is inherently malicious. That can be very difficult to overcome: it's in the same class of mythos as "all ICMP traffic is bad", and well, lots of us have spent lots of time over lots of years trying to get past that one. Getting P2P accepted looks like a much bigger hill to climb. ---rsk
Re: Special Counsel Office report web site
On Wed, Apr 17, 2019 at 09:02:52PM -0400, Sean Donelan wrote: > The Special Counsel's report is expected to be posted [...] Not quite. A *version* of the report that has been redacted by the President's hand-picked obedient lackey will be posted. I suspect that the full report will find its way to us via other means. ---rsk
Re: Contacts wanted: OVH, DigitalOcean, and Microsoft (Deutschland)
On Mon, Mar 18, 2019 at 05:02:38PM -0700, Ronald F. Guilmette wrote: > I generated the following survey, on the fly, last night, > based on a simple reverse DNS scan of the evidently relevant addrdess > ranges: > > https://pastebin.com/raw/WtM0Y5yC > > As anyone who isn't as blind as a bat can easily see, there's a bit of a > pattern here. I finally found time to check this out. And I have to ask: how in the heck did anybody accept this operation as a customer? Because it's obvious on inspection -- of the information in that paste -- that they're abusers. Let me 'splain. First, domains in certain TLDs should be considered as -- at best -- dubious until proven otherwise, because those TLDs are well-known as abuse magnets. Every domain in this sample falls in that category. Anyone making mass use of domains in those TLDs is up to something abusive. Second, anyone making mass requests for PTR records for random subdomains is up to something abusive. Third, anyone mass-registering domains whose names are permutations of each other is up to something abusive. (I'm not talking about someone registering a couple of domains that are plausible typos of a primary one or engaging in defensive registrations across a few TLDs. Look at the list, this is obviously quite different from those cases.) Fourth, anyone mass-registering domains whose names are intended to be typo'd and/or misread is up to something abusive. Anybody doing all of the above is not only up to something abusive, but they're standing on a rooftop screaming it through a bullhorn. The word "mass" is key throughout not only because it is a highly reliable indicator of ensuing abuse but because its nature makes detecting this up front quite easy. Once I got to it, it took me less than a minute of scanning that list to determine that there is absolutely no way I would accept this operation as a customer. I recognize that not everyone everyone has my experience in this area, but surely every operation should have someone equipped with modest experience and and a skeptical eye who screens new customers, and, at *minimum*, puts them on hold while some due diligence takes place. It's much easier (and cheaper) to refuse service to operations like this than to deal with the fallout that will inevitably ensue. It's also much better for the rest of us. So: how did these people ever get in the door? ---rsk
Re: Contacts wanted: OVH, DigitalOcean, and Microsoft (Deutschland)
On Tue, Mar 19, 2019 at 09:17:23AM -0700, Eric Kuhnke wrote: > Absolutely unrelated to Ronald's original post, but it's ironic that the > abuse@ address is itself heavily "abused", by commercial copyright > enforcement companies which think it's a catch-all address for things which > are not operationally related to the health of a network [snip] I've seen this movie and have implemented various mitigation approaches to it -- none of which constitute a "solution" but all of which help. 1. Block the addresses originating this traffic. There's no need for staff/processes on the receiving end to put up with spam. (If it's UBE, then it's spam -- by definition. The content and intention are irrelevant.) 2. Use procmail to redirect it where it needs to go. 3. Set up (non-public) Mailman-operated mailing lists for each role account and use the moderation queue on those as a throttling tool. (This works best in conjunction with (2). Let procmail do some of the heavy/straightforward lifting and sort the rest out later.) This also makes it easy to archive everything by subscribing an address that's an append-only mailbox. 4. Funnel the output of (2) and/or (3) into one of the many ticketing systems with priority assigned based on the characteristics of the senders as observed over time. ---rsk
Re: Contacts wanted: OVH, DigitalOcean, and Microsoft (Deutschland)
On Tue, Mar 19, 2019 at 09:23:34AM -0400, Jeff McAdams wrote: > We would prefer, but don't require, that you use the web form because that > is integrated into the workflow of the groups that respond to those > reports. Why isn't abuse@ integrated into the workflow? It darn well should be, (a) given that RFC 2142 has been "on the books" for 22 years and (b) given that methods for handling incoming abuse (or bug, or outage, or other) reports via email to role accounts are numerous and reliable. To be clear: if you want to offer a web form in addition to an abuse@ address (or a security@ address, or a postmaster@ address) that's fine. But web forms are a markedly inferior means of communication and are clearly not a substitute for well-known/standardized role addresses that route to the appropriate people/processes. ---rsk
Re: Should Netflix and Hulu give you emergency alerts?
> Just wait until your connected home speakers, smart smoke detector, smart > refrigerator, smart tv, cell phone, IP streaming box, satellite receiver, > cable box, home security panel and your Fitbit all go off warning you > of the cancellation of an Amber alert at 1:30am, because the good folks > at AlertReady.Ca and Pelmorex think that everything needs to go out at > highest precedence, because, well, think of the children! and > And thus, in the first week the system was alive, alarm fatigue set in, the > government confirmed that it cannot be trusted, and I revoked their > privilege to use my personal devices for stuff I don't want. This is why the service(s) should use confirmed opt-in on a per-device basis and offer sufficient granularity that alerts are only sent to the people who need/want them on the devices they need/want them on. To fabricate some examples: Tornado alert: why, yes, I live in southwestern Kansas so definitely send those to my home device. Silver alert: nope. I live in Queens and don't go out much and don't drive, so I won't be on the road to see the license plate you're trying to tell me about. Never send me these. Coastal flooding alert: maybe. I live 130 miles from the coast and at 550 feet, so any coastal flooding even that would affect me will be beyond catastrophic. So don't send that to my home device. However, I'm vacationing at the beach right now so send it to my mobile. This will eliminate some of the alarm fatigue as well as reducing the transmission requirements. It's just a rather straightforward exercise in database management. ---rsk
Re: Should Netflix and Hulu give you emergency alerts?
A side point: On Sat, Mar 09, 2019 at 02:04:33PM -0500, Sean Donelan wrote: > Wireless Emergency Alerts (WEA), i.e., mobile phone alerts, are less than 10 > years old. And mostly on the high-end expensive cell phones and the most > expensive carriers. People on NANOG may use mostly expensive smartphones, > but not everyone can afford smartphones. That's an excellent point that's often lost among people who work in our industry. Not everyone is so wealthy as to afford the luxury of a smartphone. And not everyone can use one. And not everyone wants one. The first two items also happen to describe the people who are most vulnerable to disasters and have the most difficulty getting assistance recovering from them: the poor and the elderly. ---rsk
