Bron Gondwana via Mailman-Developers writes: > > Also, the last I looked the draft had expired about 6 months > > previously, IIRC. So didn't seem urgent. > > You're not the first person to have this misconception, so we're > clearly doing something wrong!
Not in my case, I just haven't chased it down. It's likely I was pointed at a specific draft and just assumed it was most recent. > The IETF DKIM working group adopted a different document, which has > been progressing with frequent updates: > > https://datatracker.ietf.org/doc/draft-ietf-dkim-dkim2-spec/ That's not the draft I was looking at, it was the draft-clayton spec. AIUI that's just how this works, people just publish drafts under a proponent's name until the IETF decides to assign a manager or whatever they call it. If you're not following the list, probably you miss the change. > That's true, but I would imaging that most mailing lists aren't > auto-trusted by all recipients. Not my experience. Mailing lists seem to get a broad pass as long as they don't have a lot of addressees at one provider. I could tell stories about the definition of "a lot" .... > They'll often have people with Gmail or Yahoo accounts on them, > both of whom are planning to start enforcing DKIM2 at some point Yeah, sadly Gmail has decided to raise a finger to mailing lists over the last decade. Yahoo, well, people who leak a billion contact lists to spammers are desperate. > (and Fastmail as well, but I don't hold any unrealistic > expectations about our relative clout compared to those two). Ah, but you are not a surveillance capitalist first, and an email provider second. You have a lot more clout with me than they do when it comes to changing Mailman (and more important whether I badmouth you on our lists for making us change ;-). > The issue is the other side - a message that IS spam, you can > reliably undo the changes that the mailing list did, and check > whether the mailing list is a real mailing list that happened to > have spam sent to it, or a fake mailing list which is really just > pretending it got the original message and then sending out spam > instead. OK, since that makes business sense to you, good enough for me. > My opinion is "without undo capability, you have to build a trust > database of everything which claims to be a mailing list in the > entire world -- with undo capability, you just have to check a spam > to see where the badness was introduced". The surveillance capitalists are going to ingest everything anyway. But I see the point for Fastmail. > Here's an example Message-Instance from a list which I run at > mailman.dkim2.com: [JSON and some text elided] > The content-type needs to get the old boundary, and the subject > gets the old un-prefixed value. Yeah, I figured that out. That's not my understanding of how MIME is supposed to work, but it's just a bit of bookkeeping for intermediaries. > DKIM2 doesn't care about exact whitespace in headers - it uses the > relaxed canonicalisation, so you can fold multiple spaces to a > single space, or expand a single space to multiple including a line > wrap. That means we actually get something from DKIM2, then. (Yes, we see strict canonicalization in headers, occasionally, and I'm pretty sure that sometimes causes unnecessary validation failures on whitespace differences. Hopefully those folks will get all excited by DKIM2.) > Yeah, lists which deletes attachments 100% will just have to turn > off dkim2; or we'll need to add a "null body recipe" mode for them, > which declines to describe the body changes. I really don't want to turn off a protocol that Google and Yahoo require. We have enough problems where our users (site and list admins) get into trouble with recipient sites that weaponize DKIM or even SPF. (Sure, site admins should be doing this stuff right, but too often it happens suddenly to lists that are depended on by people in the middle of a disaster!) To do that to users deliberately .... We can always fall back to being the originating site for DKIM2 purposes. We'll have to do that for administrative mail (liveness checks, etc) and digests anyway. That should usually work. > The problem is that the 40k goes into a header field, which - even > with wrapping is still going to start hitting header limits. Ah, that is going to mean none of the resource usage issues are going to matter. > I mean, that's a take. You didn't ask for DKIM1 or DMARC either, > yet plenty of mailman sites wind up rewriting From addresses > because they have to operate with the rest of the world. DKIM1 is a one-off. Once you get it right, recipients who stick to the inferences allowed by the spec are no problem. The only problem is "excessively paranoid" recipient sites, but they probably subscribe to vulture RBLs anyway. DMARC ... it's been 12.5 years, and I'm still mad. It's an ongoing annoyance, especially Gmail's "stealth" p=reject. > It's not 1997 any more and the amount of CPU involved here is, I > believe, incredibly trivial, even on low powered boxes. Yes. I was more worried about RAM if you have concurrent runners, and queue storage. Especially for hosting services with active enough mailing lists that they have to budget for queue storage. But both seem to be moot points, since header size limits will make it impossible for queue objects to bloat very much. > I'd be happy to benchmark it against a corpus of representative > email if you have one handy. I've only been testing against my own > collections of email which may not match common usage patterns. Do > you have an existing Mailman performance suite I could use? No, just the toy messages we have in the test suite. I don't think performance is at all a problem. My experience is that the performance bottleneck is the outgoing SMTP gateway. I've worked on one site where the Mailman outgoing queues would average about 1 message in every 5 samples across 8 queues with 5 seconds between samples, while the SMTP gateway has several thousand in queue. (That's a burst phenomenon due to very large lists; the point is that Mailman performance is not the bottleneck in the list delivery process.) There are cases where Mailman's own queues can get backed up, but they're quite rare. And at less than a couple hundred bytes per message, it's just not going to make a difference. The problems I had in mind all assume either a deleted part getting copied to a diff, or an inability to delete parts at all leading to very large messages in an instance that assumes that outgoing messages are going to be under 40kB, for example. > As you saw in my example above. The diff data was shorter than > some Message-Id headers I've seen in the wild! Yes, you need to do > a separate email per recipient (but people using VERP already do > that), which might be more of a space usage issue for lists on > really small hardware I guess. Yeah, it's just not going to be a problem because of the header limitations. Messages with parts deleted will usually have to use a "null body" diff, or say they're the originating domain and probably do the DMARC From dance as well. > My "dkim2.com" test machine is running on a single CPU, 2 GB > instance deliberately so I see how it performs. That is the case I'm worried about. A typical Linux kernel and other OS processes consume about 750MB to 1GB, and Mailman is admittedly a pig: somewhere between 12 and 20 processes (depending on features configured and whether you're running multiple slices for some queues) at about 75-90MB each. Somebody with a couple of small lists should fit into that 2GB, maybe with a little tuning. But with DKIM2 overhead at less than 100kB/message, I don't see how it can possibly matter. So the only real question in my mind is whether it's worth the extra effort to preserve content-type text/plain messages without wrapping. I'll have to check, but I don't think there are any cases in stock Mailman where we edit body parts, except (1) we delete a whole part, or (2) we convert a text/html part with no text/plain alternative to text/plain. Case (2) would be treated the same way as a part deletion due to size in most cases. So all the cases where you'd have a change record would be "add M lines; copy all; add N lines", and Mailman would actually use that algorithm to construct the body. > I'll definitely run any benchmarking on that as well as my rather > more powerful laptop. Appreciated. Most serious sites will have rather more powerful hosts, but I for one will be running test instances as much as possible in that 1 CPU, 2GB Nanode! > I am persuaded that we need to add a mode to the Mailman patchset > to allow such sites to configure "strip attachments and include a > NULL body recipe" Good! Then we're pretty much 100% in agreement on what you can do for us, and what Mailman may have to accept. > - and those sites can continue to run in that > mode indefinitely (until too many receivers start rejecting NULL > recipes anyway; which will be years away) Ah, but the good thing about that is that we already have logic for stripping attachments, so it's easy to conditionalize that. It relatively rarely happens (most people know and follow the rules), but when it does, it's a pretty high-value feature as I explained previously. Aside from the null body and be DKIM2 originator approaches, I guess we can do something obnoxious like edit Content-Type (application/octet-stream) and Content-Disposition (filename=attachment#.bin) in the parts to suppress display at least. And even that we can condition on DKIM2-strict sites. You know, it would be nice to have a policy parameter in the DNS we could examine .... > This is the hole that spammers sneak through. My number one goal > that I describe everything in DKIM2 as being for is "reduce the > grey areas". Sure. Aside from some misunderstanding about how things work, and what you can do with the information, I'm just frustrated that as usual your goal (if adhered to strictly) could reduce our usable feature set. > why should I have to take my pocket knife out of my carry-on when I > get on a plane, I'm not planning to hijack the plane. Considering that if you're trained, a sharpened pencil is equally serviceable I have no problem with your pocket knife. I don't have an equally snappy response to the issue with spammers driving a truck through the null body open door, though. > Do you have an alternative solution? Because I haven't been able > to come up with one. If you do, I'm all ears. I don't, and given your spec, I don't think I will. Frustrated as I am in principle, we now have four options we could offer to list admins (no delete, null body, be originator, obfuscate). I don't know that we want to implement them all, but some subset of those should be satisfactory. If you can get null body in the spec, you've done all I can ask! > Otherwise this seems to be "I want to keep doing things which are a > known spam vector" and that's going to hurt mailing lists even more > because they'll get blocked out entirely. Not obvious. For one thing, you're checking if it's spam anyway, and if it's not spam you'll let it through. For another, I would hope that mailing lists that do DKIM2 strictly where possible (most of the time) and occasionally delete a part, use null body, and are not spam, would be granted good reputations. Your users are going to want that. I mean, I could see it going south for mailing lists, but my bet is that reputation databases are here to stay. If you can just restrict that to 100 million intermediaries insteadd of 10 billion email addresses, that's a win for you, no? > I run an email list service. I'm really keen to make this work for > mailing lists. I don't understand why you think this is bad for > mailing lists. It adds code, and in my experience, bugs. The system is designed to reject mail, and as you point out, mailing lists offer a number of ways for the bad guys to mask themselves. Some of those are going to interfere with what we consider features. > Freemailers aren't getting much out of this, it's really designed > to make mailing lists remain viable! I really don't think Google feels that way any more. They may not actually hate mailing lists, but there's no feeling of sympathy from them whatsoever. > Otherwise we wouldn't have bothered with recipes at all and just > said "no changes allowed, deal with it!" Good luck with that! As you pointed out earlier, you could milter in and milter out to compute the diff at AD boundaries. I bet that's going to be necessary in a *lot* of enterprises. For example, AWS SES replaces Message-ID outgoing. So you will want recipes to placate AWS users who relay mail that way. > I just don't see what else we can do for those lists other than > either require them to be registered in some "trusted lists" > registry In theory, we could require creation of such registries. Make the surveillance capitalists pay for it and public access be free, since it would be trivial on top of the databases they already maintain. > or accept that every spammer can pretend to be one of them. Again, I don't think DKIM2 going to allow anyone to stop checking for spam. So all this means is that on those occasions where this device is used and it turns out to be spam, you can't tell whether the list is run by the spammer or not. But if you're not keeping some kind of reputation database, why do you care when the spam was added? You're just going to forget as soon as the message is deleted. > In that case, that seems like a great first pass! We can always > come back and re-engineer the non-wrap codepath later if there's > demand. > That's good :). I'll start by reworking to the "MIME-wrap > everything". Thanks! > Oh man! Ouch. Sorry to hear that. I hope you recover well. Well, sleep is evading me at this exact moment :-/, but so far things are getting better at a steady rate. And I'll be going to my prefered ortho doc on Saturday to get a proper read. > BTW, just guessing from the ".jp" address, but I'll be in Tokyo for > JPAAWG in November, might see you there if you're joining! That sounds like a great idea! I might even do a lightning talk to plead for the liberation of the poor oppressed mailing lists. :-) As in, "hey, I'm here, I speak Japanese (sorta), if you're sympathetic to mailing lists I'd love to talk!" Regards, Steve -- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan _______________________________________________ Mailman-Developers mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
