Bron Gondwana via Mailman-Developers writes: > Yes, I'm sorry I should have pulled the Mailman community in > earlier. I didn't want to drag you through the really early stages > of "we're not even sure which headers need to be signed" pre-IETF > discussions, and of course since the work went to the IETF, > everyone is welcome! You don't need an invite.
I've participated in IETF WGs before. I am just getting out of Google Summer of Code mode, which as org admin and mentor takes a lot of energy. Also, the last I looked the draft had expired about 6 months previously, IIRC. So didn't seem urgent. > There is a way to [spell "trust me"] with DKIM2 (for the body, you > can't do valid DKIM2 at all without describing all your header > field changes), but it's strongly discouraged. The whole point of > DKIM2 is to avoid having to trust anyone, because you can verify > what they did. Sure, but that's not a Mailman site's problem if they are trusted. It only matters if receivers or gateways decide to be persnickety. BTW it's not clear to me how much you actually gain, because once the message is edited somewhere on the wire, receivers either have to verify that the edited message doesn't contain malicious content, or they have to revert and lose the value-add of the intermediary. That latter will likely piss off end users. I kinda wonder how many sites will spend the cycles on reconstructing original messages. That's an honest question, I would like to hear your data or opinion on it. > > Does this (and other such changes) require monkey-patching the > > Python email package? > > Actually no. Good! > It's purely within decorate.py. The simpler method would be to > always use MIME wrapping for DKIM2-signed incoming messages, but > there's a reason why mailing lists don't just do that with every > message anyway. I'm not sure those reasons really apply any more, at least not to our primary user base. There are specialized lists that most feed machines, but those usually set Mailman to "never edit" (except for trace headers and maybe List-* headers). > The existing wrapper code in decorate.py explicitly takes any > existing CTE and rewrites the part as unicode. I GUESS we could > have the email package have a mode which took that and > back-converted it identically, but that seems very inefficient at > best. The point of the email package is to work with a structured object that is efficient for programmers. Some transformations from wire format to user-friendly representations are not reversible in principle (for example, Unicode equivalent normal forms for composed characters). So for DKIM2 purposes we must store the wire format anyway (perhaps in encoded form, such as storing the positions of folding white space in a header field with the unfolded field). > Yes, the changes description just needs to know the number of lines > to copy from the output data to recreate the input data; and (if > any lines are changed) it has to include the input version of that > line. It's basically an "undo" diff. Which in the case of Mailman might be 100MB of deleted attachment. > If the CTE is changed, then that's a header diff, and the > description of that will be included in the header recipe. Aha! That's helpful for my understanding. Makes sense, that's how DKIM 1 would do that kind of thing. > Worst case (without adding ANY support to Mailman at all) my > wrapper milters capture the message on the way in, and on the way > out. You can diff ANY two sets of data of course, so you can > calculate the diff - but the more that Mailman re-encodes and > fiddles with things, the bigger the diff. I am just not worried about 40kb text diffs, though I guess that may piss off MTA maintainers. Not in a world where Outlook takes 4kb of text/plain and appends 40kb of text/html. I guess recipient domains might like to save a handful of cycles per message, but, uh, what's in it for Mailman sites? > A lot of this pre-cursor work is just reducing the size of the diff > to the smallest manipulation required. Sure, I understand that. You're using Mailman's CPU for that, though. We didn't ask for it, why should we pay for it? > > I'm not sure I like the implications of multiplying the > > storage requirements for large messages that way, especially for > > resource-constrained instances (eg, living in a Linode Nanode). > > Do you mean storage requirements for the archive, or storage > requirements during the message transit? The archive isn't a problem, we just strip out the DKIM2 diff data before storing if it is (most archivers strip the trace headers anyway). I'm talking about RAM and message queue storage. In particular, many sites run multiple out queues, and bloating the messages with diff data could get runners oom-killed. Also output and archive queues often end up with with dozens of message objects in them for one reason or another, you could fill the disk much faster. People really do run Mailman in ~1 vCPU + 2GB + small /var, and hosting sites do care about bloating message objects because they run on thin margins -- cost of storage matters to them. > > And I know of at least two sites with lists whose content > > frequently includes large (GBs) attachments. (Yes, I have seen > > multimegabyte files QP-encoded, although my correspondents are > > too polite to send me GBs of attachments so I can't testify to > > that level of aggravation.) > > That shouldn't cost any more if those parts aren't edited - the > lines will just be copied verbatim. You're missing the point: bandwidth for the Mailman site and disk storage for all list subscribers. I do not want Mailman to be a copy machine for everybody's 4k pet photos (unless the list owner wants that). *Especially* if nobody will actually see the cute kitties! > Strictly, DKIM2 treats the body like an array of lines. We did > think about doing MIME-structure based recipes. There are some real > advantages to understanding the structure better, but we concluded > that the added complexity didn't earn its keep. Not surprised. Not sure what could be done with it myself (you'd really have to DKIM sign each part separately for "edits by part" functionality, and I suspect that would be a huge can of worms). > But this leads to another point. > > > Obviously, it is at best highly undesirable to include those > > deleted parts verbatim in a change description. > > Deleting parts is a sin. So is sending messages that violate list policy. If people do that inadvertantly, why not strip the part and get the conversation going? This is a common situation on Mailman lists, with people sending us incident reports with screenshots of error messages and the like. They are *very* grateful to get some analysis back immediately even though they have to convert their screenshots to text etc. in their next post. > A list which strips attachments should probably just not > participate in DKIM2; or should re-originate the message (strip any > existing DKIM2, and just start with a hashes-only > `Message-Instance: m=1` on egress). So we're back to "trust me", or lose a value-added service. Once again, ongoing burdens are imposed on mailing lists. > Otherwise yes, you need to describe the entire attachment. At that > point your choice is either to encode it in the recipe (which will > make the header ungodly long) or hide it in the MIME postamble. > Either way, the wire size will still include the entire size of the > attachment, negating most of the benefits of stripping it in the > first place. No, the wire size will be the entire size of the attachment times the size of the list. Are you starting to see our problem? > > In any case where a message violates site or list policy but > > without DKIM2 we could massage it into conformance, we will have > > an option to reject it. > > Indeed, you always have that possibility! Or to forward it without > adding DKIM2 -- at least while the world still allows that. Well, yeah, that's the problem. The world by and large is happy to sacrifice mailing list affordances for dubious improvements to spam filtering that mostly benefit the large freemailers, and companies that don't want to restrict sending from p=reject addresses to transactional mail flows. I don't see any point to nonparticipation in DKIM2. I like the improvements to the chain of custody. We are not always the last hop in the chain. DKIM2 clearly has some value to Mailman sites. But the requirement for reproducibility of the original so that original signatures can be verified is going to be painful for some lists if recipient domains require that. > > To me, this all looks overengineered. > > Yes, it is a little. It's done that way for a reason, to make sure > we minimise the amount we change the lines on the wire. Certainly > just going straight to MIME wrapping for everything works (it was > my first pass); and I'd be very happy for that to be an answer that > everyone was happy with. OK, we have a starting point. Before you spend your time on reviewing generated code etc, we should talk about what you do want beyond the bare minimum we seem to agree on already, because I know a lot about mailing lists you want to know. For example: > You probably know better than me how many lists just MIME-wrap > everything. Basically, no Mailman lists do that for *everything*, because Mailman MIME-wraps only when it has to. That's almost all the time these days (most people send HTML mail), but there are still a lot of weirdos like me who tell their MUAs to ignore rich text in favor of text/ plain, and send almost all their mail as text/plain -- such posts don't get wrapped, they get prepended and appended, maybe with a change in the charset parameter or CTE. However, as I said above, pretty much all subscribers can handle wrapping the original and prepending a header (uncommon AFAIK) and appending a footer (close to universal due to CAN-SPAM, GDPR, etc). And for any who can't, the wrapped version is ugly but readable for text/plain 7bit/8bit, and the rest are unreadable anyway. (In theory text/html could be readable, but hey, Microsoft MUAs, ya know?) So in practice, "MIME-wrap a lot" to "MIME-wrap all DKIM2" is not a big change, I think. But we'll see. That's why options rule. > If that's the future, then I'm all in favour. I think it's a good idea to start there, and see if any users notice. In any case, how much more efficient a description can you get than "add N lines, copied lines L, M, add K lines"? You don't need any content in the diff because nothing was deleted. > > How about just check for a DKIM2 header? If so memoize the body > > verbatim. If the body was compound, you need to do the regular > > processing, and perhaps reject on the basis of "our rules say > > mutate but DKIM2 can't deal, so please reformulate your message > > to satisfy policy or turn off DKIM2 and we'll do it". Finally, > > MIME-wrap the original body, and compute the change description. > > In the cases of text/plain 7bit or 8bit bodies, include a note in > > the MIME prelude that "this message body was MIME-wrapped because > > the sender uses DKIM2, you need a MIME-capable MUA to view this > > as the sender intended." > > Works for me. Good! There a couple of other things we can do around the IN queue (that means before any real processing, including parsing to an email.Message object), such as check size, to avoid some copying. > Excellent. I'm not _as_ experienced with Python as I am with Perl > or C, but I've written bits of it (including a chunk of work with > offlineimap back in the day). You know, if this is "in a year or so DKIM2 is gonna be everywhere" priority, there's always Summer of Code 2027. And don't hesitate to ask me/us to do something if you're stuck for time or we seem much better able to do it. We might say "no" (likely, actually, we're anything but overstaffed), but we won't get mad. > Right now my branches have both an option to turn on dkim2 support > entirely, and a per-list switch to control which lists it's enabled > for. I agree that this is the best approach; and I'd strongly > agree with defaulting it to being turned off until DKIM2 is fully > standardised. Great! We've got a good foundation to start on. Ah, disclaimer -- don't expect actual work out of me for a couple weeks, as I messed up my back (slipped in dog pee and did a Wyle E Coyote style landing on my back, wish I had video!) Will be OK, but there's enough pain still that I shouldn't be driving or coding. BS-ing in email is my level. I can do that (enjoy, in fact). :-) Regards, Steve -- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan _______________________________________________ Mailman-Developers mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
