Re: [Mailman-Developers] ARC module implementation [was: GSOC 2016]

[Aditya Divekar]

Hi!
Sorry for the late reply. I was caught up in my mid term exams.

Stephen J. Turnbull wrote:


> In Mailman 2, there are a number of options.  The one I recommend is
>
adding aol.com and yahoo.com to the subscription and poster ban
> lists.<0.7 wink/>  This has the advantage (?) of also being usable in
> Mailman 3.  A second is to configure the list without transforming the
> message at all (no Subject tags, header, or footer), thus preserving
> any valid DKIM signature.  This also works in Mailman 3.  These are
> standard configuration options which you might use anyway for
> completely different reasons.  However, they sort undermine the
> argument for using Mailman in the first place for many sites.
>
> Yes, I think these are more of workarounds than solutions, if I'm
qualified enough to say so!


> Finally, the most sophisticated alternative is to parse the address
> out of From, and do the DMARC DNS dance to determine if the sending
> domain has a p=reject or p=quarantine policy.  If so, use either the
> From-munging strategy or the MIME-wrapping strategy.  These still have
> the disadvantages described above, but they only apply to posts From
> domains abusing DMARC.
>
> Since the receiver is going to get the mail through the mailing list
anyway, personally I think that the From munging strategy is not a bad
idea. The only case where I can imagine this will create a problem is if I
am receiving the mail as a list posting, and that suppose I want to get all
mails that  sends to the list in a different folder, then
I might face problems since the from header is changed, not allowing the
from filter. I am not sure if that problem will be encountered that often
in context of mailing lists since we usually read all mails from  the list,
(since that is approximately the point behind mailing lists) unless we skip
some by their subject.  (I personally haven't used such a feature, since I
tend to keep all non-me-referencing general mail from the list in the same
folder.) I might be wrong here.

P.S. Also, I have noticed that there is a feature defined in the handlers
to cleanse all the dkim headers from a mail sent to the list before it is
forwarded to the subscribers. We could simply use this to extract the
previous dkim signature, and generate the ARC authentication results since
its a copy of the previous results.


Aditya.
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] ARC module implementation [was: GSOC 2016]

[Barry Warsaw]

On Feb 29, 2016, at 12:08 PM, Stephen J. Turnbull wrote:

>[Aside @Barry:  what's the current state of DMARC handling in Mailman
>3?  (I assume below that there isn't any yet.)  I guess porting some
>or all of the Mailman 2 facilities be a good GSoC project?]

Correct!

Cheers,
-Barry


pgpS2KeCY9JFo.pgp
Description: OpenPGP digital signature
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] ARC module implementation [was: GSOC 2016]

[Stephen J. Turnbull]

[Aside @Barry:  what's the current state of DMARC handling in Mailman
3?  (I assume below that there isn't any yet.)  I guess porting some
or all of the Mailman 2 facilities be a good GSoC project?]

Please keep GSoC traffic on Mailman-Developers.  Other developers have
need-to-know, as do other students.  Also, there's a good chance one
of the other developers will get around to answering before I do,
especially as I just got a pile of work dumped on me at work (one of
our adjunct faculty got suddenly recalled to the Bank of Japan,
leaving 8 units of courses unstaffed from April 1).  I'll stay current
with Mailman as much as a I can, but for the next few weeks that means
responding to non-work mail in 48 hours.

 > The deliver method in deliver.py is responsible for sending the
 > messages, and the errors are caught here and then classified as
 > permanent or temporary. So currently, if I am not wrong, all the
 > rejections that Mailman faces due to the dmarc reject policy of
 > some domains, permanent error 550 I think, are caught here if there
 > are any.

I don't think so.  You'll not catch them in mta.deliver.py, which
talks to your outgoing SMTP gateway, which normally does not do DMARC
processing.  On the other hand, the outgoing gateway doesn't make
Mailman wait until it knows what its remote peer will do; it queues
the message.  So any rejections you see in mta.deliver.deliver() are
a local matter.  Permanent failures likely indicate a configuration
problem, while temporary failures are probably due to maintenance on
the MTA or network problems or the like.

It's the final recipient MTA[1] that does DMARC processing, perhaps
hours or days after deliver() has returned :-).  At that point a
(new!) bounce message is created and sent back through the mail
system to Sender or Errors-To, which is -bounces.  So this is
handled by the bounce runner, not the outgoing runner.

 > I was curious to know how this specific problem is being handled by
 > mailman right now. As in, how are the cases of subscribers of yahoo
 > using mailman currently being handled?

First, subscribers using Yahoo! are not the problem.  The DMARC
problem is that when anybody from Yahoo! posts, *all* subscribers are
at risk[2] of having their service bounce the post.  If it was just
Yahoo! subscribers, we wouldn't have to do anything except tell them
that their mail service is deliberately sabotaged by Yahoo!  It's the
collateral damage to well-behaved third parties that makes DMARC abuse
by AOL and Yahoo! so horrible.

Second, in Mailman 3, currently nothing is done AFAIK.  The first
patch was submitted against Mailman 2 before release of Mailman 3.0.
People using Mailman 3 at this point are unlikely to have large
populations of p=reject posters.

In Mailman 2, there are a number of options.  The one I recommend is
adding aol.com and yahoo.com to the subscription and poster ban
lists.<0.7 wink/>  This has the advantage (?) of also being usable in
Mailman 3.  A second is to configure the list without transforming the
message at all (no Subject tags, header, or footer), thus preserving
any valid DKIM signature.  This also works in Mailman 3.  These are
standard configuration options which you might use anyway for
completely different reasons.  However, they sort undermine the
argument for using Mailman in the first place for many sites.

The third, more pleasant, alternative is to configure the list to
transform the message by changing the *address* in RFC5322.From to the
list, and putting some amount of user identity in the display name of
the RFC5322.From (eg, "From Aditya Divekar (adityadiveka...@gmail.com)
via Mailman-Users") of all messages.  A fourth is to treat the message
as a literal forward by including the message as a message/rfc822 part.

These transformations are done by Mailman Handlers.  Each has
disadvantages.  From-munging basically prevents the recipient from
filtering on From.  Few MUAs know how to handle a message "wrapped" in
a MIME part pleasantly, and some (especially AppleMail on iPhone)
choke badly.

Finally, the most sophisticated alternative is to parse the address
out of From, and do the DMARC DNS dance to determine if the sending
domain has a p=reject or p=quarantine policy.  If so, use either the
From-munging strategy or the MIME-wrapping strategy.  These still have
the disadvantages described above, but they only apply to posts From
domains abusing DMARC.

I don't think any of these would be that hard to port to Mailman 3, it
just hasn't been done yet.

Footnotes: 
[1]  This concept is a little ambiguous, in the case of things like
.forward files that point to another mail server.

[2]  If their mail service participates in DMARC (there's no way of
knowing that unless you're a user there), and if they don't do their
own mitigation (eg, GMail treats many messages from p=reject domains
as p=quarantine).


___
Mailman-Developers mailing list

Re: [Mailman-Developers] ARC module implementation [was: GSOC 2016]

[Stephen J. Turnbull]

Aditya Divekar writes:

 > I have taken your suggestion, and started working on a separate
 > branch to generate the ams from the original dkimpy package.

Student applications will be opening shortly.  You should finish up
anything that is in your head but not in a file soon, and start
preparing your application.

We will want to see a reviewable patch on the tracker as part of that
application (it can be something trivial like a doc patch).  Exactly
what is required isn't set yet (most of us are also busy with PSF org
admin details as well as Mailman stuff), but it won't be related to
this project because this project is "too big" to seriously consider
reviewing soon.  And you don't have to worry about whether we get
around to actually merging it; it just has to be in good enough shape
to review.

The scheduling part is important, and you probably haven't done
anything like it before (scheduling a project you design is very
different from picking classes from a course catalog!)

Steve
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] ARC module implementation [was: GSOC 2016]

[Aditya Divekar]

Hi Stephen,

Stephen J. Turnbull wrote:

>
> I don't think it's a great idea to change the dkimpy library if you
> can avoid it.  The design I had in mind would add a file for
> performing ARC operations such as generating fields, and for dkimpy to
> simply provide the functions for generating the signatures.
>
> I have taken your suggestion, and started working on a separate branch to
generate the ams from the original dkimpy package.
I've created a directory where I can add all the files to generate (or
edit) the signature from the dkim library.

This probably requires some refactoring of dkimpy.  It might also
> require generalizing the canonicalization functions to handle the
> ARC-specific field sets if there are any hard-coded headers in the
> DKIM code.  You might also look at the alternatives to see if they are
> better factored for this purpose.


Yes, some minor changes will be required I believe, like changing the
default frozen headers to be signed or the tags to be added in the sign.
Still, some of them can be circumvented by editing the generated signature
in a separate file.
And yes, I'll go through the other packages you suggested before to see if
I am able to read them better than dkimpy.

>
> I haven't done cross-host work before, but you will surely need a
> gitlab account eventually to file the merge request.
>
> I will try to take a look at the code over the weekend.  I did want to
> make the comments on the factoring of the code ASAP, before you do too
> much work.
>
> I have actually just started working on the repo, and hence there are only
a couple of files that I have committed now.
And yes, I've just pushed the files to github as of now, so that I am able
to get your insight in the proceedings. I'll start working in gitlab soon
once I'm able to come up with a sketch of things.

Thanks!

Aditya
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] ARC module implementation [was: GSOC 2016]

[Stephen J. Turnbull]

Aditya Divekar writes:

 > I have read through the dkimpy library functions and it seems that the code
 > needs to be modified to generate the ARC headers, and I have started making
 > some changes.

I don't think it's a great idea to change the dkimpy library if you
can avoid it.  The design I had in mind would add a file for
performing ARC operations such as generating fields, and for dkimpy to
simply provide the functions for generating the signatures.

This probably requires some refactoring of dkimpy.  It might also
require generalizing the canonicalization functions to handle the
ARC-specific field sets if there are any hard-coded headers in the
DKIM code.  You might also look at the alternatives to see if they are
better factored for this purpose.

 > I have hosted the code at github here.
 > ,

I haven't done cross-host work before, but you will surely need a
gitlab account eventually to file the merge request.

I will try to take a look at the code over the weekend.  I did want to
make the comments on the factoring of the code ASAP, before you do too
much work.

___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Developers] ARC module implementation [was: GSOC 2016]

[Aditya Divekar]

Hi Steve,
I have read through the dkimpy library functions and it seems that the code
needs to be modified to generate the ARC headers, and I have started making
some changes. I'm making these changes in view of generating the AMS for
now using the sign(...) method of the library that is by default used for
generating the dkim signature. Since the AAR is simply a copy, it can be
picked up using a separate method and appended at the end of the AMS later.
I have hosted the code at github here.
, and will keep posting the
changes there as I proceed. Any input from your side would be great :)
Thanks!

Aditya
___
Mailman-Developers mailing list
Mailman-Developers@python.org
https://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: 
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe: 
https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9