Re: [Mimedefang] Indication of list poster's mail volume required?
I filter mail for a little under 20,000 domains. The user numbers vary all the time, but this month's stats show about 70,000 users. We handle about 1.5 million messages a day. --- Regards, George -Original Message- From: mimedefang-boun...@lists.roaringpenguin.com [mailto:mimedefang-boun...@lists.roaringpenguin.com] On Behalf Of Paul Murphy Sent: Tuesday, May 22, 2012 4:24 PM To: mimedefang@lists.roaringpenguin.com Subject: [Mimedefang] Indication of list poster's mail volume required? Perhaps it is time to encourage the list subscribers to publicise how many addresses they are filtering for, and what volume of traffic they see. That way, when someone makes a contentious statement, we know it only affects them and their dog. For example, when someone says "I've been blocking all mail claiming to be from Hotmail with an X-Outlook-Version header and never had a false positive", it would be useful to know whether they are a national ISP with 2 million customers using a cluster of 80 MD/SA hosts, or a keen geek with a Debian system on an old 386. One sees several million messages per day, and the other under 100. How much weight would you put on an opinion from either of them? In this spirit of disclosure, having started using MIMEdefang about 8 years ago in organisations of a couple of hundred staff, I now use it just for my home addresses and a couple of low volume mailing lists, so ~80 users in total of which only 2 are local. About 500 messages per week, so I'm on the lower end. Previously, my corporate server handled ~15000 messages per day. Paul. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
> And if you want to claim to know more about internet mail that google well, good luck with that. Woohoo! I proved Google wasn't following an email RFC once. Do I get a gold star? Regards, KAM ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
On Tue, May 22, 2012 at 7:41 PM, wrote: > > > Exchange and gmail claim SMTP transport but fail to follow the required > syntax. Exchange isn't natively SMTP, so that mail doesn't originate in an SMTP environment. And if you want to claim to know more about internet mail that google well, good luck with that. > RFC 5321 does not ban rejecting on that basis. It bans only the application >of SMTP syntax to non-SMTP headers. New requirements rarely/never mandate that you stop interoperating with existing behavior. As far as I know, the only time the whole internet was required to change something at the same time was Jan 1, 1983 when TCP/IP was standardized. Even if you wanted to believe that there was a mandate to reject, which there isn't, you would have to give the senders time to adapt to new requirements. A long time. > Where "MUST" is given, such means that there is something to enforce. That's not what MUST means. It applies to what _you_ generate. > I enforce it and get much less spam as a result. Well, no. Your own numbers did not show it as being an indicator of spam. -- Les Mikesell lesmikes...@gmail.com ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
On Tue, 22 May 2012 17:41:05 -0700 (PDT) kd6...@yahoo.com wrote: > You completely missed what I said earlier. That part applies to > NON-SMTP headers and says that we cannot and must not reject headers > from other transports on the grounds that they don't meet SMTP's > syntax. It doesn't apply to headers which fall under SMTP > environment or generation, nor do I enforce SMTP syntax compliance on > non-SMTP generated headers. That's not how I read the RFC. It says as one consequence of non-SMTP environments, there may be noncompliant Received: headers. It says a receiver MUST NOT reject mail because of noncompliant trace headers. It doesn't say you CAN reject noncompliant trace headers if you (somehow?) know they were inserted under SMTP. > 4.4. Trace Information Yes, I know what a sender MUST do. You are ignoring what a receiver MUST NOT do. > Exchange and gmail claim SMTP transport but fail to follow the > required syntax. RFC 5321 does not ban rejecting on that basis. What part of: "receiving systems MUST NOT reject mail based on the format of a trace header field" is unclear? It doesn't say MUST NOT reject mail based on the format of a trace header field inserted by a non-SMTP protocol. It says MUST NOT, period. > Where "MUST" is given, such means that there is something to > enforce. I enforce it and get much less spam as a result. You don't > and you get spammed. That's your problem. But I don't get spammed, so it's not my problem. I use actual working anti-spam techniques to combat spam rather than fascist RFC interpretations that might let me giggle with glee over the ignorance of Microsoft but actually stop hardly any spam. I also happen to receive lots of legitimate mail that makes my company quite a bit of money that would be lost were I to be as pedantic as you, so... who has the problem, again? (As per another poster's request for disclosure: I run servers that process about 600K messages/day. Across our entire customer base, our software processes probably 60M messages/day. How many messages/day do you place at risk with your policies?) Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
--- On Tue, 5/22/12, David F. Skoll wrote:
> On Tue, 22 May 2012 16:18:49 -0700 (PDT) kd6...@yahoo.com wrote:
> > Put that in contrast where RFC 5321 says MUST with regard to using
> > the syntax listed therein for generating trace headers, your
> > statement and policy loses every time. A "must use" directive has no
> > discretion. I reject not because I choose to but because the
> > standard says I must.
>
> I think you are reading a different version of RFC 5321 than the rest
> of us. My version says:
>
> ... As another consequence of trace header fields arising in
> non-SMTP environments, receiving systems MUST NOT reject mail based
> on the format of a trace header field and SHOULD be extremely
> robust in the light of unexpected information or formats in those
> header fields.
>
> It seems to me that your rejection of email solely because of an
> invalid trace header violates a MUST NOT dictum of RFC 5321.
You completely missed what I said earlier. That part applies to NON-SMTP
headers and says that we cannot and must not reject headers from other
transports on the grounds that they don't meet SMTP's syntax. It doesn't apply
to headers which fall under SMTP environment or generation, nor do I enforce
SMTP syntax compliance on non-SMTP generated headers. RFC 5321 section 3.7.2
indicates that under SMTP, a received header MUST be inserted, and in section
4.4, the following syntax MUST be used:
4.4. Trace Information
When an SMTP server receives a message for delivery or further
processing, it MUST insert trace ("time stamp" or "Received")
information at the beginning of the message content, as discussed in
Section 4.1.1.4.
This line MUST be structured as follows:
...
[skip to the bottom of page 58 for the ABNF syntax defining the headers.]
If a message contains a header which does NOT have "with SMTP" (in any form
matching the regex given in a prior message), I do NOT apply the rules which
require and enforce the "from", "by", "via", "id", and "for" clauses' syntax,
and that is pursuant to section 3.7.2's prohibition. However, if a message
claims "with SMTP" for a given received header, enforcement of the required
syntax of section 4.4 must and does happen (at my server).
Exchange and gmail claim SMTP transport but fail to follow the required syntax.
RFC 5321 does not ban rejecting on that basis. It bans only the application
of SMTP syntax to non-SMTP headers.
Regardless of SMTP or not, messages having unregistered "with" protocols are
rejected for that reason even though the authority for that is only a "should"
(not a must) because proptocol field registration is required under the overall
STD 10 (which is more than just RFC 5321 and 5322).
Where "MUST" is given, such means that there is something to enforce. I
enforce it and get much less spam as a result. You don't and you get spammed.
That's your problem.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
On Tue, 22 May 2012 16:18:49 -0700 (PDT) kd6...@yahoo.com wrote: > Put that in contrast where RFC 5321 says MUST with regard to using > the syntax listed therein for generating trace headers, your > statement and policy loses every time. A "must use" directive has no > discretion. I reject not because I choose to but because the > standard says I must. I think you are reading a different version of RFC 5321 than the rest of us. My version says: ... As another consequence of trace header fields arising in non-SMTP environments, receiving systems MUST NOT reject mail based on the format of a trace header field and SHOULD be extremely robust in the light of unexpected information or formats in those header fields. It seems to me that your rejection of email solely because of an invalid trace header violates a MUST NOT dictum of RFC 5321. Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
--- On Tue, 5/22/12, Les Mikesell wrote: > ... Unless you like to reject just because you can. Per rfc760 and a > concept assumed through the rfcs:: "In general, an implementation > should be conservative in its sending behavior, and liberal in its > receiving behavior." Put that in contrast where RFC 5321 says MUST with regard to using the syntax listed therein for generating trace headers, your statement and policy loses every time. A "must use" directive has no discretion. I reject not because I choose to but because the standard says I must. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Indication of list poster's mail volume required?
Perhaps it is time to encourage the list subscribers to publicise how many addresses they are filtering for, and what volume of traffic they see. That way, when someone makes a contentious statement, we know it only affects them and their dog. For example, when someone says "I've been blocking all mail claiming to be from Hotmail with an X-Outlook-Version header and never had a false positive", it would be useful to know whether they are a national ISP with 2 million customers using a cluster of 80 MD/SA hosts, or a keen geek with a Debian system on an old 386. One sees several million messages per day, and the other under 100. How much weight would you put on an opinion from either of them? In this spirit of disclosure, having started using MIMEdefang about 8 years ago in organisations of a couple of hundred staff, I now use it just for my home addresses and a couple of low volume mailing lists, so ~80 users in total of which only 2 are local. About 500 messages per week, so I'm on the lower end. Previously, my corporate server handled ~15000 messages per day. Paul. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
All, I think this thread has played itself out. We can just accept the fact that kd6...@yahoo.com is to pedantry as RMS is to Free Software and move along. :) Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
On Tue, May 22, 2012 at 3:31 PM, wrote: > >> > Over 90% of the messages so rejected are clearly spam >> (i.e. sent to a spamtrap mailbox) or have other problems. >> >> That doesn't seem like a particularly strong metric to >> me. What's your overall spam/non-spam ratio? > > In 2012, 50% to date. My current count has 4 more spams than not. > In 2011, 70% spam to 30% not. I no longer have statistics for 2010 or > earler. I replaced my server with new hardware in February 2011. > > This counts only messages that make it to SpamAssassin scoring and are > therefore accepted by the server. Messages rejected by the MTA for any > reason do not get scored. For example, on some days, I have over 200 > connections (separate addresses) rejected due to not having forward-confirmed > reverse DNS entries on the incoming clients. So 90% spam is probably not unusually high for "all mail" - and I wouldn't consider finding some attribute on 90% spam, 10% non-spam email to be a particularly useful indicator that you should reject. Unless you like to reject just because you can. Per rfc760 and a concept assumed through the rfcs:: "In general, an implementation should be conservative in its sending behavior, and liberal in its receiving behavior." -- Les Mikesell lesmikes...@gmail.com Mail to unknown users won't be in the above counts, nor will SPF-failed messages (rejected directly at the "MAIL FROM" SMTP state), etc. Of course, messages with malformed Received headers are rejected by the MTA and not in the count either. I do not greylist, but I do have a fake high MX entry that always tempfails. I do not retain my mail rejection logs for longer than a week and delete them after I have reviewed them so I don't have precise counts to share. > > I do note that more than 90% of my current spam is such because it's > addressed to my spamtraps directly. I've received only about 10 spams this > year which have made it into my inbox, and those were messages which received > sn SA score above my threshold but under 10. Most of the time, the score is > under 4 for non-spam and over 20 for spam, with this year's spam highscore > being 83 (and a fraction). > ___ > NOTE: If there is a disclaimer or other legal boilerplate in the above > message, it is NULL AND VOID. You may ignore it. > > Visit http://www.mimedefang.org and http://www.roaringpenguin.com > MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com > http://lists.roaringpenguin.com/mailman/listinfo/mimedefang ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
--- On Tue, 5/22/12, Les Mikesell wrote: > On Tue, May 22, 2012 at 1:42 PM, wrote: > > Over 90% of the messages so rejected are clearly spam > (i.e. sent to a spamtrap mailbox) or have other problems. > > That doesn't seem like a particularly strong metric to > me. What's your overall spam/non-spam ratio? In 2012, 50% to date. My current count has 4 more spams than not. In 2011, 70% spam to 30% not. I no longer have statistics for 2010 or earler. I replaced my server with new hardware in February 2011. This counts only messages that make it to SpamAssassin scoring and are therefore accepted by the server. Messages rejected by the MTA for any reason do not get scored. For example, on some days, I have over 200 connections (separate addresses) rejected due to not having forward-confirmed reverse DNS entries on the incoming clients. Mail to unknown users won't be in the above counts, nor will SPF-failed messages (rejected directly at the "MAIL FROM" SMTP state), etc. Of course, messages with malformed Received headers are rejected by the MTA and not in the count either. I do not greylist, but I do have a fake high MX entry that always tempfails. I do not retain my mail rejection logs for longer than a week and delete them after I have reviewed them so I don't have precise counts to share. I do note that more than 90% of my current spam is such because it's addressed to my spamtraps directly. I've received only about 10 spams this year which have made it into my inbox, and those were messages which received sn SA score above my threshold but under 10. Most of the time, the score is under 4 for non-spam and over 20 for spam, with this year's spam highscore being 83 (and a fraction). ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
On Tue, May 22, 2012 at 1:42 PM, wrote: > Over 90% of the messages so rejected are clearly spam (i.e. sent to a > spamtrap mailbox) or have other problems. That doesn't seem like a particularly strong metric to me. What's your overall spam/non-spam ratio? -- Les Mikesell lesmikes...@gmail.com ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Received headers in general
--- On Tue, 5/22/12, George Roberts wrote:
> > Exchange uses SMTP but generates
> a syntactically incorrect header. Similarly
> > with Google's gmail (it often omits the "from" clause when required),
> > Yahoo's use of an unregistered protocol ("with NNFMP"*), qmail, and of late,
> > exim.
>
> Do you also then block mail from Gmail, Yahoo, qmail and
> exim if their Received lines are incorrectly formatted?
Yes. I reject ALL messages with incorrectly formatted Received lines.
Note that as long as there's no "with" clause, a syntactically correct line
consists of some random text, a semicolon, and a date stamp. If a "with"
clause is present, its protocl is checked against a list which conisists of the
valid types listed by the IANA, plus this regex: "(HT|NN)TPS?A?". If it does
not match, it is rejected. If it matches "(D|E|UTF8)?(L|S)MTP8?S?A?", then
"from" and "by" is required, "via" (if present) is checked for an atom, and the
"id" and "for" fields (if present) are also checked for validity. The
rejection message cites the section of the RFC which the message violates.
Over 90% of the messages so rejected are clearly spam (i.e. sent to a spamtrap
mailbox) or have other problems. Since messages often have multiple received
headers, the bad header is displayed at the end of the rejection line after a
colon.
For example, here is the sendmail rule rejecting a bogus "with" protocol:
R$* with $- $*$#error $@ 5.5.2 $: "554 Received header unknown WITH
protocol \"" $2 "\" (see http://www.iana.org/assignments/mail-parameters):"
$&{currHeader}
Rules checking valid protocols appear before this rule.
As RFC 5321 indicates that the syntax for "Received:" headers is required for
SMTP-transmitted messages (section 4.4), I have every right to reject any
message via SMTP (or that claims such by including "with SMTP") that does not
match the given syntax as a malformed message -- and I do so.
The procedure of checking the "with" clause against various protocols (SMTP or
not, or not present) is consistent with RFC 5321 Section 3.7.2's requirement
not to reject non-SMTP environment generated received headers on the grounds of
not meeting the SMTP required syntax for that header class. RFC 5322's
received header syntax (section 3.6.7) requires the semicolon and date stamp
for ALL messages (SMTP or not) transmitted on the Internet, so any message with
a received header lacking a semicolon or valid date stamp is also subject to
rejection as a malformed message regardless of how it was injected or
transmitted.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] X-Auto-Response-Suppress header
> Exchange uses SMTP but generates a syntactically incorrect header. Similarly
> with Google's gmail (it often omits the "from" clause when required),
> Yahoo's use of an unregistered protocol ("with NNFMP"*), qmail, and of late,
> exim.
Do you also then block mail from Gmail, Yahoo, qmail and exim if their Received
lines are incorrectly formatted?
George
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] X-Auto-Response-Suppress header
On Mon, 2012-05-21 at 12:22 -0700, kd6...@yahoo.com wrote: > --- On Mon, 5/21/12, Bernd Petrovitsch wrote: > > On Don, 2012-05-17 at 16:02 -0700, kd6...@yahoo.com wrote: > > > ... > > > Beliefs like yours are the problem. Policies like mine cause the > > > solution. > > > > Perhaps it is more annoying if you add these rules to SpamAssassin and > > score spam points for it. > > Definently not. A rejected message (returned to the sender) gets more > action (or administrative notice) than one accepted as spam therefore > unanswered. Maybe, but in commercial environments it depends if the sender needs more from the receiver (or has more force/power/) or vice versa to decide which of both opinions on the issues decide the following actions. And that can well be "turn off the blocking". Please note that I didn't mention anything on what is correct and what is wrong, good or bad or ugly, standard-compliant or not, etc. because that does not matter in any way there - it is a religious matter for these people in believing in M$FT. No, I do not like it either but the majority of people obviously have no problem with it. Yes, I reply inline and delete full-quotes (if I happen to answer them). Yes, I mark all of these "I'm, not in my office until" mails as spam (especially if I get more than one inclusive-or they go over a mailing list) because that *is* spam as in "unsolicited bulk email". Just plain rejecting mails simply kills the communication (as it kills the business relation) so that won't you get that far. Any better idea than being a PITA and flagging it as spam to get in the long run people to think about it? Bernd -- Bernd Petrovitsch Email : be...@petrovitsch.priv.at LUGA : http://www.luga.at ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
