Re: Rule to detect non-standard headers that aren't X- prefixed

[Bill Cole]

On 2022-05-10 at 20:20:14 UTC-0400 (Tue, 10 May 2022 18:20:14 -0600)
Philip Prindeville 
is rumored to have said:

On May 10, 2022, at 5:57 PM, Martin Gregorie  
wrote:


On Tue, 2022-05-10 at 17:29 -0600, Philip Prindeville wrote:


You're correct that they're different in every message received.


So write a rule that fires on any header name that *doesn't* match
anything in the list of legit headers as defined in the relevant 
RFCs.



See my original message.

I can't think of a single way to match each header, and then test for 
any of them not matching the pattern...


As documented in the POD in Mail::SpamAssassin::Conf, a header rule 
checking "ALL:raw" actually matches against the pristine header section, 
in which you could check for lines that do not begin with the 'standard' 
headers.


Unfortunately, as noted elsewhere in the thread, this pattern uses 
one-time header names AND there is nothing wrong about using random 
words as header names without a leading 'X-' so it's likely a low-yield 
approach.




--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire

Unsubscribe

[Chad]

On May 10, 2022, at 9:16 PM, Bill Cole 
 wrote:

On 2022-05-10 at 18:10:23 UTC-0400 (Tue, 10 May 2022 16:10:23 -0600)
Philip Prindeville 
is rumored to have said:

> Anyone have a rule to detect the following nonsense headers seen in this 
> message I got?

No, and complicating your circumstance: RFC6648

Here's the title & abstract:


  Deprecating the "X-" Prefix and Similar Constructs
   in Application Protocols

Abstract

  Historically, designers and implementers of application protocols
  have often distinguished between standardized and unstandardized
  parameters by prefixing the names of unstandardized parameters with
  the string "X-" or similar constructs.  In practice, that convention
  causes more problems than it solves.  Therefore, this document
  deprecates the convention for newly defined parameters with textual
  (as opposed to numerical) names in application protocols.



-- 
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire


smime.p7s
Description: S/MIME cryptographic signature

Re: Rule to detect non-standard headers that aren't X- prefixed

[Bill Cole]

On 2022-05-10 at 18:10:23 UTC-0400 (Tue, 10 May 2022 16:10:23 -0600)
Philip Prindeville 
is rumored to have said:

Anyone have a rule to detect the following nonsense headers seen in 
this message I got?


No, and complicating your circumstance: RFC6648

Here's the title & abstract:


   Deprecating the "X-" Prefix and Similar Constructs
in Application Protocols

Abstract

   Historically, designers and implementers of application protocols
   have often distinguished between standardized and unstandardized
   parameters by prefixing the names of unstandardized parameters with
   the string "X-" or similar constructs.  In practice, that convention
   causes more problems than it solves.  Therefore, this document
   deprecates the convention for newly defined parameters with textual
   (as opposed to numerical) names in application protocols.



--
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire

Re: Rule to detect non-standard headers that aren't X- prefixed

[Loren Wilton]

Minicomputers-Exhume: sides
Malthus-Films: 88976dea
Parasitic-Homogeneity: db5da28ba3e69a
Capitalizations-Grievously: oilers


It looks like the pattern is
   /[A-Z][a-z]{1,20}-[A-Z][a-z]{1.20}\:\s{1,10}[\w\d]{3,20}/
or something close to that.
Obviously it can mutate, but generally these are made by a tool, and until a 
new version of the tool comes along, they will be stable.


Try someting like
   header  LW_BOGUS_HEADERS ALL =~ 
/[A-Z][a-z]{1,20}-[A-Z][a-z]{1.20}\:\s{1,10}[\w\d]{3,20}\n/is 

Re: Rule to detect non-standard headers that aren't X- prefixed

[Philip Prindeville]

> On May 10, 2022, at 5:57 PM, Martin Gregorie  wrote:
> 
> On Tue, 2022-05-10 at 17:29 -0600, Philip Prindeville wrote:
>> 
>> You're correct that they're different in every message received.
>> 
> So write a rule that fires on any header name that *doesn't* match
> anything in the list of legit headers as defined in the relevant RFCs.


See my original message.

I can't think of a single way to match each header, and then test for any of 
them not matching the pattern...


> 
> Of course you may need to extend that list to include some extras, such
> as headers injected by SA itself, as well as DMARC, DKIM, SPF etc.


That's the easy part.


> 
> Martin
> 
> 

Re: Rule to detect non-standard headers that aren't X- prefixed

[Philip Prindeville]

> On May 10, 2022, at 5:57 PM, Martin Gregorie  wrote:
> 
> On Tue, 2022-05-10 at 17:29 -0600, Philip Prindeville wrote:
>> 
>> You're correct that they're different in every message received.
>> 
> So write a rule that fires on any header name that *doesn't* match
> anything in the list of legit headers as defined in the relevant RFCs.


See my original message.

I can't think of a single way to match each header, and then test for any of 
them not matching the pattern...


> 
> Of course you may need to extend that list to include some extras, such
> as headers injected by SA itself, as well as DMARC, DKIM, SPF etc.


That's the easy part.


> 
> Martin
> 
> 

Re: Rule to detect non-standard headers that aren't X- prefixed

[Martin Gregorie]

On Tue, 2022-05-10 at 17:29 -0600, Philip Prindeville wrote:
> 
> You're correct that they're different in every message received.
> 
So write a rule that fires on any header name that *doesn't* match
anything in the list of legit headers as defined in the relevant RFCs.

Of course you may need to extend that list to include some extras, such
as headers injected by SA itself, as well as DMARC, DKIM, SPF etc.

Martin

Re: Rule to detect non-standard headers that aren't X- prefixed

[Philip Prindeville]

> On May 10, 2022, at 4:58 PM, Kevin A. McGrail  wrote:
> 
> On 5/10/2022 6:10 PM, Philip Prindeville wrote:
>> Anyone have a rule to detect the following nonsense headers seen in this 
>> message I got?
> 
> Interesting. Those look more like something that Bayesian learning would be 
> best to handle.
> 
> But, have you built a corpora of spam and ham?  Do a list of headers that 
> appear in ham and spam corpora and xor out the spam ones.  Then write a rule 
> if any of those exist.  They look like they might change a lot and they are 
> randomized to avoid these type of issues so I see your dilemma and a plugin 
> might be needed.
> 
> Regards,
> KAM


You're correct that they're different in every message received.

Re: DMARC fails for valid record?

[Kevin A. McGrail]

I believe this is a bug and fixed in trunk.

On 5/10/2022 1:55 PM, Bill Cole wrote:

Looks like a bug. It should not be possible to hit DKIM_VALID_AU and also 
DMARC_REJECT and/or KAM_DMARC_REJECT


--
Kevin A. McGrail
kmcgr...@apache.org

Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

Re: Rule to detect non-standard headers that aren't X- prefixed

[Kevin A. McGrail]

On 5/10/2022 6:10 PM, Philip Prindeville wrote:

Anyone have a rule to detect the following nonsense headers seen in this 
message I got?


Interesting. Those look more like something that Bayesian learning would 
be best to handle.


But, have you built a corpora of spam and ham?  Do a list of headers 
that appear in ham and spam corpora and xor out the spam ones.  Then 
write a rule if any of those exist.  They look like they might change a 
lot and they are randomized to avoid these type of issues so I see your 
dilemma and a plugin might be needed.


Regards,
KAM

--
Kevin A. McGrail
kmcgr...@apache.org

Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

Re: DMARC fails for valid record?

[Benny Pedersen]

On 2022-05-10 20:39, Matus UHLAR - fantomas wrote:
On Monday, May 9th, 2022 at 20:35, Alex  
wrote:
I'm trying to understand why this email from a bank fails DMARC when 
mxlookup says the DMARC record is just fine.

https://pastebin.com/0T4Gjn3v




* 1.8 DMARC_REJECT DMARC reject policy
* 6.0 KAM_DMARC_REJECT DKIM has Failed or SPF has failed on the 
message

* and the domain has a DMARC reject policy




It also passes SPF and DKIM



On 2022-05-09 at 17:28:59 UTC-0400 (Mon, 09 May 2022 21:28:59 +)
Laurent S. <110ef9e3086d8405c2929e34be5b4...@protonmail.ch>
is rumored to have said:
As far as I understand, for DMARC to be valid, the enveloppe sender 
address and the header From needs to have the same domain.


On 10.05.22 13:53, Bill Cole wrote:

Not so.

One of SPF (using the domain of the envelope sender) or DKIM (using 
the domain of the signature) must validate AND the domain used in the 
validation must match the domain of the author identified by the From 
header.


correct, however:

From: nore...@ess.firstdata.com
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
s=6g5c7kdjkv3qjrxjsdzn3325ejghli53; d=ess.firstdata.com;
t=1652117979;
h=Date:From:Reply-To:To:Message-ID:Subject:MIME-Version:Content-Type;
bh=gRPH1y61kVZSDVPNuLr2WQo4Q0dpMd1ELWBGEE4Kp8c=;
b=MHojQsOqw1AZHyOIUQahSlbOQMMfufMtRltQ/Y3RCuYVO628KuErabQFB38mc82y
XcsgPG5Xl5Mck5OwlsK3vrS2cmVxfbBlgVRm6yzZehHaJ54Jakjqb5psalWNE5YN2Dw
h1tHFhykima88hgeOzw/KI8y8VidzkeEI/nHOMkk=
Authentication-Results: mail03.example.com (amavisd-new);
dkim=pass (1024-bit key) header.d=ess.firstdata.com
header.b="MHojQsOq"; dkim=pass (1024-bit key) header.d=amazonses.com
header.b="dwNxlXrW"

so the mail looks to be DMARC valid, while SA produces:

 *  6.0 KAM_DMARC_REJECT DKIM has Failed or SPF has failed on the 
message

 *  and the domain has a DMARC reject policy


dkim=pass (1024-bit key) header.d=amazonses.com
header.b="dwNxlXrW"

this does not pass, why do amazonses add dkim :(

when multiple dkim signers is added all must pass for dmarc pass, i 
belive this is the kam fails ?


when amazonses drops dkim signing on forwared mails it begins to be 
stable, what amazonses should do here is to arc seal and arc sign, but 
this must be done before breaking dkim when forwarding


we still wait for spamassassin 4.0.0

note to pmc members is that dmarc plugin do work with spamassassin 
3.4.6, super, i can provide dmarc rule to public so askdns is not used 
for dmarc rules anymore when dmarc plugin is loaded, i belive pmc 
members can do this if version ... aswell :)

Rule to detect non-standard headers that aren't X- prefixed

[Philip Prindeville]

Anyone have a rule to detect the following nonsense headers seen in this 
message I got?

Return-Path: 
Received: from cp24.deluxehosting.com (cp24.deluxehosting.com [207.55.244.13])
by mail (envelope-sender ) (MIMEDefang) with ESMTP 
id 23C2ch8H717309
for ; Mon, 11 Apr 2022 20:38:50 -0600
To: "xy...@redfish-solutions.com" 
From: "Nabil, Home Depot" 
Message-ID: <35ee7c.8b8cf6.a...@uakron.edu>
Date: Mon, 11 Apr 2022 22:38:48 + (UTC)
Minicomputers-Exhume: sides
Subject: Nabil, 1 searches this week
Malthus-Films: 88976dea
List-Unsubscribe: 

Parasitic-Homogeneity: db5da28ba3e69a
MIME-Version: 1.0
Capitalizations-Grievously: oilers
Content-type: multipart/mixed; boundary="--=_1649731129-716331-86"

Obviously, the following bogus header names are present:

Minicomputers-Exhume
Malthus-Films
Parasitic-Homogeneity
Capitalizations-Grievously

The list of legitimate headers is quite small, per RFC-2822 Section 3.6 and 
3.6.7 (odd that 3.6.8 doesn't call out the X-* requirement).

I'd like to fingerprint messages based on non-standard header names.

Has anyone undertaken this already?  I tried playing with:

header __L_NON_STD_HEADERS  ALL !~ 
/^(Return-Path|Received|Resent-Date|Resent-From|Resent-Sender|Resent-To|Resent-Cc|Resent-Bcc|Resent-Message-ID|Date|From|Sender|Reply-To|To|Cc|Bcc|Message-ID|In-Reply-To|References|Subject|Comments|Keywords|Content-Type|Content-Transfer-Encoding|MIME-Version|DKIM-Signature|X-([A-Z][a-z]+(-[A-Z][a-z]*)*))\:/m

But that will only match if *none* of the headers are standard ones, so that 
won't work... I really need to examine the headers one-by-one.

Thanks,

-Philip

Re: DMARC fails for valid record?

[Matus UHLAR - fantomas]

On Monday, May 9th, 2022 at 20:35, Alex  wrote:

I'm trying to understand why this email from a bank fails DMARC when mxlookup 
says the DMARC record is just fine.
https://pastebin.com/0T4Gjn3v




* 1.8 DMARC_REJECT DMARC reject policy
* 6.0 KAM_DMARC_REJECT DKIM has Failed or SPF has failed on the message
* and the domain has a DMARC reject policy




It also passes SPF and DKIM



On 2022-05-09 at 17:28:59 UTC-0400 (Mon, 09 May 2022 21:28:59 +)
Laurent S. <110ef9e3086d8405c2929e34be5b4...@protonmail.ch>
is rumored to have said:
As far as I understand, for DMARC to be valid, the enveloppe sender 
address and the header From needs to have the same domain.


On 10.05.22 13:53, Bill Cole wrote:

Not so.

One of SPF (using the domain of the envelope sender) or DKIM (using the 
domain of the signature) must validate AND the domain used in the 
validation must match the domain of the author identified by the From 
header.


correct, however:

From: nore...@ess.firstdata.com
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
s=6g5c7kdjkv3qjrxjsdzn3325ejghli53; d=ess.firstdata.com;
t=1652117979;
h=Date:From:Reply-To:To:Message-ID:Subject:MIME-Version:Content-Type;
bh=gRPH1y61kVZSDVPNuLr2WQo4Q0dpMd1ELWBGEE4Kp8c=;
b=MHojQsOqw1AZHyOIUQahSlbOQMMfufMtRltQ/Y3RCuYVO628KuErabQFB38mc82y
XcsgPG5Xl5Mck5OwlsK3vrS2cmVxfbBlgVRm6yzZehHaJ54Jakjqb5psalWNE5YN2Dw
h1tHFhykima88hgeOzw/KI8y8VidzkeEI/nHOMkk=
Authentication-Results: mail03.example.com (amavisd-new);
dkim=pass (1024-bit key) header.d=ess.firstdata.com
header.b="MHojQsOq"; dkim=pass (1024-bit key) header.d=amazonses.com
header.b="dwNxlXrW"

so the mail looks to be DMARC valid, while SA produces:

 *  6.0 KAM_DMARC_REJECT DKIM has Failed or SPF has failed on the message
 *  and the domain has a DMARC reject policy

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
(R)etry, (A)bort, (C)ancer

Re: DMARC fails for valid record?

[Bill Cole]

On 2022-05-09 at 14:35:58 UTC-0400 (Mon, 9 May 2022 14:35:58 -0400)
Alex 
is rumored to have said:

> Hi,
>
> I'm trying to understand why this email from a bank fails DMARC when
> mxlookup says the DMARC record is just fine.
>
> https://pastebin.com/0T4Gjn3v
>
>  *  1.8 DMARC_REJECT DMARC reject policy
>  *  6.0 KAM_DMARC_REJECT DKIM has Failed or SPF has failed on the message
>  *  and the domain has a DMARC reject policy
>
> It also passes SPF and DKIM
>
>  *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
>  * -0.0 SPF_PASS SPF: sender matches SPF record
>  * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
>  *   domain
>  * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
>  *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
>  *  valid
>
> I'm using a local DNS resolver, not a public server.

Looks like a bug. It should not be possible to hit DKIM_VALID_AU and also 
DMARC_REJECT and/or KAM_DMARC_REJECT


-- 
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire

Re: DMARC fails for valid record?

[Bill Cole]

On 2022-05-09 at 17:28:59 UTC-0400 (Mon, 09 May 2022 21:28:59 +)
Laurent S. <110ef9e3086d8405c2929e34be5b4...@protonmail.ch>
is rumored to have said:

> On Monday, May 9th, 2022 at 20:35, Alex  wrote:
>
>
>> I'm trying to understand why this email from a bank fails DMARC when 
>> mxlookup says the DMARC record is just fine.
>> https://pastebin.com/0T4Gjn3v
>>
>
>> * 1.8 DMARC_REJECT DMARC reject policy
>> * 6.0 KAM_DMARC_REJECT DKIM has Failed or SPF has failed on the message
>> * and the domain has a DMARC reject policy
>>
>
>> It also passes SPF and DKIM
>
> As far as I understand, for DMARC to be valid, the enveloppe sender address 
> and the header From needs to have the same domain.

Not so.

One of SPF (using the domain of the envelope sender) or DKIM (using the domain 
of the signature) must validate AND the domain used in the validation must 
match the domain of the author identified by the From header.


-- 
Bill Cole
b...@scconsult.com or billc...@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire