Unicode right to left HTML override obsfucation

[Sean Doherty]

Is there any rules available for catching messages that use the unicode 
right to left override in HTML to reverse text (sample attached)?

For instance 'H‬olle‮ W#8236;dlro‮' would render as
'Hello World' 

I've seen a couple of these sneak thru recently. I don't want to create
a rule to just look for ‬ and ‬ as I'm not sure what the FP 
rate would be like. Is there legitmate reasons for using these tags?

Regards,
- Sean


Date: Fri, 11 Nov 2005 06:07:23 +
From: Verification <[EMAIL PROTECTED]>
Subject: copperfasten.com ID: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Message-id: <[EMAIL PROTECTED]>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.2180
X-Mailer: Microsoft Outlook Express V6.00.2900.2180
Content-type: multipart/alternative;
 boundary="Boundary_(ID_TJ35Q6wQVlLk2Tu4Hqj1Ew)"
Importance: Normal
X-Priority: 3 (Normal)
X-MSMail-priority: Normal
X-Spam-Status: No, score=2.838 tagged_above=-999 required=5 tests=[BAYES_60=1,
 FORGED_OUTLOOK_TAGS=0.074, HTML_40_50=0.035, HTML_MESSAGE=0.001,
 HTTP_ESCAPED_HOST=0.477, HTTP_EXCESSIVE_ESCAPES=0.151, MIME_HTML_MOSTLY=1.023,
 MPART_ALT_DIFF=0.066, URI_REDIRECTOR=0.011]
X-Spam-Score: 2.838
X-Spam-Level: **

This is a multi-part message in MIME format.

--Boundary_(ID_TJ35Q6wQVlLk2Tu4Hqj1Ew)
Content-type: text/plain; charset=iso-8859-1
Content-transfer-encoding: 7BIT

De?ra? copperfasten.com M?rebme?,

We must c?ceh?k t?tah? yo?ru? copperfasten.com ID was re?deretsig? by r?ae?l
peop?el?. So, to h?le?p copperfasten.com preve?tn? autom?eta?d
re?oitartsig?ns, p?sael?e c?cil?k on t?sih? li?kn? and co?etelpm? co?ed?
verific?noita? p?or?cess:

http://copperfasten.com/MuyGXKwbw4sGA8DOv5EYUMdXY1qWQDOJbK8OdmxJoXgu0vPNsudsuHii85
c6
  

Th?kna? you. 


--Boundary_(ID_TJ35Q6wQVlLk2Tu4Hqj1Ew)
Content-type: text/html; charset=iso-8859-1
Content-transfer-encoding: 7BIT



De‮ra‬ copperfasten.com M‮rebme‬,

We must c‮ceh‬k t‮tah‬ yo‮ru‬ 
copperfasten.com ID was re‮deretsig‬ by r‮ae‬l 
peop‮el‬. So, to h‮le‬p copperfasten.com 
preve‮tn‬ autom‮eta‬d
re‮oitartsig‬ns, p‮sael‬e c‮cil‬k 
on t‮sih‬ li‮kn‬ and co‮etelpm‬ 
co‮ed‬ verific‮noita‬ p‮or‬cess:

http://www.google.sk/url?q=http://%73%09%74%61%4e%64%41%72%74%7a%09a.c%6fM/%63gi-%62i%6e%09/%70%6f%63%09%68/%72%09%65%09%64%69r.%63%67%09i?s=copperfasten.com";>http://copperfasten.com/MuyGXKwbw4sGA8DOv5EYUMdXY1qWQDOJbK8OdmxJoXgu0vPNsudsuHii85c6
 
Th‮kna‬ you.


--Boundary_(ID_TJ35Q6wQVlLk2Tu4Hqj1Ew)--

Re: SA 3.01 scoring very low

[Sean Doherty]

On Thu, 2004-11-04 at 15:04, Dave Goodrich wrote: 
> > Check out trusted_network section of Mail::SpamAssassin::Conf
> > i.e no RBL tests on trusted networks.
> "If you're running with DNS checks enabled, SpamAssassin includes code 
> to infer your trusted networks on the fly, so this may not be necessary. 
> (Thanks to Scott Banister and Andrew Flury for the inspiration for this 
> algorithm.) This inference works as follows:"
> 
> This seems backwards to me. If a user does nothing, then his network 
> will be considered trusted by default? We are an ISP, and SA is running 
> on our toasters. I don't want any machine trusted as that leaves a door 
> open for my smtp relay users (viruses, trojans, just bad folks) to spam 
> local users.
> 
> JMHO, but shouldn't all networks be considered untrusted unless a user 
> specifies otherwise?

I got to agree with you there - especially given that the inference
algorithm doesn't work in every environment.

- Sean

Re: SA 3.01 scoring very low

[Sean Doherty]

On Thu, 2004-11-04 at 14:14, Dave Goodrich wrote:
> Sean Doherty wrote:
> > On Wed, 2004-11-03 at 21:40, Dave Goodrich wrote:
> > 
> >>Good afternoon,
> >>
> >>I just finished testing an upgrade of SA to 3.01 and my scores fell 
> >>through the floor. Read the docs, tried to use the Wiki, followed 
> >>everyone else's upgrade on the list. Not sure just what went wrong.
> > 
> > 
> >>X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on avhost.tls.net
> >>X-Spam-Status: No, score=0.6 required=5.0 tests=ALL_TRUSTED,DRUGS_ERECTILE,
> >> FROM_NO_LOWER,INVALID_DATE,MISSING_SUBJECT,RM_hm_EmtyMsgid
> >> autolearn=disabled version=3.0.1
> > 
> > 
> > You need to specify trusted_networks in local.cf, otherwise 
> > you're going to continue to hit the ALL_TRUSTED rule which can 
> > *decrease* your score by up to -3.3. If you don't specify
> > trusted_networks then SpamAssassin infers what your trusted 
> > networks are - and the inference algorithm may not always get 
> > the correct result. For instance if your mail relay/server is 
> > on a private network and NATed thru a firewall, then the 
> > algorithm may infer incorrectly that the connecting mail server 
> > is trusted. i.e. the algorithm assumes that since you're a 
> > private address, then the next hop server must belong to you 
> > since your MX must be public. However it does not take NAT 
> > into account. Setting trusted_networks appropriately will solve 
> > this issue (I don't think SA 2.64 has the ALL_TRUSTED rule - or 
> > at least it scores low).
> I will look into that, I didn't set it as I want no network to be 
> trusted. I'll reread what I can find on that.

Just set trusted_network 127.0.0.1

> > 
> > Since you hit ALL_TRUSTED certain other DNS based tests are not 
> > run.
> Eh? Where do I find this out?

Check out trusted_network section of Mail::SpamAssassin::Conf
i.e no RBL tests on trusted networks.

> I don't want any networks trusted, infered or otherwise. So I left 
> trusted_networks and internal_networks both blank.

My understanding is that if unset trusted_networks will be infered.
Setting it to the loopback address and/or the host IP address will
prevent this.

>  > Also skip_rbl_checks will do just that.
> Umm I don't follow you there, are you saying skip_rbl_checks will skip 
> SURBL? Because if it does, I'll need to go back to 2.64.

No. Just pointing out that no RBL tests will not be run.

Also, Matt Kettler pointed out in this thread that reason for the
ALL_TRUSTED firing may not be entirely related invalid inference
of trust, but because the Received headers had unknown format in 
the debug output.

- Sean

Re: {SPAM} SA 3.01 scoring very low

[Sean Doherty]

On Wed, 2004-11-03 at 21:52, Matt Kettler wrote:
> At 04:40 PM 11/3/2004, Dave Goodrich wrote:
> >Good afternoon,
> >
> >I just finished testing an upgrade of SA to 3.01 and my scores fell 
> >through the floor. Read the docs, tried to use the Wiki, followed everyone 
> >else's upgrade on the list. Not sure just what went wrong.
> >
> >DAve
> >
> >Here is a sample output of spamassassin -D < test_spam (a known spam that 
> >had been caught and scored as follows,
> 
> 
> >debug: received-header: unknown format:
> >debug: received-header: unknown format:
> >debug: received-header: unknown format:
> >debug: received-header: unknown format:
> 
> 
> 
> There's the cause of your problem.. SA is having problems parsing your 
> received headers.
> 
> As a result, SA is failing to properly detect a trust path, and is 
> triggering ALL_TRUSTED, which should never happen for outside mail.

> In the short term, force ALL_TRUSTED to 0

Matt, does this mean that even if trusted_networks is set in local.cf,
SpamAssassin will fire the ALL_TRUSTED rule even if it can't parse 
the received headers? i.e. Since there are no parsable received 
headers, SA will assume that all must have been trusted? 
Seems a bit aggressive to me...

- Sean

Re: SA 3.01 scoring very low

[Sean Doherty]

On Wed, 2004-11-03 at 21:40, Dave Goodrich wrote:
> Good afternoon,
> 
> I just finished testing an upgrade of SA to 3.01 and my scores fell 
> through the floor. Read the docs, tried to use the Wiki, followed 
> everyone else's upgrade on the list. Not sure just what went wrong.

> X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on avhost.tls.net
> X-Spam-Status: No, score=0.6 required=5.0 tests=ALL_TRUSTED,DRUGS_ERECTILE,
>  FROM_NO_LOWER,INVALID_DATE,MISSING_SUBJECT,RM_hm_EmtyMsgid
>  autolearn=disabled version=3.0.1

You need to specify trusted_networks in local.cf, otherwise 
you're going to continue to hit the ALL_TRUSTED rule which can 
*decrease* your score by up to -3.3. If you don't specify
trusted_networks then SpamAssassin infers what your trusted 
networks are - and the inference algorithm may not always get 
the correct result. For instance if your mail relay/server is 
on a private network and NATed thru a firewall, then the 
algorithm may infer incorrectly that the connecting mail server 
is trusted. i.e. the algorithm assumes that since you're a 
private address, then the next hop server must belong to you 
since your MX must be public. However it does not take NAT 
into account. Setting trusted_networks appropriately will solve 
this issue (I don't think SA 2.64 has the ALL_TRUSTED rule - or 
at least it scores low).

Since you hit ALL_TRUSTED certain other DNS based tests are not 
run.

Also is dns unavailable (dns_available no)? This may explain
why you're not getting SURBL hits (which you should if dns
is fully operational). Also skip_rbl_checks will do just that.

Regards,
- Sean

Re: AWL and ABL Re: trusted_networks and ALL_TRUSTED

[Sean Doherty]

On Tue, 2004-11-02 at 15:16, George Georgalis wrote:

> >> The setup I use routes mail at the tcp level, it's basically impossible
> >> for a message to reach spam assassin if it's from a trusted network.

> >So why not set trusted_networks to 127.0.0.1. That way you can
> >be certain that the rule will never fire. You'll also get the
> >benefit of the DNS blocklists been checked for the addresses in
> >the Received headers - with your current setup, its possible 
> >that some of these will be marked as trusted, and as such you'll
> >lose the benefit of the RBL check.
> 
> There is lots of reasons not to do something. What I'm not seeing
> is a reason why I can't stop trusted_networks from using cpu/dns.

> your idea sounds okay for some applications (and I'm changing from
> 192.168 to 127.0.0.1 as a matter of course), but I don't want every
> address in headers looked up. I don't want any of them looked up.
> I hope it's okay for me to be that way.
> 
> I am concerned about the IP a message is coming from, but in my setup,
> that is dealt with before SA ever sees the message.

You can stop dns lookups by setting "dns_available no" which 
results in the following if trusted_networks is unset.

debug: received-header: cannot use DNS, do not trust any hosts from here
on

However, this also disables SURBLs - which you probably still want!
I don't think its possible to disable DNS lookups for trusted networks
without also disabling it for the SURBLs.

- Sean

Re: AWL and ABL Re: trusted_networks and ALL_TRUSTED

[Sean Doherty]

On Tue, 2004-11-02 at 12:50, George Georgalis wrote: 
> >Do you mean -0.001? Why would you want to penalise mail
> >coming thru a trusted path?
> 
> It really doesn't matter to me what the score is, I just want to disable
> the test.
> http://bugzilla.spamassassin.org/show_bug.cgi?id=3406
> 
> My /etc/spamassassin is the reference I replicate out to my other
> systems, and systems of my clients, which may or may not be on nat and
> certainly are on different networks.
> 
> The setup I use routes mail at the tcp level, it's basically impossible
> for a message to reach spam assassin if it's from a trusted network.
So why not set trusted_networks to 127.0.0.1. That way you can
be certain that the rule will never fire. You'll also get the
benefit of the DNS blocklists been checked for the addresses in
the Received headers - with your current setup, its possible 
that some of these will be marked as trusted, and as such you'll
lose the benefit of the RBL check.

> I had scored ALL_TRUSTED to 0 but then decided I needed to know in
> test reports what was happening. I don't know how much cpu this test
> uses, but I'd like it to go away completely, or at have the option of
> disabling it.
> 
> // George

Re: AWL and ABL Re: trusted_networks and ALL_TRUSTED

[Sean Doherty]

On Mon, 2004-11-01 at 20:37, George Georgalis wrote:

> skip_rbl_checks 1
> use_bayes 0
> 
> noautolearn 1
> use_auto_whitelist 0
> score AWL 0.001
> 
> trusted_networks 192.168.
> score ALL_TRUSTED 0.001

Do you mean -0.001? Why would you want to penalise mail
coming thru a trusted path?

- Sean

Re: trusted_networks and ALL_TRUSTED

[Sean Doherty]

On Mon, 2004-11-01 at 19:28, Justin Mason wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> 
> Jim Maul writes:
> > This is exactly how i have my system setup.  I have a 192.168 IP 
> > assigned to my server.  It has no public IP assigned to it.  However, i 
> > have a router/firewall in front of it which has a public ip assigned to 
> > its wan interface which then does NAT/port forwarding to my qmail 
> > server.  It works extremely well for our purposes.  It sounds to me that 
> > if i upgraded to 3.0 (still running 2.64) i would then have the same 
> > issue with the trusted networks.  It doesnt really sound correct.  Just 
> > because my machine doesnt have a public ip does NOT mean that mail 
> > passes through a trusted source first..unless you are calling my little 
> > SMC barricade a trusted source.
> 
> there's a very easy way to deal with this, and it's what you should
> use.   set trusted_networks.   That's exactly why there's a parameter
> there to set ;)

> Basically, SpamAssassin can't know all about your network setup unless
> you tell it.  it'll try to guess, but there's only so far guessing
> will go, and without information from you, it's pretty much impossible
> to guess this.

So shouldn't SpamAssassin take a conservative approach when guessing,
and advising via the debug output that the user should set 
trusted_networks.

- Sean

Re: trusted_networks and ALL_TRUSTED

[Sean Doherty]

On Mon, 2004-11-01 at 18:24, Matt Kettler wrote:
> At 01:07 PM 11/1/2004, Sean Doherty wrote:
> > > so the *next* step must be the external MX.
> >
> >My 10.x server is inside a firewall which NATs port 25 so this
> >conclusion is not correct. I imagine that my setup isn't all
> >that different from a lot of other peoples.
> 
> Yes, it is incorrect, but SA can't know that. Thus, SA assumes, 
> incorrectly, that any 10.x host must not be externally addressable. It's 
> not a very good assumption in modern networks, but there's not much else 
> one can do.
> 
> SA's trust path code has pretty much always been incompatible with NAT'ed 
> mailservers. And it's hard for SA to autodetect such things from mail headers.

Obviously, there's no way to deduce that the mail path has come
through a NAT'ed firewall, and as such in certain situations
guessing the trusted_networks is not the correct thing to do.

I have always (incorrectly) been running SpamAssassin without
trusted_networks been set, which when running SA 2.64 resulted
in no DNSBL checks. Because I was running Bayes, SURBLs and
a bunch of custom rules I wasn't seeing FPs. 

However, after upgrading to 3.0 I suddenly started seeing much 
more FPs, which I could be attributed to ALL_TRUSTED. Setting 
trusted_networks appropriately has solved the problem, however, 
given that the inference algorithm doesn't deal well with NAT'ed
networks - which IMO is quite common for SMEs - there should be 
perhaps something in the debug output which informs the user 
that trusted_networks is not set and as such will be guessed.

Another option would be to either trust nobody, or not run
those tests that rely on knowing what the trusted networks are.

Regards,
- Sean

Re: trusted_networks and ALL_TRUSTED

[Sean Doherty]

Justin,

> > - if any addresses of the 'by' host is in a reserved network range, 
> >   then it's trusted
> > 
> > However, I would have thought that this would imply that the 10.0.0.53
> > host is trusted and not any servers connecting to it. 
> 
> The problem is that 10.x is a private net, therefore SpamAssassin infers
> it cannot possibly be the external MX sitting out there on the internet.
> (for a host to be sitting on the public internet accepting SMTP
> connections, it'd obviously need a public IP addr.)
> 
> so the *next* step must be the external MX.

My 10.x server is inside a firewall which NATs port 25 so this
conclusion is not correct. I imagine that my setup isn't all 
that different from a lot of other peoples. 

> > Can someone please clarify this for me? Also should I be specifying
> > 10.0.0.53 in trusted_networks in local.cf?
> 
> Yep, that's right -- and trusted_networks will fix it.

Yes trusted_networks does indeed fix the issue, but I'm still
not so sure that the algorithm to deduce trusted_networks is
correct (if not specified). 

For an inbound only relay is it correct to say that trusted_networks
should only contain the IP address of the relay itself?

For an inbound/outbound relay it should contain the local 
network/mask or eg downstream Exchange server + relay host?

Regards,
- Sean

trusted_networks and ALL_TRUSTED

[Sean Doherty]

Hi,

I'm looking for some clarification on trusted_networks, the 
ALL_TRUSTED rule, and in particular how trusted_networks are 
inferred if not specified in local.cf.

Since upgrading to 3.0.1 I have seen an increase in false
negatives, which would have otherwise been caught if not for
the ALL_TRUSTED rule firing.

I don't have trusted_networks set in local.cf, so SpamAssassin
will use the inference algorithm as specified in the docs:

- if the 'from' IP address is on the same /16 network as the top
  Received line's 'by' host, it's trusted 
- if the address of the 'from' host is in a reserved network range, 
  then it's trusted 
- if any addresses of the 'by' host is in a reserved network range, 
  then it's trusted

My postfix mail server, that runs SpamAssasin is in a reserved
network range (10.0.0.53) and processes only incoming mail. The
following msg snippet (Received headers) results in the ALL_TRUSTED 
rule firing:

Received: from 206.81.84.119 (unknown [206.81.84.119]) by
marvin.copperfasten.com (Postfix) with SMTP id 127ACEBC7F for
<[EMAIL PROTECTED]>; Mon,  1 Nov 2004 11:09:24 + (GMT)
Received: from 206.81.84.119 by mail003.datapropo.com; Mon, 01 Nov 2004
16:02:51 +0500

With trusted_networks unset I get the following with I debug
the msg with Spamassassin:

debug: looking up PTR record for '206.81.84.119'
debug: PTR for '206.81.84.119': '206-81-84-119.info-goals.com'
debug: received-header: parsed as [ ip=206.81.84.119
rdns=206-81-84-119.info-goals.com helo=206.81.84.119
by=marvin.copperfasten.com ident= envfrom= intl=0 id=127ACEBC7F ]
debug: looking up A records for 'marvin.copperfasten.com'
debug: A records for 'marvin.copperfasten.com': 10.0.0.53
debug: looking up A records for 'marvin.copperfasten.com'
debug: A records for 'marvin.copperfasten.com': 10.0.0.53
debug: received-header: 'by' marvin.copperfasten.com has reserved IP
10.0.0.53
debug: received-header: 'by' marvin.copperfasten.com has no public IPs
debug: received-header: relay 206.81.84.119 trusted? yes internal? no

I'm assuming that 206.81.84.119 is trusted since the following
condition of the inference algorithm fires:

- if any addresses of the 'by' host is in a reserved network range, 
  then it's trusted

However, I would have thought that this would imply that the 10.0.0.53
host is trusted and not any servers connecting to it. 

Can someone please clarify this for me? Also should I be specifying
10.0.0.53 in trusted_networks in local.cf?

Regards,
- Sean

dccifd question

[Sean Doherty]

Hi,

Spamassassins DCC configuration option "use_dcc" specifies 
whether to use DCC or not. However, it appears that 
Spamassassin will perform a dcc check if dccifd is available 
(if the socket specified under dcc_dccifd_pathor exists) or 
use_dcc is set to 1. The same logic is in both 2.64 and 3.0. 
Can anyone explain the reasoning behind this? I would have 
thought that if use_dcc = 0, then that would be that and no 
dcc checks would be performed. 

sub check_dcc { 
  my ($self, $fulltext) = @_;
  my $have_dccifd = $self->is_dccifd_available();

  return 0 unless ($have_dccifd || $self->is_dcc_available());
  ...

If I comment don't specify dcc_dccifd_path and dcc_home in 
local.cf then no dcc checks will occur, so that workaround 
is fine. Likewise, if I stop the dccifd daemon.

This has caught me out before at a customer site, where 
DCC ports where blocked by their firewall. I set use_dcc to 
0, but didn't stop the dccifd daemon. This resulted in 
dcc_timeout delay for each check...

Why isn't use_dcc the overriding parameter. If it only 
controls DCCproc then the documentation should say this.

TIA,
- Sean