Re: Problem with TxRep's HELO handling
Hi!
On 18.06.2015 00:39, RW wrote:
> On Wed, 17 Jun 2015 23:08:55 +0200
> Franz Schwartau wrote:
>
>> Hi!
>>
>> A few days ago I replaced AWL with TxRep. From time to time I get
>> unusual high scores caused by TxRep since then.
>>
>> So I started debugging the TxRep plugin a bit. The high scores are
>> caused by "HELO: localhost" after sa-learn of a spam mailbox.
>>
>> In check_senders_reputation() line 1252 reads:
>>
>> foreach my $rly ( @{$pms->{relays_trusted}},
>> @{$pms->{relays_untrusted}} ) {
>>
>> Thus every relay parsed from Received headers is used. This leads to
>> $helo = 'localhost' (line 1256) if there is no from in a Received
>> header.
> ...
>> I don't get why TxRep evaluates every relay in line 1252 and
>> following. Shouldn't it just use the latest relay?
>
> As far as I can see it does. It works it's way back through the headers
> setting $helo as it goes, so $helo ends-up being set on the last
> received header tested that matches the criteria (the last tested being
> from the lowest received header in the email).
Yes, TxRep right now uses the _last_ Received header. IMHO it should use
the _latest_ Received header, only.
Let's have a look at the following series of Received header which cause
the problem:
Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com
[209.85.212.175])
by mx1.domain.com (envelope-sender )
(MIMEDefang) with ESMTP id t5HJ97If029681
for ; Wed, 17 Jun 2015 21:09:10 +0200
Received: by wiwd19 with SMTP id d19so1242876wiw.0
for ; Wed, 17 Jun 2015 12:09:07 -0700 (PDT)
Received: from localhost ([188.95.50.54])
by mx.google.com with ESMTPSA id
ka7sm8287084wjc.36.2015.06.17.12.09.06
(version=TLSv1 cipher=RC4-SHA bits=128/128);
Wed, 17 Jun 2015 12:09:07 -0700 (PDT)
The lastest (third) Received header causes $helo to be set to 'localhost'.
It would make more sense if TxRep uses the lastest (first) Received
header setting $helo to 'mail-wi0-f175.google.com'.
Best regards
Franz
Problem with TxRep's HELO handling
Hi!
A few days ago I replaced AWL with TxRep. From time to time I get
unusual high scores caused by TxRep since then.
So I started debugging the TxRep plugin a bit. The high scores are
caused by "HELO: localhost" after sa-learn of a spam mailbox.
In check_senders_reputation() line 1252 reads:
foreach my $rly ( @{$pms->{relays_trusted}}, @{$pms->{relays_untrusted}} ) {
Thus every relay parsed from Received headers is used. This leads to
$helo = 'localhost' (line 1256) if there is no from in a Received header.
Please see the attached log for details. Please note that the high
scores don't show up in this example. Don't get confused... ;-)
I don't get why TxRep evaluates every relay in line 1252 and following.
Shouldn't it just use the latest relay?
Best regards
Franz
1 $rly = {'as_string' => '[ ip=209.85.212.175
rdns=mail-wi0-f175.google.com helo=mail-wi0-f175.google.com by=mx1.domain.com
ident= envfrom=ale...@price2spy.com intl=0 id=t5HJ97If029681 auth= msa=0
]','lc_rdns' => 'mail-wi0-f175.google.com','rdns' =>
'mail-wi0-f175.google.com','ip' => '209.85.212.175','lc_by' =>
'mx1.domain.com','envfrom' => 'ale...@price2spy.com','ip_private' => '','ident'
=> '','id' => 't5HJ97If029681','helo' => 'mail-wi0-f175.google.com','internal'
=> 0,'auth' => '','lc_helo' => 'mail-wi0-f175.google.com','by' =>
'mx1.domain.com','msa' => 0};
2 $helo = 'mail-wi0-f175.google.com';
3 $rly = {'by' => 'mx.google.com','lc_helo' => 'localhost','auth' =>
'ESMTPSA','internal' => 0,'no_reverse_dns' => 1,'envfrom' => '','ip_private' =>
'','lc_by' => 'mx.google.com','as_string' => '[ ip=188.95.50.54 rdns=
helo=localhost by=mx.google.com ident= envfrom= intl=0
id=ka7sm8287084wjc.36.2015.06.17.12.09.06 auth=ESMTPSA msa=0 ]','rdns' =>
'','msa' => 0,'id' => 'ka7sm8287084wjc.36.2015.06.17.12.09.06','helo' =>
'localhost','ident' => '','ip' => '188.95.50.54','lc_rdns' => ''};
4 $helo = 'localhost';
5 TxRep: reputation: none, count: 0, weight: 1.0, delta: 0.000, MSG_ID:
6299928cd975ec60a0c99da65600c69b1184f902@sa_generated
6 TxRep: active, 6299928cd975ec60a0c99da65600c69b1184f902@sa_generated
pre-score: -1.885, autolearn score: 0.397, IP: 188.95.50.54, address:
ale...@price2spy.com (unsigned)
7 TxRep: reputation: 0.396, count: 1, weight: 10.0, delta: -0.001,
EMAIL_IP: ale...@price2spy.com
8 $rly = {'id' => 't5HJ97nj029682','helo' =>
'mail-wi0-f176.google.com','auth' => '','internal' => 0,'ident' => '','by' =>
'mx1.domain.com','msa' => 0,'lc_helo' => 'mail-wi0-f176.google.com','lc_by' =>
'mx1.domain.com','ip' => '209.85.212.176','as_string' => '[ ip=209.85.212.176
rdns=mail-wi0-f176.google.com helo=mail-wi0-f176.google.com by=mx1.domain.com
ident= envfrom=ale...@price2spy.com intl=0 id=t5HJ97nj029682 auth= msa=0
]','lc_rdns' => 'mail-wi0-f176.google.com','rdns' =>
'mail-wi0-f176.google.com','envfrom' => 'ale...@price2spy.com','ip_private' =>
''};
9 $helo = 'mail-wi0-f176.google.com';
10 $rly = {'no_reverse_dns' => 1,'envfrom' => '','ip_private' =>
'','lc_by' => 'mx.google.com','as_string' => '[ ip=188.95.50.54 rdns=
helo=localhost by=mx.google.com ident= envfrom= intl=0
id=ka7sm8287084wjc.36.2015.06.17.12.09.06 auth=ESMTPSA msa=0 ]','rdns' =>
'','by' => 'mx.google.com','lc_helo' => 'localhost','auth' =>
'ESMTPSA','internal' => 0,'ip' => '188.95.50.54','lc_rdns' => '','msa' =>
0,'id' => 'ka7sm8287084wjc.36.2015.06.17.12.09.06','helo' =>
'localhost','ident' => ''};
11 $helo = 'localhost';
12 TxRep: reputation: none, count: 0, weight: 1.0, delta: 0.000, MSG_ID:
6299928cd975ec60a0c99da65600c69b1184f902@sa_generated
13 TxRep: active, 6299928cd975ec60a0c99da65600c69b1184f902@sa_generated
pre-score: -1.885, autolearn score: 0.397, IP: 188.95.50.54, address:
ale...@price2spy.com (unsigned)
14 TxRep: reputation: 0.396, count: 1, weight: 10.0, delta: -0.001,
EMAIL_IP: ale...@price2spy.com
15 TxRep: reputation: 0.396, count: 1, weight: 2.0, delta: -0.001, DOMAIN:
price2spy.com
16 TxRep: reputation: 0.397, count: 1, weight: 2.0, delta: 0.000, DOMAIN:
price2spy.com
17 TxRep: reputation: 0.396, count: 1, weight: 0.5, delta: -0.001, HELO:
localhost
18 TxRep: reputation: 0.397, count: 1, weight: 0.5, delta: 0.000, HELO:
localhost
19 TxRep: reputation: 0.396, count: 1, weight: 3.0, delta: -0.001, EMAIL:
ale...@price2spy.com
20 TxRep: reputation: 0.397, count: 1, weight: 3.0, delta: 0.000, EMAIL:
ale...@price2spy.com
21 TxRep: reputation: 0.396, count: 1, weight: 4.0, delta: -0.001, IP:
188.95.50.54
22 TxRep: reputation: 0.397, count: 1, weight: 4.0, delta: 0.000, IP:
188.95.50.54
23 TxRep: got_hit: -0.000
24 TxRep: post-TxRep score: -1.885
25 TxRep: reputation: none, count: 0, weight: 1.0, delta: 0.000, MSG_ID:
6299928cd975ec60a0c99da65600c69b1184f902@sa_generated
26 TxRep: got_hit: -0.000
27 TxRep: post-TxRep score: -1.885
28 TxRep: reputa
ADSP entry for survey.ebay.com in 60_adsp_override_dkim.cf
Hi! Yesterday we received an e-mail which hits DKIM_ADSP_DISCARD, DKIM_SIGNED and DKIM_VALID from ebay_inv...@survey.ebay.com. 60_adsp_override_dkim.cf lists: adsp_override *.ebay.com survey.ebay.com signs their e-mails with ebay.com (please see the attachments for details). So either the entry for *.ebay.com should be removed from 60_adsp_override_dkim.cf or an entry for survey.ebay.com should be added to match the signing policy. Best regards Franz Received: from mxphxpool23.ebay.com (mxphxpool23.ebay.com [66.211.161.23]) by xxx (envelope-sender ) (MIMEDefang) with ESMTP id q5OIHotO096102 for ; Sun, 24 Jun 2012 20:18:09 +0200 Received: from sr-wv00-8300043 (phxlb132.phx.ebay.com [10.14.183.203]) by mxphxpool23.ebay.com (8.13.8/8.13.8) with ESMTP id q5OIHTwb016608 for ; Sun, 24 Jun 2012 11:17:49 -0700 (GMT) X-DKIM: Sendmail DKIM Filter v2.8.3 mxphxpool23.ebay.com q5OIHTwb016608 DKIM-Signature: v=1; a=rsa-sha1; c=simple/simple; d=ebay.com; s=dkim; t=1340561869; i=@ebay.com; bh=xu0Vu1TvEd9HpMnCVKN61t0LU+M=; h=Message-ID:Date:From:Reply-To:To:Subject:Mime-Version: Content-Type; b=QFFBudlAFYlVPr17M2zEBFtAzIxhf6TCpp89xGwRTRutuyIjgvvvFtOSob2pM8se4 WpFASPn+nglSmSqMeJ+4XPGAxb31BqqmYqOBq3LhwG7tunlWZBbnzj5iDjmhT2/TZN BgILq8GF6ZxPkMvfLmcRgKDKnVVUIj2gD7Gdd4YI= DomainKey-Signature: a=rsa-sha1; s=dksm28; d=ebay.com; c=nofws; q=dns; h=message-id:date:from:reply-to:to:subject:mime-version: content-type:x-ebay-mailtracker; b=HoArGOQTdvJEFnCn3oIGRkDqwrC8ZUFKpvq1EZN0ROX4J/a/sASj2L6Ur9Kj93ccv vKA0CYGZxk+JLbcHVFc1A== Message-ID: <566501828.1340561869333.JavaMail.SYSTEM@sr-wv00-8300043> Date: Sun, 24 Jun 2012 11:17:49 -0700 (MST) From: ebay_inv...@survey.ebay.com Reply-To: ebayresea...@ebay.com To: x...@yyy.com Subject: =?ISO-8859-1?Q?XXX,_Bitte_beantworten_Sie_drei_Fra?= =?ISO-8859-1?Q?gen_zu_Ihrem_zuletzt_get=E4tigten_Kauf_bei_eBay.?= Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=1086013627.1340561869317.JavaMail.SYSTEM.sr-wv00-8300043 X-eBay-MailTracker: 10331.681.77. Jun 25 10:47:44.407 [19887] dbg: dkim: using Mail::DKIM version 0.39 Jun 25 10:47:44.410 [19887] dbg: dkim: performing public key lookup and signature verification Jun 25 10:47:44.415 [19887] dbg: dkim: DKIM, i=@ebay.com, d=ebay.com, s=dkim, a=rsa-sha1, c=simple/simple, pass, does not match author domain Jun 25 10:47:44.415 [19887] dbg: dkim: DK, i=ebay_inv...@survey.ebay.com, d=ebay.com, s=dksm28, a=rsa-sha1, c=nofws, pass, does not match author domain Jun 25 10:47:44.415 [19887] dbg: dkim: signature verification result: PASS Jun 25 10:47:44.415 [19887] dbg: dkim: adsp override for domain survey.ebay.com Jun 25 10:47:44.415 [19887] dbg: dkim: adsp result: D/discardable (override by *.ebay.com), author domain 'survey.ebay.com' Jun 25 10:47:44.433 [19887] dbg: dkim: VALID signature by ebay.com, author ebay_inv...@survey.ebay.com, no valid matches Jun 25 10:47:44.433 [19887] dbg: dkim: VALID signature by ebay.com, author ebay_inv...@survey.ebay.com, no valid matches Jun 25 10:47:44.433 [19887] dbg: dkim: author ebay_inv...@survey.ebay.com, not in any dkim whitelist
Re: Fake MX
Hi Matt! On 08.12.2010 18:33, Matt wrote: > Anyone using the Fake MX trick? > > http://www.webhostingtalk.com/wiki/Fake_MX > > Is it safe to use a fake high and low mx? The term Fake MX doesn't seem to be used consistently. We are using a Fake MX, which responds a temporary error to every e-mail. At the high end there is no problem with this setup but on the low end we experienced problems with certain mailers. So we listed a so called Null MX on the low end. This is a connected ip address with the smtp port closed. This is prefered to a unconnected/unrouted ip address. To sum it up: MX 10 nullmx.domain.com. MX 20 mail.domain.com.<-- the real smtp server MX 30 fakemx.domain.com. As a matter of fact there is no smtp log on the Null MX. So it is sometimes hard to debug e-mail problems. Best regards Franz
Re: URI with spaces are not recognized
Hi John!
John Hardin wrote:
> On Fri, 13 Feb 2009, Benny Pedersen wrote:
>
>> On Fri, February 13, 2009 18:12, John Hardin wrote:
>>> If a URI rule works, what's wrong with a body rule?
>>
>> nothing wroung making bad rules either, point is that if bad rules
>> is needed one have also bad behaving browser problem
>
> Why should the fact that a mail client won't render that URI as a
> clickable link mean there shouldn't be a rule for it? Spammers have been
> obfuscating URIs in this manner for a long time. There's nothing wrong
> with rules for obfuscated URIs.
Thanks for pointing out! :-) Our primary goal is to identify spam, not
to prevent people from typing these obfuscated URLs in their browser...
> OT: Benny, could you refrain from setting your Reply-To to the email
> address of the original poster? Setting it to the mailing list address
> is fine, but setting it to the original poster is just
> passive-aggressive rudeness.
>
> On Fri, 13 Feb 2009, Franz Schwartau wrote:
>
>> So, does anyone know a more general solution for this kind of spam
>> instead of individual body rules?
>
> You might try a rule like:
>
> body URI_SPC_OBFU_SPC
> /\bwww\s{1,20}\.\s{1,20}\w{5,20}\s{1,20}\.\s{1,20}net\b/i
>
> I think it would be risky to make the URI parser attempt too much
> deobfuscation; however, accepting \s+\.\s+ as \. might be justified.
> Perhaps \s+dot\s+ as well.
>
> If the spammer uses something more complex they're reducing the
> likelihood the recipient will bother to deobfuscate the URI, and it's
> more likely to be caught by bayes, so I'd suggest the ROI to SA for
> making it more aggressive isn't large enough.
I thought about this generic body rule, too. Unfortunally this rule
catches also legitimate mistyped URLs containing spaces. Think of users
typing URLs fast and hitting the space bar accidentally while typing. ;-)
After reading PerMsgStatus.pm again another idea came up. Instead of
modifying $schemelessRE (which wouldn't help anyway) the URLs containing
spaces are replaced by URLs without spaces before spamassassin gathers
URIs. Thus all URI specific rules can be applied (e. g. uri directive
and URI blacklists).
The regexp is kept simple intentionally and matches legitimate (without
spaces) URLs as well but this doesn't hurt much.
This patch works for me and perhaps someone else finds it useful.
Comments are welcome, too. :-)
Best regards
Franz
--- PerMsgStatus.pm.new.orig2009-02-14 11:21:20.0 +0100
+++ PerMsgStatus.pm.new 2009-02-14 11:20:54.0 +0100
@@ -1417,7 +1417,13 @@
=cut
sub get_decoded_stripped_body_text_array {
- return $_[0]->{msg}->get_rendered_body_text_array();
+ my $textary = $_[0]->{msg}->get_rendered_body_text_array();
+
+ for (@$textary) {
+
s/(www)\s{0,2}\.\s{0,2}([a-z\d._-]{10,32})\s{0,2}\.\s{0,2}((net|org))/$1.$2.$3/i;
+ }
+
+ return $textary;
}
###
Re: URI with spaces are not recognized
C'mon...
Patient: "Doctor, if I press down here it really hurts..."
Doctor: "Don't press there then."
You won't solve a problem by defining there is no problem.
In these spams people are requested to remove the spaces when entering
the given string ("url") in their browser.
Benny Pedersen wrote:
> On Thu, February 12, 2009 18:26, Franz Schwartau wrote:
>> www . abcdef . net
>>
>> After reading the source for a while I found that $schemelessRE in
>> line 1720 of Mail::SpamAssassin::PerMsgStatus.pm seems to be
>> responsible for that. Unfortunally this regexp doesn't care
>> about whitespaces.
>
> give me a url to a browser that can show above url is simple :)
>
> even my firefox in my nokia phone wont show this, did i miss another
> one ?
>
>> Has anyone a solution?
>
> none so far have a problem ?
>
>> Would be fine if I could use the "uri" directive
>> or even some uribl on this kind of "urls".
>
> it will if there was a problem
Re: URI with spaces are not recognized
Hi John,
thanks for your answer. Probably I should have written more about my
problem.
We're getting a lot of spam with obfuscated urls in the form
www . domain . net
The domain part changes quite often (about daily). The number of domains
is nearly 100 by now. Of course we have body rules for each domain/url
similar to your rule but our time to detect new domains/urls is too slow
(actually our customer has to tell us, that spam got through, which is
quite bad). All these "urls" point to the same content, resolve to the
same ip and are listed in some url black lists. Since spamassassin
doesn't recognize these obfuscated urls, url specific rules don't match.
So, does anyone know a more general solution for this kind of spam
instead of individual body rules?
Best regards
Franz
John Hardin wrote:
> On Fri, 13 Feb 2009, Benny Pedersen wrote:
>
>> On Thu, February 12, 2009 18:26, Franz Schwartau wrote:
>>> www . abcdef . net
>>> Would be fine if I could use the "uri" directive
>
> If a URI rule works, what's wrong with a body rule?
>
> body URI_SPC_OBFU_nn
> /\bwww\s{1,20}\.\s{1,20}abcdef\s{1,20}\.\s{1,20}net\b/i
URI with spaces are not recognized
Hi! A lot of spams arrives here with URI like strings containing spaces, e.g.: www . abcdef . net After reading the source for a while I found that $schemelessRE in line 1720 of Mail::SpamAssassin::PerMsgStatus.pm seems to be responsible for that. Unfortunally this regexp doesn't care about whitespaces. Has anyone a solution? Would be fine if I could use the "uri" directive or even some uribl on this kind of "urls". Best regards Franz
