Re: [Mimedefang] sendmail and filter_helo interaction

[John Rudd]

Dirk the Daring wrote:



# Check #4
# If the HELO is an FQDN, the index and rindex of "." will not 
be the same

# This catches the spammer using domain.tld (which will slip
#   by Check #2)
if ( index($helo, ".") == rindex($helo, ".") )
{
# Reject connection - invalid HELO
md_syslog('alert', "Non-FQDN HELO $helo by Host $hostip");
return('REJECT', "INVALID HELO/EHLO: $helo is 
not FQDN");

}





   As I wrote previously, my entire filter is heavily logged. My 
analysis of those logs indicates that only about 50% of foreign 
mailhosts connecting to my network get past HELO. Based on the 
I-think-reasonable assumption that no "legitimate" mail server would be 
tripped up by GREETPAUSE, RATECONTROL, CONNCONTROL or the tests I have 
in filter_helo, my conclusion is that those 50% are spammers, and I'm 
effectively stopping them by the end of HELO.





Given that I don't think check #4 is valid, I'm not sure I believe your 
claim.  For one, depending on the configuration I'm using, you might end 
up rejecting my email, because my mail server's hostname is the 
registered domain name (rudd.cc) ... and I'm not a spammer.


(I don't recall any prohibition on a host's name being just its 
registered domain, domain.tld)


I'm also curious why you're using a lot of index/rindex calls instead of 
regular expressions (I'm not enough of an expert to know if one is 
honestly faster than the other).  For the above one, why not:


$helo =~ /^[^\.]+\.[^\.]+$/

(from the start of the string, one or more non-dots, followed by 1 dot, 
followed by one or more non-dots, and then the end of the string; you 
can only match this expression if you have exactly 1 dot in the strong)


Or,

(($helo =~ /\./) && ($helo !~ /\..+\./)

(contains at least one dot, AND does not contain: a dot, at least any 
one other character, and then another dot, anywhere in the string; 
again, you can only match these two expressions if you have exactly one 
dot in the string)



$helo =~ /\./

also works for your "index of . isn't -1" check.

___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] sendmail and filter_helo interaction

[Richard Laager]

On Thu, 2006-11-09 at 23:06 -0500, Dirk the Daring wrote:
>   # Check #3
>   # HELO should not contain "localhost"

How effective is this for you? Do you run into false positives?

>   # Check #4
>   # If the HELO is an FQDN, the index and rindex of "." will not 
> be the same
>   # This catches the spammer using domain.tld (which will slip
>   #   by Check #2)

I check that the HELO must have a ".", but I haven't gone any further
than that. Does this work well for you? Any false positives?

Richard



signature.asc
Description: This is a digitally signed message part
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

[Mimedefang] sendmail and filter_helo interaction

[Dirk the Daring]

   Jim McCullars and I have been discussing filter_helo offlist, and 
David's observation (supported by Jim's experimentation) that if 
filter_helo returns a REJECT, the connection is not immediately rejected, 
but rather is rejected after MAIL FROM.


   It happens that I have been using a heavily-logged filter, and examined 
some test connections that Jim made to my mailserver. Some interesting 
results came out of that. I'm using sendmail v8.13.8 and MIMEDefang v2.57 
on Solaris 9 with Perl v5.8.6.


   It is true that even if filter_helo returns REJECT, the connection is 
not immediately dropped. The sending host can still issue another command, 
such as MAIL FROM.


   However, it is also true that if the sending host does issue another 
command, like MAIL FROM, that there is no corresponding MILTER call. 
MIMEDefang never sees a call to filter_sender if filter_helo returns a 
REJECT.


   So it *appears* like the connection is maintained, but it also seems 
that the SMTP conversation *effectively ends* if filter_helo returns 
REJECT. The connecting host can issue another command, but it will be 
ignored.


   I've theorized that if the connecting host issues a RSET followed by 
another (valid) HELO, the connection can proceed and be successful. This 
might be why the connection is not immediately dropped. Also, I use 
FEATURE(`delay_checks'), which may have something to do with it.


   Something else also came out of the experiments. If a connecting host 
violates GREETPAUSE (sends before presentation of the banner), its HELO is 
still passed via MILTER call to MIMEDefang.


   However, if RATECONTROL is tripped, there is no MILTER call. Presumbly, 
this is true of CONNCONTROL.


   Finally, I got asked about my filter_helo code. My current filter has 
a *lot* of logging statements, because I've been experimenting and need 
to meter its function. I'm also not a Perl hacker, so I doubt my code 
is the most-efficient way to write this. Here it is...feel free to 
use/adapt the code as may suit your needs, or ignore my code and write 
your own:


# Some global variables used by the filter* functions
###
# Declare a hash of
#   - Key: IP addresses we consider internal
#   - Value: Flag as to if host is exempt from AV scans (0=No, 1=Yes)
###
%OurHosts=( "127.0.0.1", 0,
"192.168.2.2", 0,
"10.2.3.4", 0 );

###
# Declare a hash of
#   - Key: Domain Names we host
#   - Value: A flag as to if the Domain should be
#   receiving E-Mail (0=No, 1=Yes)
###
%OurDomains=(   "mydomain.tld", 1,
"otherdomain.tld", 1,
"notadomain.tld", 0 );

[..other code..]

#***
# %PROCEDURE: filter_helo
# %ARGUMENTS:
#  IP address of remote host; hostname of remote host; HELO string
#  presented by remote host
# %RETURNS:
#  2-5 element array (see documentation)
# %DESCRIPTION:
#  Called after SMTP connection has been established and sending host has
#  given a HELO statement, but not MAIL FROM: or RCPT TO:
#***
sub filter_helo ()
{
# Read the parameters passed to the function
my($hostip, $hostname, $helo) = @_;

# Local string for Domain name processing
my($domainstring);
# Local variable for string indexing
my($subindex)=0;

# Search the list of our hosts using the $hostip argument
if ( exists($OurHosts{$hostip}) )
{
# Recognize our internal host
md_syslog('info', "Internal Host $hostip HELO $helo");
# Don't look at it further
return('CONTINUE', 'ok');
}
else
{
# Foreign host
md_syslog('info', "Foreign Host $hostip HELO $helo");
}

# Check if the HELO is an IP address
if ($helo =~ /^(\[?)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(\]?)$/ )
{
# HELO looks like an IP - the comparison will split the string
#   into 3 variables; $1 will have [ or be undefined, $2 
will
#   have the IP address without any brackets, $3 will have
#   ] or be undefined ($1 and $3 are undefined if HELO 
lacked
#   square brackets)
md_syslog('info', "IP HELO $helo");

# Check #0
# The IP address portion should *not* be identical to the
#   original HELO string - if it is, the original HELO 
lacked
#   brackets and therefore is invalid (this is "safer" than
#   trying to evaluate $1 and $3 directly, as they may be
#   undefined, or hav

[Mimedefang] filter_helo saved

[Dirk the Daring]

On Thu, 9 Nov 2006, "David F. Skoll" <[EMAIL PROTECTED]> wrote:


I have reconsidered and reinstated filter_helo.


   Thank you very much, David. I really appreciate your work on 
MIMEDefang.


Dirk
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

[Mimedefang] The campaign to save filter_helo

[Mark van Proctor]

Dirk:
 
Any chance of seeing this fantastic filter_helo that blocks out 50% of your
SPAM?
 
Thanks in advance,
 
Mark
 

Mark van Proctor
Information Systems Analyst 

acQuire Technology Solutions Pty Ltd

Australia Chile Canada United Kingdom 



==
This email (including all attachments) is the sole property of acQuire
Technology Solutions Pty Ltd and may be confidential. If you are not the
intended recipient, you must not use or forward the information contained in
it. This message may not be reproduced or otherwise republished without the
written consent of the sender. If you have received this message in error,
please delete the email and notify the sender.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] Cannot get spamassassin to use bayes

[John Nemeth]

   Date: Tue, 17 Oct 2006 15:37:40 +0200
   From: Andrea Venturoli <[EMAIL PROTECTED]>
   Reply-To: mimedefang@lists.roaringpenguin.com

   As per subject, I'm running the most recent stable versions of 
   sendmail+mimedefang+spamassassin on FreeBSD 5.4. Due to the latest 
   overwhelming increase in spam getting in I tryed enabling bayesian 
   filters: I trained spamassassin and manually checked that it works quite 
   effectively; however I cannot get it to apply bayes when called from 
   mimedefang.

   I've put:

   use_bayes 1
   bayes_path /usr/local/etc/mail/spamassassin

   in /usr/local/etc/mimedefang/sa-mimedefang.cf.

 This will cause you to have one Bayesian database for all users.
Bayesian testing works best when done for each user individually.  In
order to do this, you may want to store the bayes stuff in an SQL
database.  However, I have to warn you that this can seriously bloat
MySQL databases.  Mine are currently running at just under 4 GB and
99% of that is attributable to using it to store SpamAssassin bayes
stuff.

 Having gotten the warning out of the way...  You will need to be
using stream_by_recipient() to be able to process each user
individually.  Also you will need

$SASpamTester->signal_user_changed({ username => $user });

prior to the call to spam_assassin_check();  In order to get the
$SASpamTester variable iniitialised, you will need this code in your
filter (it used to be in the sample filter, but has since been
removed):

# The next lines force SpamAssassin modules to be loaded and rules
# to be compiled immediately.  This may improve performance on busy
# mail servers.  Comment the lines out if you don't like them.
if ($Features{"SpamAssassin"}) {
spam_assassin_init()->compile_now(1) if defined(spam_assassin_init());
}

You will also need to add some lines to your
/etc/mail/sa-mimedefang.cf file:

bayes_store_module Mail::SpamAssassin::BayesStore::MySQL
bayes_sql_dsn  DBI:mysql:spamassassin:localhost
bayes_sql_username spamuser
bayes_sql_password XXX
bayes_auto_learn_threshold_nonspam  0.4
bayes_auto_learn_threshold_spam 10.0

bayes_ignore_to users@spamassassin.apache.org

You will need to setup a bunch of tables as well.  See the README
files in the sql directory of the SpamAssassin distribution on how to
setup the database tables.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] filter_helo called after mail from?

[David F. Skoll]

Anne Bennett wrote:

> [HELO info] comes in as a parameter in filter_relay.

You must be running an old version of MIMEDefang, because that hasn't
been the case since version 2.43.

Regards,

David.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] The campaign to save filter_helo

[John]

Thanks from me too...

At 09:35 AM 11/9/2006, you wrote:

Dirk the Daring wrote:

>When I re-wrote my filter to take advantage of filter_helo, I also
> inserted quite a few logging statements. Mainly to insure that my filter
> did what I wanted it to do.

Doh... oh, well.  I have reconsidered and reinstated filter_helo.

Regards,

David.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang



John Jaeger - Billings, Montana

EMail To: 
Home Page   : 

PGP:
RSA Key ID: 0xAAEC7751  

"Our liberty is protected by four boxes...
The ballot box, the jury box, the soap box, and the cartridge box."
   - Anonymous

"Soap Box" didn't work, now using the "Cartridge Box" 3/20/2003

___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] filter_helo called after mail from?

[Anne Bennett]

Jonas Eckerman <[EMAIL PROTECTED]>:

>> We do our HELO checks in filter_relay,
> 
> How do you do that?
> 
> I thought that neither the $Helo variable nor the commands file (from 
> wich the helo string can be read) was available that early in the
> mimedefang process.

It comes in as a parameter in filter_relay.  Pseudo-code (my real code
has lots of calls to site-specific logging etc.):


[...]
# 2002/11/25 Anne Bennett: our machines:
$re_localhost= '127\.0\.0\.1';
$re_our_networks = '132\.205\.\d+\.\d+';
$re_our_domains  = 
'(?:([\w\-\.]+\.)?(concordia\.ca|concordia\.montreal\.qc\.ca|myconcordia\.ca))';
[...]

sub filter_relay($$$)
{ 
  my ( $relayip, $relayname, $helo ) = @_;

  [...]

  # 2003/09/04 Anne Bennett: strip square brackets from helo string if any:
  $stripped_helo =  $helo;
  $stripped_helo =~ s/^\[+//;
  $stripped_helo =~ s/\]+$//;

  # Reject any (external) HELO/EHLO that pretend to be one of ours.
  # 2002/11/25 Anne Bennett: log only.
  # 2003/06/23 Sylvain Robitaille: in production with rejection.
  if ( ( $relayip !~ /^($re_our_networks|$re_localhost)$/ ) &&
   ( $stripped_helo =~ /^($re_our_domains)$/i ) )
  {
[reject with:]
...  "IP $relayip faked HELO/EHLO with our name '$helo'";
  }

  # 2003/05/05 Sylvain Robitaille: Also check for obviously forged
  #   external numeric HELO/EHLO strings (numeric string does not match
  #   relay's IP address).
  # 2003/06/13 Sylvain Robitaille: In practice the faked numeric HELO
  #   strings can trigger too many false positives, given that many
  #   sending sites may use NAT or double-homed hosts.
  # 2003/09/04 Anne Bennett: however no one should be faking as
  #   one of our networks.
  if ( ( $relayip !~ /^($re_our_networks|$re_localhost)$/ ) &&
   ( $stripped_helo =~ /^\d+\.\d+\.\d+\.\d+$/ ) &&
   ( $stripped_helo =~ /^$re_our_networks|$re_localhost$/ )   )
  {
[just testing, hey, I forgot to check this and put it into production!]
[hmph, doesn't seem to catch anything]
... "IP $relayip faked numeric HELO/EHLO with our IP '$helo'";
  }

...
}



Note: the code above actually hasn't caught much lately.  I think it did
help when we first put it in, though.


Anne.
-- 
Ms. Anne Bennett, Senior Sysadmin, ENCS, Concordia University, Montreal H3G 1M8
[EMAIL PROTECTED]+1 514 848-2424 x2285
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] More on filter_helo

[David F. Skoll]

Philip Prindeville wrote:

> So... duplicating filter_helo() in filter_vrfy(), filter_expn(),
> filter_rtrn(),
> and any other command that can come immediately after a HELO is
> somehow better.

Because of the way MIMEDefang works, the filter_vrfy() may be called in
a completely different Perl process than the filter_helo() was, so you
still would need to pass the HELO information as a parameter if you
want to use it in filter_vrfy().

That's why filter_recipient() includes all the info in filter_sender(),
and filter_sender() includes all of filter_helo(), etc.

--
David.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] More on filter_helo

[Philip Prindeville]

David F. Skoll wrote:

>Philip Prindeville wrote:
>
>  
>
>>What if the milter interface were to evolve to have an additional
>>hook for other commands (like VRFY/EXPN/RTRN, etc) and
>>you wanted to base how you handled those commands on what
>>you had seen in the HELO?
>>
>>
>
>It would be handled the same way as filter_sender is:  The filter_vrfy
>routine would be given the VRFY argument and the HELO argument.
>
>Regards,
>
>David.
>  
>

So... duplicating filter_helo() in filter_vrfy(), filter_expn(),
filter_rtrn(),
and any other command that can come immediately after a HELO is
somehow better.

Ok.  I guess I'm just not smart enough to see how.

-Philip

___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] The campaign to save filter_helo

[David F. Skoll]

Dirk the Daring wrote:

>When I re-wrote my filter to take advantage of filter_helo, I also
> inserted quite a few logging statements. Mainly to insure that my filter
> did what I wanted it to do.

Doh... oh, well.  I have reconsidered and reinstated filter_helo.

Regards,

David.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

[Mimedefang] The campaign to save filter_helo

[Dirk the Daring]

On Thu, 9 Nov 2006, "David F. Skoll" <[EMAIL PROTECTED]> wrote:


Dirk the Daring wrote:


   I use filter_helo and am quite happy with it. I successfully reject
obviously fraudulent HELOs at filter_helo.


At least, you *think* you do.

If you test it, you'll discover they're only rejected at MAIL FROM: time.


   No, I'm fairly sure about this.

   When I re-wrote my filter to take advantage of filter_helo, I also 
inserted quite a few logging statements. Mainly to insure that my filter 
did what I wanted it to do.


   My examination of my logs quite clearly shows that when filter_helo 
ended with a return('REJECT'), the connection progressed no further. I 
have plenty of examples of this.


   Also, after some analysis, I found I was able to reject some 50% of 
foreign (not on my network) SMTP connections by the end of filter_helo. 
Between sendmail's GREETPAUSE, RATECONTROL and CONNCONTROL; and using 
filter_helo to detect obviously fraudulent HELOs, I dropped half the 
spammers that much sooner.



I've already removed filter_helo from the svn version of MIMEDefang;


   I appreciate all the work you do, and I've always been very happy with 
MIMEDefang. I'm constantly referring people looking for a better anti-SPAM 
solution to the RP website.


   I really wish you'd reconsider removal of filter_helo. My personal 
anti-SPAM philosophy is "Reject early, reject often" and filter_helo helps 
me do that.



you can just move your test unchanged into filter_sender.


   That can be done and will work, but it allows spammers to waste that 
much more of my mail relay's CPU and my network's bandwidth. If I know 
they're fraudulent at HELO, why let them lie to me again at MAIL FROM ?

___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Re: [Mimedefang] More on filter_helo

[David F. Skoll]

Philip Prindeville wrote:

> What if the milter interface were to evolve to have an additional
> hook for other commands (like VRFY/EXPN/RTRN, etc) and
> you wanted to base how you handled those commands on what
> you had seen in the HELO?

It would be handled the same way as filter_sender is:  The filter_vrfy
routine would be given the VRFY argument and the HELO argument.

Regards,

David.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID.  You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang