Re: [spamdyke-users] Updated Spamdyke Statistics Script
Hi Erik,
Am 19.10.2008 3:13 Uhr, Eric Shubert schrieb:
> Felix Buenemann wrote:
>> Hi Erik,
>>
>> Am 18.10.2008 20:39 Uhr, Eric Shubert schrieb:
>>> Sergio Minini wrote:
Erik, try:
# cat /var/log/maillog | ./spamdyke_stats.pl
>> [...]
% Valid: 29.11%
% Spam : 69.42%
% Error: 1.47%
Good luck/
Sergio
>>> Thanks, but that doesn't work for me. My spamdyke log messages are going
>>> to the qmail smtp log (log-target=0|stderr). Perhaps that's where the
>>> problem lies, as the formatting would be slightly different.
>>>
>> Yes, the script currently expects syslog syntax. If you can provide a
>> few sample lines from the log and I can modify the script to work with it.
>>
>> -- Felix
>
> Thanks, Felix. Here's a sample rejection (each line begins with @4):
> @400048fa5df51149c60c tcpserver: status: 1/100
> @400048fa5df51149d5ac tcpserver: pid 22865 from 209.133.101.250
> @400048fa5df51149e164 tcpserver: ok 22865
> doris.shubes:192.168.171.11:25 :209.133.101.250::58673
> @400048fa5dfc34b1ebec DENIED_SENDER_NO_MX from:
> [EMAIL PROTECTED]
> to: [EMAIL PROTECTED] origin_ip: 209.1
> 33.101.250 origin_rdns: broadcaster.eonline.com auth: (unknown)
> @400048fa5dfd01593124 tcpserver: end 22865 status 0
> @400048fa5dfd015a7d2c tcpserver: status: 0/100
>
> In case you don't know, the first (@4000...) field is a date/time stamp,
> and can be converted to local time with the tai64nlocal program. I don't
> know if that'll be necessary or not.
>
> Thanks again. I'm eager to see the results.
Please test the attached version. You have to pass --nosyslog parameter
to the script. Also please check which of the $linematch variants
beginning at line 33 is the fastest (run time cat /your/log |
spamdyke-stats.pl --nosyslog to find out) and report it back to me.
Btw. the new version is about 25% faster than the last one, due to
optimizations of the regular expression matching, so updating might be
useful to others aswell.
-- Felix
#!/usr/bin/perl -w
# build 2008101905
use diagnostics;
use strict;
use Getopt::Long;
my $tldtop = 0;
my $detailed = 1;
my $syslog = 1;
GetOptions (
"tld=i" => \$tldtop,
"detail!" => \$detailed,
"syslog!" => \$syslog
) or exit 1;
# Usage: # cat /var/log/qmail/smtpd/current | ./this_file
my %status = ();# hash of status connections
my %origin = ();# hash of tld per status code
my %originsum = (); # hash of tld per status code sums
my %rblstat = (); # hash of DNSBL lists matched
my %rhsblstat = (); # hash of RHSBL lists matched
my %rdnsblstat = ();# hash of patterns in IP_IN_RDNS_BLACKLIST matched
my ($allow, $deny, $error, $allowpercentage, $errorpercentage, $spampercentage,
$sum, $rblsum, $rhsblsum, $rdnsblsum);
$allow = 0;
$deny = 0;
$error = 0;
my $linematch = "";
if($syslog) {
$linematch = 'spamdyke\[\d+\]:';
} else {
$linematch = '[EMAIL PROTECTED],24}';
#$linematch = '[EMAIL PROTECTED],24}';
#$linematch = '[EMAIL PROTECTED] ]+';
}
while(<>){
#if( m/(spamdyke\[\d+\]:|[EMAIL PROTECTED],24})
(ALLOWED|ERROR|TIMEOUT|((DENIED|FILTER)_[^ ]+))(.*)/ ){
if( m/$linematch (.*)/ ){
my $line = $1;
if( $line =~ m/^(ALLOWED|ERROR|TIMEOUT|((DENIED|FILTER)_[^
]+))/ ) {
my $sdstatus = $1;
if( $sdstatus =~ m/FILTER_RBL_MATCH/ ){
$line =~ m/rbl: (\S+)/;
$rblstat{$1}++;
$rblsum++;
}
elsif( $sdstatus =~ m/FILTER_RHSBL_MATCH/ ){
$line =~ m/rhsbl: (\S+)/;
$rhsblstat{$1}++;
$rhsblsum++;
}
elsif( $sdstatus =~ m/FILTER_IP_IN_RDNS_BLACKLIST/ ){
$line =~ m/keyword: (\S+)/;
$rdnsblstat{$1}++;
$rdnsblsum++;
}
next if $sdstatus =~ m/CHKUSER|FILTER_.*/;
$status{$sdstatus}++;
if($tldtop and $line =~ m/ origin_rdns: ([^ ]+)/) {
my $rdns = $1;
$originsum{$sdstatus}++;
if($rdns =~ m/^\(unknown\)$/){
#$origin{$sdstatus}{'unknown'}++;
next;
} elsif($rdns =~ m/\.(com|net)$/){
$origin{$sdstatus}{$1}++;
} elsif($rdns =~
m/\.([a-z]{2,2}\.[a-z]{2,2})$/){ # co.uk
$origin{$sdstatus}{$1}++;
} elsif($rdns =~ m/\.([a-z]{2,})$/){ # de, ru,
...
Re: [spamdyke-users] Updated Spamdyke Statistics Script
Felix Buenemann wrote: > Hi Erik, > > Am 18.10.2008 20:39 Uhr, Eric Shubert schrieb: >> Sergio Minini wrote: >>> Erik, try: >>> # cat /var/log/maillog | ./spamdyke_stats.pl > [...] >>> % Valid: 29.11% >>> % Spam : 69.42% >>> % Error: 1.47% >>> >>> Good luck/ >>> Sergio >> Thanks, but that doesn't work for me. My spamdyke log messages are going >> to the qmail smtp log (log-target=0|stderr). Perhaps that's where the >> problem lies, as the formatting would be slightly different. >> > Yes, the script currently expects syslog syntax. If you can provide a > few sample lines from the log and I can modify the script to work with it. > > -- Felix Thanks, Felix. Here's a sample rejection (each line begins with @4): @400048fa5df51149c60c tcpserver: status: 1/100 @400048fa5df51149d5ac tcpserver: pid 22865 from 209.133.101.250 @400048fa5df51149e164 tcpserver: ok 22865 doris.shubes:192.168.171.11:25 :209.133.101.250::58673 @400048fa5dfc34b1ebec DENIED_SENDER_NO_MX from: [EMAIL PROTECTED] to: [EMAIL PROTECTED] origin_ip: 209.1 33.101.250 origin_rdns: broadcaster.eonline.com auth: (unknown) @400048fa5dfd01593124 tcpserver: end 22865 status 0 @400048fa5dfd015a7d2c tcpserver: status: 0/100 In case you don't know, the first (@4000...) field is a date/time stamp, and can be converted to local time with the tai64nlocal program. I don't know if that'll be necessary or not. Thanks again. I'm eager to see the results. -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Updated Spamdyke Statistics Script
Hi Erik, Am 18.10.2008 20:39 Uhr, Eric Shubert schrieb: > Sergio Minini wrote: >> Erik, try: >> # cat /var/log/maillog | ./spamdyke_stats.pl [...] >> % Valid: 29.11% >> % Spam : 69.42% >> % Error: 1.47% >> >> Good luck/ >> Sergio > > Thanks, but that doesn't work for me. My spamdyke log messages are going > to the qmail smtp log (log-target=0|stderr). Perhaps that's where the > problem lies, as the formatting would be slightly different. > Yes, the script currently expects syslog syntax. If you can provide a few sample lines from the log and I can modify the script to work with it. -- Felix ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Updated Spamdyke Statistics Script
Sergio Minini wrote: > Erik, try: > # cat /var/log/maillog | ./spamdyke_stats.pl > 17661 ALLOWED > 14224 DENIED_RBL_MATCH > -- Breakdown -- > 84.25% zen.spamhaus.org > 15.75% bl.spamcop.net > --- > 12330 DENIED_RDNS_RESOLVE > 10299 DENIED_RDNS_MISSING > 4296DENIED_GRAYLISTED > 651 ERROR > 457 DENIED_BLACKLIST_IP > 412 DENIED_OTHER > 239 TIMEOUT > 59 DENIED_SENDER_BLACKLISTED > 35 DENIED_TOO_MANY_RECIPIENTS > > Allowed: 17661 > Denied : 42112 > Errors : 890 > Total : 60663 > % Valid: 29.11% > % Spam : 69.42% > % Error: 1.47% > > Good luck/ > Sergio Thanks, but that doesn't work for me. My spamdyke log messages are going to the qmail smtp log (log-target=0|stderr). Perhaps that's where the problem lies, as the formatting would be slightly different. -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Updated Spamdyke Statistics Script
Erik, try: # cat /var/log/maillog | ./spamdyke_stats.pl 17661 ALLOWED 14224 DENIED_RBL_MATCH -- Breakdown -- 84.25% zen.spamhaus.org 15.75% bl.spamcop.net --- 12330 DENIED_RDNS_RESOLVE 10299 DENIED_RDNS_MISSING 4296DENIED_GRAYLISTED 651 ERROR 457 DENIED_BLACKLIST_IP 412 DENIED_OTHER 239 TIMEOUT 59 DENIED_SENDER_BLACKLISTED 35 DENIED_TOO_MANY_RECIPIENTS Allowed: 17661 Denied : 42112 Errors : 890 Total : 60663 % Valid: 29.11% % Spam : 69.42% % Error: 1.47% Good luck/ Sergio ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Updated Spamdyke Statistics Script
# cat /var/log/qmail/smtp/current | ./spamdyke-stats.pl Allowed: 0 Denied : 0 Errors : 0 Total : 0 % Valid: 0.00% % Spam : 0.00% % Error: 0.00% # ?? (The log file is far from empty) Felix Buenemann wrote: > Try the attached version. > > Am 09.10.2008 18:38 Uhr, Eric Shubert schrieb: >> Appears to need a bit of work yet: >> # cat /var/log/qmail/smtp/current | ./spamdyke-stats.pl >> Use of uninitialized value in addition (+) at ./spamdyke-stats.pl line 79, <> >> line 8340 (#1) >> (W uninitialized) An undefined value was used as if it were already >> defined. It was interpreted as a "" or a 0, but maybe it was a mistake. >> To suppress this warning assign a defined value to your variables. >> >> To help you figure out what was undefined, perl tells you what operation >> you used the undefined value in. Note, however, that perl optimizes your >> program and the operation displayed in the warning may not necessarily >> appear literally in your program. For example, "that $foo" is >> usually optimized into "that " . $foo, and the warning will refer to >> the concatenation (.) operator, even though there is no . in your >> program. >> >> Use of uninitialized value in division (/) at ./spamdyke-stats.pl line 79, <> >> line 8340 (#1) >> >> Illegal division by zero at ./spamdyke-stats.pl line 79, <> line 8340 (#2) >> (F) You tried to divide a number by 0. Either something was wrong in >> your logic, or you need to put a conditional in to guard against >> meaningless input. >> >> Uncaught exception from user code: >> Illegal division by zero at ./spamdyke-stats.pl line 79, <> line >> 8340. >> at ./spamdyke-stats.pl line 79 >> # >> > > > > > ___ > spamdyke-users mailing list > spamdyke-users@spamdyke.org > http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Updated Spamdyke Statistics Script
Try the attached version.
Am 09.10.2008 18:38 Uhr, Eric Shubert schrieb:
> Appears to need a bit of work yet:
> # cat /var/log/qmail/smtp/current | ./spamdyke-stats.pl
> Use of uninitialized value in addition (+) at ./spamdyke-stats.pl line 79, <>
> line 8340 (#1)
> (W uninitialized) An undefined value was used as if it were already
> defined. It was interpreted as a "" or a 0, but maybe it was a mistake.
> To suppress this warning assign a defined value to your variables.
>
> To help you figure out what was undefined, perl tells you what operation
> you used the undefined value in. Note, however, that perl optimizes your
> program and the operation displayed in the warning may not necessarily
> appear literally in your program. For example, "that $foo" is
> usually optimized into "that " . $foo, and the warning will refer to
> the concatenation (.) operator, even though there is no . in your
> program.
>
> Use of uninitialized value in division (/) at ./spamdyke-stats.pl line 79, <>
> line 8340 (#1)
>
> Illegal division by zero at ./spamdyke-stats.pl line 79, <> line 8340 (#2)
> (F) You tried to divide a number by 0. Either something was wrong in
> your logic, or you need to put a conditional in to guard against
> meaningless input.
>
> Uncaught exception from user code:
> Illegal division by zero at ./spamdyke-stats.pl line 79, <> line 8340.
> at ./spamdyke-stats.pl line 79
> #
>
#!/usr/bin/perl -w
use diagnostics;
use strict;
use Getopt::Long;
my $tldtop = 0;
my $detailed = 1;
GetOptions ("tld=i" => \$tldtop,
"detail!" => \$detailed) or exit 1;
# Usage: # cat /var/log/qmail/smtpd/current | ./this_file
my %status = ();# hash of status connections
my %origin = ();# hash of tld per status code
my %originsum = (); # hash of tld per status code sums
my %rblstat = (); # hash of DNSBL lists matched
my %rhsblstat = (); # hash of RHSBL lists matched
my %rdnsblstat = ();# hash of patterns in IP_IN_RDNS_BLACKLIST matched
my ($allow, $deny, $error, $allowpercentage, $errorpercentage, $spampercentage,
$sum, $rblsum, $rhsblsum, $rdnsblsum);
$allow = 0;
$deny = 0;
$error = 0;
while(<>){
my $line = $_;
if( m/spamdyke\[\d+\]: / ){
my ($a, $b, $c, $d) = split(/:/ , $line);
my ($e, $sdstatus) = split(/ /, $d);
#print "$b\n";
#next if $sdstatus eq "CHKUSER";
if( $sdstatus =~ m/FILTER_RBL_MATCH/ ){
$line =~ m/rbl: (\S+)/;
$rblstat{$1}++;
$rblsum++;
}
elsif( $sdstatus =~ m/FILTER_RHSBL_MATCH/ ){
$line =~ m/rhsbl: (\S+)/;
$rhsblstat{$1}++;
$rhsblsum++;
}
elsif( $sdstatus =~ m/FILTER_IP_IN_RDNS_BLACKLIST/ ){
$line =~ m/keyword: (\S+)/;
$rdnsblstat{$1}++;
$rdnsblsum++;
}
next if $sdstatus =~ m/CHKUSER|(FILTER|DEBUG|EXCESSIVE)_.*/;
$status{$sdstatus}++;
if($tldtop and $line =~ m/ origin_rdns: ([^ ]+)/) {
my $rdns = $1;
$originsum{$sdstatus}++;
if($rdns =~ m/^\(unknown\)$/){
#$origin{$sdstatus}{'unknown'}++;
next;
} elsif($rdns =~ m/\.(com|net)$/){
$origin{$sdstatus}{$1}++;
} elsif($rdns =~ m/\.([a-z]{2,2}\.[a-z]{2,2})$/){ #
co.uk
$origin{$sdstatus}{$1}++;
} elsif($rdns =~ m/\.([a-z]{2,})$/){ # de, ru, ...
$origin{$sdstatus}{$1}++
} else {
#$origin{$sdstatus}{'unknown'}++;
next;
}
}
}
}
foreach my $stat (sort keys %status){
if( $stat =~ m/ALLOWED/ ){
$allow = $status{$stat};
}
elsif( $stat =~ m/TIMEOUT|ERROR/ ){
$error += $status{$stat};
}
else{
$deny += $status{$stat};
}
}
my $aed_sum = $allow+$error+$deny;
if($aed_sum > 0) {
$spampercentage = sprintf("%2.2f", ($deny/($aed_sum)*100) );
$errorpercentage = sprintf("%2.2f", ($error/($aed_sum)*100) );
$allowpercentage = sprintf("%2.2f", ($allow/($aed_sum)*100) );
} else {
$spampercentage = $errorpercentage = $allowpercentage =
sprintf("%2.2f", 0);
}
foreach my $key (sort { $status{$b} <=> $status{$a} || $a cmp $b; } keys
%status){
print "$status{$key}\t$key\n";
if($detailed and $key eq "DENIED_RBL_MATCH" ){
