[AMaViS-user] Announce: Amavis log reporter updated v1.48.8
Hello Amavis users, I've made enough changes in the Amavis log reporting utility to probably warrant this more general list announcement. The updates since my May 6th announcement are: New Features: - Ability to limit each detailed section's level 1 output (i.e. Top N). Variables that control depth levels in detailed reports may now be specified as m.n, where m is the maximum level to output, and n specifies the number of level 1 items output. For example, the setting: $amavis_SpamBlocked = 2.10 will output the top 10 level 1 items, with each of those items providing 2 sub-levels of detail. - Show SA test scores in spam/ham reports - Add additional ccats MTA-BLOCKED, OVERSIZED, OTHER - Added content-type section (log_level = 2) - Added SpamAssassin bypassed count summary ($sa_mail_body_size_limit) - Rework white/blacklisted section; sender is now tracked. Additional changes: - Change: In percentile reports, when N% is requested for output, show *at least* N% (previously was *at most*, which might have produced no values) - Change: SARules config variable renamed to Show_SARules - Change: option --percentiles changed to --timing_percentiles - NOTE: The internal code has been reorganized to favor newer amavis releases. Some older amavis log entries may be flagged as Unmatched. Please report any Unmatched entries and I will correct the problems immediately. - Fix: config file was required in standalone mode - Fix: only print report headers for SA rules report when data exists - Fix: quarantine cutoff level reports were missing (incorrectly ignored all do_quarantine_and_notify messages) - Fix: continuation lines identification; several log entries that end with ... are not continuation lines, causing subsequent log line to be ignored. Continuation lines are exactly 980 chars - see comments in source. - Fix: end points of percentiles calculations - Fix: add zero-width assertions and use strict IP RE in bycount sort subroutine to match IP addresses more reliably - Fix: alignment issue in Top N Spam/Ham reports - Allow supplementary reports to be printed even when TotalMsgs is 0; This typically would not occur in practice, but is useful for debugging. - Fix: when both SPAM and SPAM-TAG are present, don't double count (which doubles SA test score accumulators). - Fix: problem setting SARules config variable; renamed Show_SARules - Fix: the option --timing_percentiles was not being recognized, due to name change from --percentiles. Both are recognized. - Fix: capture MailZu quarantine release messages - Fix: handle amavis' Hits that includes, but does not add/subtract boost scores (eg. 1.03-3.5) - Fix: ignore a few more log lines for log_level 3 - Report Config files read entry in Startup section - Report file(1) bad/extra output messages - Increased width of timing report for maia - Changed NoSubject title to more accurately indicate Subject header insertion - Internal: group similar messages - Internal: minor changes to sync up with postfix-logwatch release As always, the latest amavis logwatch filter can be downloaded from: http://www.mikecappella.com/logwatch Download and expand the amavis-logwatch.tgz file, and see the enclosed README file for installation, usage, and customization instructions. As a logwatch filter, the filter has been tested with logwatch 7.3.x but probably works in the 7.x series. I have not determined the oldest version of logwatch that will work with this filter, but users have reported that it works with 7.1. Feedback is welcome and encouraged. If you have log lines that are not captured or processed correctly, please send me a copy of the line in some form of archive so that whitespace is not altered, and I'll update the script. Either alter private information, or leave it as is, and rest assured your data will remain confidential. Thanks to Eray Aslan, Dan Horne, Erik Weber, Julien Muhlenpfordt and everyone else who has provided feedback and improvement suggestions. MrC --- - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
Re: [AMaViS-user] Amavis-new memory usage
Hi, I am also facing the same problem while upgrading amavis to version 2.4.5. I have not included any blacklist rule in spamassassin PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 8487 amavis16 0 273m 263m 5004 S 0.0 13.0 1:42.57 amavisd 8484 amavis15 0 260m 250m 4960 S 0.0 12.3 2:19.15 amavisd 8481 amavis16 0 217m 207m 5000 S 0.0 10.2 1:53.16 amavisd 8479 amavis16 0 217m 207m 4960 S 0.0 10.2 1:26.47 amavisd 8483 amavis16 0 208m 193m 4952 S 0.0 9.5 1:37.21 amavisd 8482 amavis16 0 142m 131m 4960 S 0.0 6.5 1:45.12 amavisd 8480 amavis16 0 133m 121m 4952 S 4.7 6.0 0:44.37 amavisd --Prashant Dabre Gary V wrote: Steven wrote: We're running amavis-new on an internal mail server that doesn't see a whole lot of traffic and only has about 15 users, but seeing a LOT of memory usage at all times by amavisd. The server is a VMware virtual image of CentOS 4.3. It's got two processors and about 1.5GB of ram allocated. We're using postfix with virtual users in MySQL as our MTA and the amavis version is 2.5.0 This is the sort of system usage we see from amavis all the time. Once the process gets ramped up and the children get created it sits at this memory usage and basically never moves up or down more than a percentage point. top - 09:32:20 up 7 days, 20:01, 1 user, load average: 0.69, 0.60, 0.39 Tasks: 137 total, 1 running, 136 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2% us, 0.5% sy, 0.0% ni, 99.3% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 1814836k total, 1400840k used, 413996k free,67656k buffers Swap: 2031608k total, 208k used, 2031400k free, 257344k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 1212 amavis18 0 437m 412m 2968 S 0.0 23.3 3:11.92 amavisd 4760 amavis16 0 436m 411m 2876 S 0.0 23.2 1:08.40 amavisd 21985 amavis16 0 434m 409m 2732 S 0.0 23.1 2:03.53 amavisd Turning off Clam altogether has no affect. Does anyone have any idea why we're seeing this and what we can do to bring it down? A few hundred megs of memory per process is pretty steep. Thanks, Steven This looks like you have included a *blacklist* rule set. http://marc.info/?l=spamassassin-usersm=118346997827114 Gary V - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/ -- The sender of this email subscribes to Perimeter eSecurity's email anti-virus service. This email has been scanned for malicious code and is believed to be virus free. For more information on email security please visit: http://www.perimeterusa.com/email-defense-content.html This communication is confidential, intended only for the named recipient(s) above and may contain trade secrets or other information that is exempt from disclosure under applicable law. Any use, dissemination, distribution or copying of this communication by anyone other than the named recipient(s) is strictly prohibited. If you have received this communication in error, please delete the email and immediately notify our Command Center at 203-541-3444. Thanks - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
[AMaViS-user] interesting patch proposition
Hi,
We, at Lerkins Group have used amavisd-new for our anti-spam anti-virus
cluster, and we
have found that that would be very usable to have the ability to resend the
message
that came from host X to host Y, based on config specified in amavisd.conf.
As there was no such ability in amavisd-new we have developed a patch for
amavisd, that
enables such feature very easily while giving the ability to keep old behaviour
untouched.
Now the question is, would it be possible to incorporate this patch to amavisd
official release, so anybody could use it, and also it would be easier to
maintain amavisd
after using that patch (right now if we want to upgrade, we have to apply this
patch afer
prebuild stage of FreeBSD port on each machine)?
The patch is pasted below. Thanks a lot for any reply.
Kind regards
--
Bartłomiej Rutkowski [EMAIL PROTECTED]
--- amavisd.original2006-08-17 13:18:43.0 +0200
+++ amavisd.patched 2006-08-22 12:25:56.0 +0200
@@ -2277,9 +2277,71 @@
($proc_fh, $pid); # return pipe file handle to the subprocess and its PID
}
+# This sub gets two params 1st one is new format of $forward_method
+# smtp: { [smtphellohost1.mydomain] - [10.10.3.13]:10025;
+# [smtphellohost2.mydomain]- [10.10.3.12]:10025 }
+# second one is smtphelo host for which we are looking forward_method
+# in result we get:
+# string smtp:[10.10.3.13]:10025 for smtphellohost1.mydomain
+# string smtp:[10.10.3.12]:10025 for smtphellohost2.mydomain
+# empty string for anything else
+sub getmethod4smtphelo($$){
+my($method,$smtphelo) = @_;
+my($returnmethod)=;
+my(%dynforw_host); #here we keep assoc. table host where e-mail has to be
forwarded
+my(%dynforw_port); #as %dynforw_host but for ports
+my($innermethod, @fortab, $temp, $hellosource, $forwardhost, $forwardport);
+
+$method =~ /(^[ ,\t]*smtp:[ ,\t]*{[ ,\t]*) (.+) ([ ,\t]*\})/xis;
+$innermethod=$2; #we are getting rid of smtp:{ and } at the end;
+
+if ($innermethod ne ''){
+ $innermethod =~ s/[\s]+//gis;
+@fortab = split(/\;/gis, $innermethod); #table of forwads
+ foreach $temp (@fortab) {
+ #every row has the following statement
+ #[smtphellohost.mydomain]-[10.10.3.13]:10025
+ $temp =~/\[(.*)\]\-\[(.*)\]:(\d*)/gis;
+ $hellosource = $1;
+ $forwardhost = $2;
+ $forwardport = $3;
+
+ #we have two association tables where we keep
+ #host and ports where the messages should be fowarded
+ $dynforw_host{$hellosource}=$forwardhost;
+ $dynforw_port{$hellosource}=$forwardport;
+ }#foreach
+}#if ($innermethod ne '')
+
+if ($dynforw_host{$smtphelo} ne '' $dynforw_port{$smtphelo} ne ''){
+ $returnmethod = sprintf(smtp:[%s]:%s,$dynforw_host{$smtphelo},
$dynforw_port{$smtphelo});
+}else{
+ $returnmethod = '';
+}
+
+$returnmethod = Amavis::Util::untaint($returnmethod);
+$returnmethod;
+}
+
+
sub dynamic_destination($$$) {
my($method,$conn,$force_dynamic) = @_;
my($client_ip) = !defined($conn) ? undef : $conn-client_ip;
+ my($smtp_helo) = !defined($conn) ? undef : $conn-smtp_helo;
+ my($new_formethod) = ;
+
+ if ($method =~ /^smtp:\{/i) { #new format of forward method
+$method = sprintf(%s, getmethod4smtphelo($method, $smtp_helo));
+$new_formethod = $method;
+if ($new_formethod eq ){
+do_log(-1, Something wrong with forward_method=$method. I can't find
host to forward for smtp_helo=$smtp_helo);
+}else{
+do_log(1, I changed forward_method basing on smtp_helo=$smtp_helo.
New method is $new_formethod);
+$method=$new_formethod;
+
+}
+ }
+
if ($method =~ /^[A-Za-z0-9]*:/) {
my(@list); $list[0] = ''; my($j) = 0;
for ($method =~ /\G \[ (?: \\. | [^\]\\] )* \] | (?: \\. | [^\\] )*
-
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/___
AMaViS-user mailing list
AMaViS-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
[AMaViS-user] Always - BAD HEADER, Missing required header field: Date
Hello i have a Problem with amavis-new and amavis-milter. Both latest Versions (fresh compiled) on a CentOs5. I have a Problem that the Milter filling every Mail Header with a: One Example: Jul 4 10:51:29 myserver sendmail[16298]: l648pSuU016298: Milter insert (0): header: X-Spam-Status: No, score=1.184 tagged_above=- required=3.4\n\ttests=[ALL_TRUSTED=-1.44, INVALID_MSGID=1.71, MISSING_HEADERS=0.189,\n\tTO_CC_NONE=0.134, UPPERCASE_50_75=0.591] Jul 4 10:51:29 myserver sendmail[16298]: l648pSuU016298: Milter insert (0): header: X-Spam-Level: * Jul 4 10:51:29 myserver sendmail[16298]: l648pSuU016298: Milter insert (0): header: X-Spam-Score: 1.184 Jul 4 10:51:29 myserver sendmail[16298]: l648pSuU016298: Milter insert (0): header: X-Spam-Flag: NO Jul 4 10:51:29 myserver sendmail[16298]: l648pSuU016298: Milter insert (0): header: X-Amavis-Alert: BAD HEADER, Missing required header field: Date Jul 4 10:51:29 myserver sendmail[16298]: l648pSuU016298: Milter insert (0): header: X-Virus-Scanned: amavisd-new at polyag.net Here comes the Header from the Mail above: Date: Wed, 4 Jul 2007 10:33:41 +0200 What could be my Problem ? Thanks Steve G - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
[AMaViS-user] OT: pdf attachment spam
Hello list, This is a slightly off topic question, but ... I would like to know if there is a way to inspect pdf attachments through amavisd-new (or any tool called through) ? Thanks In advance, -- - Philippe Latu IUT 'A' Paul Sabatier philippe.latu(at)iut-tlse3.fr - +33562258028 Enseignant/Chargé de mission Systèmes Réseau 115 C Route de Narbonne - BP67701 - 31077 Toulouse Cedex 4 / GNU/Linux philippe.latu(at)linux-france.org (GPG|PGP) KeyId 0x742A6424 http://www.linux-france.org/~platu / This message was sent using IMP, the Internet Messaging Program. - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
Re: [AMaViS-user] interesting patch proposition
Bartłomiej wrote: Hi, We, at Lerkins Group have used amavisd-new for our anti-spam anti-virus cluster, and we have found that that would be very usable to have the ability to resend the message that came from host X to host Y, based on config specified in amavisd.conf. As there was no such ability in amavisd-new we have developed a patch for amavisd, that enables such feature very easily while giving the ability to keep old behaviour untouched. Now the question is, would it be possible to incorporate this patch to amavisd official release, so anybody could use it, and also it would be easier to maintain amavisd after using that patch (right now if we want to upgrade, we have to apply this patch afer prebuild stage of FreeBSD port on each machine)? The patch is pasted below. Thanks a lot for any reply. Kind regards Evaluation will have to wait until Mark returns in 2 1/2 weeks. Gary V - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
Re: [AMaViS-user] Amavis-new memory usage
Prashant wrote: Hi, I am also facing the same problem while upgrading amavis to version 2.4.5. I have not included any blacklist rule in spamassassin PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 8487 amavis16 0 273m 263m 5004 S 0.0 13.0 1:42.57 amavisd 8484 amavis15 0 260m 250m 4960 S 0.0 12.3 2:19.15 amavisd 8481 amavis16 0 217m 207m 5000 S 0.0 10.2 1:53.16 amavisd 8479 amavis16 0 217m 207m 4960 S 0.0 10.2 1:26.47 amavisd 8483 amavis16 0 208m 193m 4952 S 0.0 9.5 1:37.21 amavisd 8482 amavis16 0 142m 131m 4960 S 0.0 6.5 1:45.12 amavisd 8480 amavis16 0 133m 121m 4952 S 4.7 6.0 0:44.37 amavisd --Prashant Dabre Right, it seems your problem is unrelated. I don't recall seeing anything like this happen before so the cause is unknown and I'm not sure how you would go about trying to debug this. Mark is out of town for 3 weeks, he may offer some idea when he returns. Gary V - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
Re: [AMaViS-user] Always - BAD HEADER, Missing required header field: Date
Stefan.G wrote: Sometimes i get not the Date BAD HEADER Error X-Amavis-Alert: BAD HEADER, Non-encoded 8-bit data (char E2 hex): X-eBay-due:\n\t\\342\\25431,76\\n Some webmail and bulkware clients are broken and send 8bit headers without encoding them. There's nothing you can do about it, except disabling the check or living with it. As far as you don't block/quarantine because of bad header, you can live with the warnings. - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
Re: [AMaViS-user] Always - BAD HEADER, Missing required header field: Date
On Wed, Jul 04, 2007 at 05:27:47PM +0200, mouss wrote: Stefan.G wrote: Sometimes i get not the Date BAD HEADER Error X-Amavis-Alert: BAD HEADER, Non-encoded 8-bit data (char E2 hex): X-eBay-due:\n\t\\342\\25431,76\\n Some webmail and bulkware clients are broken and send 8bit headers without encoding them. There's nothing you can do about it, except disabling the check or living with it. As far as you don't block/quarantine because of bad header, you can live with the warnings. Ok . Can i get Problems witht spamass. when i disable the bad header check - quarantine mode ? - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/ - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ AMaViS-user mailing list AMaViS-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
